Reconnect to nodes

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Reconnect to nodes

Roberto Ostinelli
Dear list,
A simple question: am I correct that, when a node is removed because of a net split, you need to have your own application logic to reconnect to it, and nothing in the VM will try doing that for you?

Let me show you an example. I have two nodes: [hidden email] and [hidden email] that are connected to each other:


On node 2 I listen for nodedown events of node 1:

([hidden email])1> monitor_node('[hidden email]', true).
true

On node 1, I simulate a net splits with the best option I've found until now, i.e suspending the net_kernel process:

([hidden email])2> sys:suspend(net_kernel).
ok

After ~60 seconds on node 2 I get:

=ERROR REPORT==== 22-Apr-2016::11:28:21 ===
** Node '[hidden email]' not responding **
** Removing (timedout) connection **
([hidden email])2> flush().
Shell got {nodedown,'[hidden email]'}

Now the two nodes are disconnected:

([hidden email])3> nodes().
[]

([hidden email])3> nodes().
[]

Even when I resume the net_kernel process:

([hidden email])4> sys:resume(net_kernel). 
ok

The nodes do not reconnect:

([hidden email])5> nodes().
[]

I'm ok with this, though I would like to confirm that my understanding is correct.
If so, does everyone just implement some standard connection manager that does only reconnections?

Thank you,
r.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Boroska András
Hi Roberto,

Your assumptions are correct. Note, however, that rpc calls via the rpc module will reconnect the nodes automatically which is a nice convenience.

Br,
Andras 

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Serge Aleynikov-3
In reply to this post by Roberto Ostinelli
Roberto,

This is the expected behavior. Note that the nodes will automatically reconnect by default when either node has a process that sends a message to a process on the remote node.  This reconnection behavior can be modified by setting the kernel's 'dist_auto_connect' option.

Other applications (such as mnesia) may require custom recovery from a network split, which is one of the reasons why automatic reconnection may not be desirable.

Regards,

Serge

On Fri, Apr 22, 2016 at 5:39 AM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
A simple question: am I correct that, when a node is removed because of a net split, you need to have your own application logic to reconnect to it, and nothing in the VM will try doing that for you?

Let me show you an example. I have two nodes: [hidden email] and [hidden email] that are connected to each other:


On node 2 I listen for nodedown events of node 1:

([hidden email])1> monitor_node('[hidden email]', true).
true

On node 1, I simulate a net splits with the best option I've found until now, i.e suspending the net_kernel process:

([hidden email])2> sys:suspend(net_kernel).
ok

After ~60 seconds on node 2 I get:

=ERROR REPORT==== 22-Apr-2016::11:28:21 ===
** Node '[hidden email]' not responding **
** Removing (timedout) connection **
([hidden email])2> flush().
Shell got {nodedown,'[hidden email]'}

Now the two nodes are disconnected:

([hidden email])3> nodes().
[]

([hidden email])3> nodes().
[]

Even when I resume the net_kernel process:

([hidden email])4> sys:resume(net_kernel). 
ok

The nodes do not reconnect:

([hidden email])5> nodes().
[]

I'm ok with this, though I would like to confirm that my understanding is correct.
If so, does everyone just implement some standard connection manager that does only reconnections?

Thank you,
r.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Roberto Ostinelli
Thank you both for confirming.
Hence I guess some kind of a basic reconnection manager might be helpful here.

Thanks!

r.

On Fri, Apr 22, 2016 at 12:35 PM, Serge Aleynikov <[hidden email]> wrote:
Roberto,

This is the expected behavior. Note that the nodes will automatically reconnect by default when either node has a process that sends a message to a process on the remote node.  This reconnection behavior can be modified by setting the kernel's 'dist_auto_connect' option.

Other applications (such as mnesia) may require custom recovery from a network split, which is one of the reasons why automatic reconnection may not be desirable.

Regards,

Serge

On Fri, Apr 22, 2016 at 5:39 AM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
A simple question: am I correct that, when a node is removed because of a net split, you need to have your own application logic to reconnect to it, and nothing in the VM will try doing that for you?

Let me show you an example. I have two nodes: [hidden email] and [hidden email] that are connected to each other:


On node 2 I listen for nodedown events of node 1:

([hidden email])1> monitor_node('[hidden email]', true).
true

On node 1, I simulate a net splits with the best option I've found until now, i.e suspending the net_kernel process:

([hidden email])2> sys:suspend(net_kernel).
ok

After ~60 seconds on node 2 I get:

=ERROR REPORT==== 22-Apr-2016::11:28:21 ===
** Node '[hidden email]' not responding **
** Removing (timedout) connection **
([hidden email])2> flush().
Shell got {nodedown,'[hidden email]'}

Now the two nodes are disconnected:

([hidden email])3> nodes().
[]

([hidden email])3> nodes().
[]

Even when I resume the net_kernel process:

([hidden email])4> sys:resume(net_kernel). 
ok

The nodes do not reconnect:

([hidden email])5> nodes().
[]

I'm ok with this, though I would like to confirm that my understanding is correct.
If so, does everyone just implement some standard connection manager that does only reconnections?

Thank you,
r.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Serge Aleynikov-3
You can take a look at this app that handles detection of network splits and auto-reconnecting:


Serge

On Fri, Apr 22, 2016 at 7:42 AM, Roberto Ostinelli <[hidden email]> wrote:
Thank you both for confirming.
Hence I guess some kind of a basic reconnection manager might be helpful here.

Thanks!

r.

On Fri, Apr 22, 2016 at 12:35 PM, Serge Aleynikov <[hidden email]> wrote:
Roberto,

This is the expected behavior. Note that the nodes will automatically reconnect by default when either node has a process that sends a message to a process on the remote node.  This reconnection behavior can be modified by setting the kernel's 'dist_auto_connect' option.

Other applications (such as mnesia) may require custom recovery from a network split, which is one of the reasons why automatic reconnection may not be desirable.

Regards,

Serge

On Fri, Apr 22, 2016 at 5:39 AM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
A simple question: am I correct that, when a node is removed because of a net split, you need to have your own application logic to reconnect to it, and nothing in the VM will try doing that for you?

Let me show you an example. I have two nodes: [hidden email] and [hidden email] that are connected to each other:


On node 2 I listen for nodedown events of node 1:

([hidden email])1> monitor_node('[hidden email]', true).
true

On node 1, I simulate a net splits with the best option I've found until now, i.e suspending the net_kernel process:

([hidden email])2> sys:suspend(net_kernel).
ok

After ~60 seconds on node 2 I get:

=ERROR REPORT==== 22-Apr-2016::11:28:21 ===
** Node '[hidden email]' not responding **
** Removing (timedout) connection **
([hidden email])2> flush().
Shell got {nodedown,'[hidden email]'}

Now the two nodes are disconnected:

([hidden email])3> nodes().
[]

([hidden email])3> nodes().
[]

Even when I resume the net_kernel process:

([hidden email])4> sys:resume(net_kernel). 
ok

The nodes do not reconnect:

([hidden email])5> nodes().
[]

I'm ok with this, though I would like to confirm that my understanding is correct.
If so, does everyone just implement some standard connection manager that does only reconnections?

Thank you,
r.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Roberto Ostinelli
Thanks!

_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions
Reply | Threaded
Open this post in threaded view
|

Re: Reconnect to nodes

Michael Truog
In reply to this post by Roberto Ostinelli
This is in CloudI (in cloudi_core) as shown at http://cloudi.org/api.html#2_nodes_set with reconnect_start and reconnect_delay determining the interval for checking the node connections.  If you need help using cloudi_core, that is at https://github.com/CloudI/CloudI/tree/master/examples/hello_world5 .

On 04/22/2016 04:42 AM, Roberto Ostinelli wrote:
Thank you both for confirming.
Hence I guess some kind of a basic reconnection manager might be helpful here.

Thanks!

r.

On Fri, Apr 22, 2016 at 12:35 PM, Serge Aleynikov <[hidden email]> wrote:
Roberto,

This is the expected behavior. Note that the nodes will automatically reconnect by default when either node has a process that sends a message to a process on the remote node.  This reconnection behavior can be modified by setting the kernel's 'dist_auto_connect' option.

Other applications (such as mnesia) may require custom recovery from a network split, which is one of the reasons why automatic reconnection may not be desirable.

Regards,

Serge

On Fri, Apr 22, 2016 at 5:39 AM, Roberto Ostinelli <[hidden email]> wrote:
Dear list,
A simple question: am I correct that, when a node is removed because of a net split, you need to have your own application logic to reconnect to it, and nothing in the VM will try doing that for you?

Let me show you an example. I have two nodes: [hidden email] and [hidden email] that are connected to each other:

([hidden email])1> nodes().

On node 2 I listen for nodedown events of node 1:

([hidden email])1> monitor_node('[hidden email]', true).
true

On node 1, I simulate a net splits with the best option I've found until now, i.e suspending the net_kernel process:

([hidden email])2> sys:suspend(net_kernel).
ok

After ~60 seconds on node 2 I get:

=ERROR REPORT==== 22-Apr-2016::11:28:21 ===
** Node '[hidden email]' not responding **
** Removing (timedout) connection **
([hidden email])2> flush().
Shell got {nodedown,'[hidden email]'}

Now the two nodes are disconnected:

([hidden email])3> nodes().
[]

([hidden email])3> nodes().
[]

Even when I resume the net_kernel process:

([hidden email])4> sys:resume(net_kernel). 
ok

The nodes do not reconnect:

([hidden email])5> nodes().
[]

I'm ok with this, though I would like to confirm that my understanding is correct.
If so, does everyone just implement some standard connection manager that does only reconnections?

Thank you,
r.




_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions





_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions


_______________________________________________
erlang-questions mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-questions