can nodes fail/recover too fast to be seen?

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Jonathan Leivent
In Erlang, is it possible for a monitored node to fail and recover so
quickly that nodes monitoring it won't detect the failure?  Or, is there
some kind of internal persistent state that prevents this?

If such quick fail/recover without detection cases are possible, I think
I have to roll my own monitoring scheme, and not rely on
erlang:monitor_node.

-- Jonathan

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.

Cheers
Tim

On 5 Jul 2013, at 15:49, Jonathan Leivent <jleivent> wrote:

> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?  Or, is there some kind of internal persistent state that prevents this?
>
> If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node.
>
> -- Jonathan
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Sergej Jurecko
Well yes erlang does reconnect, but you still get a nodedown/nodeup message no?


Sergej

On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:

> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.
>
> Cheers
> Tim
>
> On 5 Jul 2013, at 15:49, Jonathan Leivent <jleivent> wrote:
>
>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?  Or, is there some kind of internal persistent state that prevents this?
>>
>> If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node.
>>
>> -- Jonathan
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Gleb Peregud-2
If it was an intermittent network issue, TCP can mask the problem and
Erlang would never know about it. And I believe Erlang depends on TCP
and explicit pings to detect dead nodes. But if remote failed node has
been restarted in mean time, Erlang will detect it as, IIRC, it
maintains some kind of "node version" in it's distribution protocol
state.

On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko <sergej.jurecko> wrote:

> Well yes erlang does reconnect, but you still get a nodedown/nodeup message no?
>
>
> Sergej
>
> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
>
>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.
>>
>> Cheers
>> Tim
>>
>> On 5 Jul 2013, at 15:49, Jonathan Leivent <jleivent> wrote:
>>
>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?  Or, is there some kind of internal persistent state that prevents this?
>>>
>>> If such quick fail/recover without detection cases are possible, I think I have to roll my own monitoring scheme, and not rely on erlang:monitor_node.
>>>
>>> -- Jonathan
>>> _______________________________________________
>>> erlang-questions mailing list
>>> erlang-questions
>>> http://erlang.org/mailman/listinfo/erlang-questions
>> _______________________________________________
>> erlang-questions mailing list
>> erlang-questions
>> http://erlang.org/mailman/listinfo/erlang-questions
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Mike Oxford-2
net_ticktime defines "how quickly" the nodes ping each other and, thus,
notice that a node is down.

http://www.erlang.org/doc/man/net_kernel.html#set_net_ticktime-1

-mox


On Fri, Jul 5, 2013 at 10:00 AM, Gleb Peregud <gleber.p> wrote:

> If it was an intermittent network issue, TCP can mask the problem and
> Erlang would never know about it. And I believe Erlang depends on TCP
> and explicit pings to detect dead nodes. But if remote failed node has
> been restarted in mean time, Erlang will detect it as, IIRC, it
> maintains some kind of "node version" in it's distribution protocol
> state.
>
> On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko <sergej.jurecko>
> wrote:
> > Well yes erlang does reconnect, but you still get a nodedown/nodeup
> message no?
> >
> >
> > Sergej
> >
> > On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
> >
> >> As i understand it, this can and does happen, because erlang does
> automatic reconnect in order to provide reliable communications.
> >>
> >> Cheers
> >> Tim
> >>
> >> On 5 Jul 2013, at 15:49, Jonathan Leivent <jleivent> wrote:
> >>
> >>> In Erlang, is it possible for a monitored node to fail and recover so
> quickly that nodes monitoring it won't detect the failure?  Or, is there
> some kind of internal persistent state that prevents this?
> >>>
> >>> If such quick fail/recover without detection cases are possible, I
> think I have to roll my own monitoring scheme, and not rely on
> erlang:monitor_node.
> >>>
> >>> -- Jonathan
> >>> _______________________________________________
> >>> erlang-questions mailing list
> >>> erlang-questions
> >>> http://erlang.org/mailman/listinfo/erlang-questions
> >> _______________________________________________
> >> erlang-questions mailing list
> >> erlang-questions
> >> http://erlang.org/mailman/listinfo/erlang-questions
> >
> > _______________________________________________
> > erlang-questions mailing list
> > erlang-questions
> > http://erlang.org/mailman/listinfo/erlang-questions
> _______________________________________________
> erlang-questions mailing list
> erlang-questions
> http://erlang.org/mailman/listinfo/erlang-questions
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130705/2de52046/attachment.html>

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
In reply to this post by Gleb Peregud-2
On 5 Jul 2013, at 18:00, Gleb Peregud wrote:

> If it was an intermittent network issue, TCP can mask the problem and
> Erlang would never know about it. And I believe Erlang depends on TCP
> and explicit pings to detect dead nodes. But if remote failed node has
> been restarted in mean time, Erlang will detect it as, IIRC, it
> maintains some kind of "node version" in it's distribution protocol
> state.
>

That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way.

But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending.

There are a couple of papers worth reading in this space:

[1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund)
[2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle)

Cheers,
Tim

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
In reply to this post by Mike Oxford-2
On 5 Jul 2013, at 18:32, Mike Oxford wrote:

> net_ticktime defines "how quickly" the nodes ping each other and, thus, notice that a node is down.
>

Sure, but you've got to be careful with that one. Setting the net_ticktime too short can lead to false positives, i.e., thinking nodes are down when they're simply busy, perhaps responding to some other party or struggling due to a busy_dist_port. Setting the net_ticktime too long might have the opposite effect.

Another one worth a read in that space is "On Failure Detection Algorithms in Overlay Networks - Shelley Q. Zhuang, Dennis Geels, Ion Stoica, Randy H. Katz".

Cheers,
Tim

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Jonathan Leivent
In reply to this post by Tim Watson
On 07/05/2013 01:33 PM, Tim Watson wrote:

> On 5 Jul 2013, at 18:00, Gleb Peregud wrote:
>
>> If it was an intermittent network issue, TCP can mask the problem and
>> Erlang would never know about it. And I believe Erlang depends on TCP
>> and explicit pings to detect dead nodes. But if remote failed node has
>> been restarted in mean time, Erlang will detect it as, IIRC, it
>> maintains some kind of "node version" in it's distribution protocol
>> state.
>>
>
> That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way.
>
> But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending.

Is this even true for net_kernel:monitor_nodes?  The doc suggests that
there is an ordering between messages and nodeup/nodedown notifications,
at least in later releases.

> There are a couple of papers worth reading in this space:
>
> [1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund)
> [2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle)
>

Thanks - I just skimmed the first paper.  I don't have access to the second.

-- Jonathan


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
On 5 Jul 2013, at 20:10, Jonathan Leivent wrote:

> On 07/05/2013 01:33 PM, Tim Watson wrote:
>> On 5 Jul 2013, at 18:00, Gleb Peregud wrote:
>>
>>> If it was an intermittent network issue, TCP can mask the problem and
>>> Erlang would never know about it. And I believe Erlang depends on TCP
>>> and explicit pings to detect dead nodes. But if remote failed node has
>>> been restarted in mean time, Erlang will detect it as, IIRC, it
>>> maintains some kind of "node version" in it's distribution protocol
>>> state.
>>>
>>
>> That's correct, so issues can arise if the node goes away and comes back within net_ticktime and TCP masks the fact. You can also run into problems with pid re-use in this way.
>>
>> But perhaps more importantly than this, if a node goes away and the runtime does notice, there's no guarantee that you'll see the 'DOWN' (or nodedown) message in any particular order with regards other communications flying around the system. This can make it difficult to identify which messages might potentially need re-sending.
>
> Is this even true for net_kernel:monitor_nodes?  The doc suggests that there is an ordering between messages and nodeup/nodedown notifications, at least in later releases.
>

That ordering guarantee only applies to the process that called monitor_nodes, and afaict it won't apply unless you're using (what the runtime deems to be) a new connection. If you get hit by the situation where the runtime doesn't notice the node has gone down, that's a different matter - as Gleb mentioned, TCP can mask a disconnect/reconnect and net_ticktime isn't instantaneous (for obvious reasons).

>> There are a couple of papers worth reading in this space:
>>
>> [1] Programming Distributed Erlang Applications: Pitfalls and Recipes (Hans Svensson, Lars-?ke Fredlund)
>> [2] A Unified Semantics for Future Erlang (Hans Svensson, Lars-?ke Fredlund, Clara Benac Earle)
>>
>
> Thanks - I just skimmed the first paper.  I don't have access to the second.

You'll need an ACM (or similar) account. The first one's more relevant anyway.

Cheers,
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20130705/fc32bc09/attachment.html>

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Per Hedeland-4
In reply to this post by Gleb Peregud-2
Gleb Peregud <gleber.p> wrote:
>
>If it was an intermittent network issue, TCP can mask the problem and
>Erlang would never know about it.

This is a completely different (non-)problem, the question was about a
node failing and recovering. If both nodes keep running and there is an
"intermittent network issue" that "TCP can mask", there *is* no problem
- TCP deals with it, that's arguably its main purpose in life.

>On Fri, Jul 5, 2013 at 5:24 PM, Sergej Jurecko <sergej.jurecko> wrote:
>> Well yes erlang does reconnect, but you still get a nodedown/nodeup message no?

Yes, and 'DOWN' for the process(es). But in both cases only if you
monitor (or link => exit signal).

>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
>>
>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.

No.

>>> On 5 Jul 2013, at 15:49, Jonathan Leivent <jleivent> wrote:
>>>
>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?

No. The TCP connection to the old node instance cannot be used for
communication with the new node instance, i.e. there is no way that
communication with the new node instance can be established without the
local VM generating node_down/'DOWN'/exit messages for the old instance.

>>>>  Or, is there some kind of internal persistent state that prevents this?

This is where it potentially gets interesting - i.e. assuming *no*
monitoring or linking - and that's where the "creation" part of a node
identifier comes into play. If a distributed node restarts, it will get
a new "creation" value courtesy of epmd, and any any pid() values
referring to the old node instance will be invalid.

However the "creation" is only 2 bits, so if a node restarts frequently,
old pid() values may become "valid" once again, i.e. referring to a new,
different process. Is it a problem in practice, outside academic
research papers? I've never heard of such a case. If you really have
this problem, you should probably look at why the node keeps
restarting...

--Per Hedeland

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
Thanks for the clarifications Per - that's cleared up a few things that I was unaware of.

On 5 Jul 2013, at 20:23, Per Hedeland wrote:
>>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
>>>
>>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.
>
> No.

So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects?

>>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?
>
> No. The TCP connection to the old node instance cannot be used for
> communication with the new node instance, i.e. there is no way that
> communication with the new node instance can be established without the
> local VM generating node_down/'DOWN'/exit messages for the old instance.
>

Just out of interest, is this enforced by epmd or internally? Also, it would be worth making this explicit in the documentation somewhere, since this question comes up frequently.

>>>>> Or, is there some kind of internal persistent state that prevents this?
>
> This is where it potentially gets interesting - i.e. assuming *no*
> monitoring or linking - and that's where the "creation" part of a node
> identifier comes into play. If a distributed node restarts, it will get
> a new "creation" value courtesy of epmd, and any any pid() values
> referring to the old node instance will be invalid.
>

Does this depend on epmd having stayed up and running the whole time, or does epmd now have some local persistent state?

Cheers,
Tim


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Jonathan Leivent
In reply to this post by Per Hedeland-4

>>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?
>
> No. The TCP connection to the old node instance cannot be used for
> communication with the new node instance, i.e. there is no way that
> communication with the new node instance can be established without the
> local VM generating node_down/'DOWN'/exit messages for the old instance.

OK - so it is TCP that saves the day.  I'm not sure I want to be that
dependent on TCP, though.  But, for now, it's an convenient solution.

>
>>>>>   Or, is there some kind of internal persistent state that prevents this?
>
> This is where it potentially gets interesting - i.e. assuming *no*
> monitoring or linking - and that's where the "creation" part of a node
> identifier comes into play. If a distributed node restarts, it will get
> a new "creation" value courtesy of epmd, and any any pid() values
> referring to the old node instance will be invalid.
>
> However the "creation" is only 2 bits, so if a node restarts frequently,
> old pid() values may become "valid" once again, i.e. referring to a new,
> different process. Is it a problem in practice, outside academic
> research papers? I've never heard of such a case. If you really have
> this problem, you should probably look at why the node keeps
> restarting...
>
> --Per Hedeland

That's interesting.  I'll make sure I monitor a node before relying on
pid values from that node.

Thanks,
Jonathan


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Per Hedeland-4
In reply to this post by Tim Watson
Tim Watson <watson.timothy> wrote:
>
>On 5 Jul 2013, at 20:23, Per Hedeland wrote:
>>>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
>>>>
>>>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.
>>
>> No.
>
>So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects?

No, of course they can be dropped - if the destination node goes down,
it's impossible to know whether a given, sent message was a) received at
the remote *host*, b) received by the remote Erlang node, c) received by
the remote Erlang process, d) processed by the remote Erlang process, or
e) none of the above. But if you monitor/link (e.g. use
gen_server:call()), you will know that "badness happened", and can take
corrective action. "Re-sending only messages that need to be re-sent" is
not possible in general, and this is not specific to Erlang distribution.

See also http://www.erlang.org/faq/academic.html#id58000, which Matthias
Lang was kind enough to write up in a nice form based on some ramblings
of mine in the distant past. It could probably use s/link/monitor/, but
the general principle holds.

>>>>>> In Erlang, is it possible for a monitored node to fail and recover so quickly that nodes monitoring it won't detect the failure?
>>
>> No. The TCP connection to the old node instance cannot be used for
>> communication with the new node instance, i.e. there is no way that
>> communication with the new node instance can be established without the
>> local VM generating node_down/'DOWN'/exit messages for the old instance.
>>
>
>Just out of interest, is this enforced by epmd or internally?

epmd has no role in inter-node communication once the connection has
been established. TCP enforces "cannot be used for ...". The
VM/net_kernel will not make a new connection until it has decided that
the old one isn't working any more, and at that point it will generate
the node_down/'DOWN'/exit messages.

>>>>>> Or, is there some kind of internal persistent state that prevents this?
>>
>> This is where it potentially gets interesting - i.e. assuming *no*
>> monitoring or linking - and that's where the "creation" part of a node
>> identifier comes into play. If a distributed node restarts, it will get
>> a new "creation" value courtesy of epmd, and any any pid() values
>> referring to the old node instance will be invalid.
>>
>
>Does this depend on epmd having stayed up and running the whole time, or does epmd now have some local persistent state?

Good point - it depends on epmd having stayed up and running, i.e. if
the *host* reboots, there is a 25% possibility of the new node instance
getting the same "creation" value. However, see the FAQ again - if your
communication is critical, you can't depend on "creation" - it won't
tell you about failures anyway. It's just a way to try to prevent that
messages get delivered to the wrong process.

--Per

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Per Hedeland-4
In reply to this post by Jonathan Leivent
Jonathan Leivent <jleivent> wrote:
>
>OK - so it is TCP that saves the day.  I'm not sure I want to be that
>dependent on TCP, though.

I'm not sure how to interpret that statement. If you mean

- "I want to be able to use something other than TCP" - fine, as long as
it provides the same guarantees, and you can figure out how to hook it
into the Erlang distribution.

- "I want more reliability than TCP can give me" - again fine, assuming
that you understand precisely what it is that TCP gives you, i.e. that
it either fulfills its goal of providing an ordered, reliable byte
stream or tells you about its failure to do so. See the FAQ entry I
pointed to - and remember that the lack of an application-level ack is
no guarantee that the message wasn't processed *at all*.

- "I don't think TCP is reliable enough to be dependant on" - in that
case I would suggest that you forget about the Internet and networking
in general.

--Per

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Jonathan Leivent
On 07/05/2013 04:31 PM, Per Hedeland wrote:

> Jonathan Leivent <jleivent> wrote:
>>
>> OK - so it is TCP that saves the day.  I'm not sure I want to be that
>> dependent on TCP, though.
>
> I'm not sure how to interpret that statement. If you mean
>
> - "I want to be able to use something other than TCP" - fine, as long as
> it provides the same guarantees, and you can figure out how to hook it
> into the Erlang distribution.

I might want to use UDP in the future.  Or, something that is just
slightly more than UDP - such as providing traffic control, but not
reliability.  My app is supposed to be able to handle
lost/duplicate/unordered messages on is own - so much of what TCP
provides is redundant.


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Per Hedeland-4
Jonathan Leivent <jleivent> wrote:

>
>On 07/05/2013 04:31 PM, Per Hedeland wrote:
>> Jonathan Leivent <jleivent> wrote:
>>>
>>> OK - so it is TCP that saves the day.  I'm not sure I want to be that
>>> dependent on TCP, though.
>>
>> I'm not sure how to interpret that statement. If you mean
>>
>> - "I want to be able to use something other than TCP" - fine, as long as
>> it provides the same guarantees, and you can figure out how to hook it
>> into the Erlang distribution.
>
>I might want to use UDP in the future.  Or, something that is just
>slightly more than UDP - such as providing traffic control, but not
>reliability. My app is supposed to be able to handle
>lost/duplicate/unordered messages on is own - so much of what TCP
>provides is redundant.

So use UDP to send Erlang terms as binaries over plain sockets, gen_udp
is a fine module. But don't try to implement Erlang distribution on top
of UDP - some number of years from now you will at best have
re-implemented TCP with characteristics that are significantly inferior
to the real thing. Or you may have something that *isn't* Erlang
distribution.

--Per

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
In reply to this post by Per Hedeland-4
On 5 Jul 2013, at 21:07, Per Hedeland <per> wrote:

>
> No, of course they can be dropped - if the destination node goes down,
> it's impossible to know whether a given, sent message was a) received at
> the remote *host*, b) received by the remote Erlang node, c) received by
> the remote Erlang process, d) processed by the remote Erlang process, or
> e) none of the above. But if you monitor/link (e.g. use
> gen_server:call()), you will know that "badness happened", and can take
> corrective action. "Re-sending only messages that need to be re-sent" is
> not possible in general, and this is not specific to Erlang distribution.
>

Yeah I know about those issues. We were discussing it on another thread earlier today. :)

> See also http://www.erlang.org/faq/academic.html#id58000, which Matthias
> Lang was kind enough to write up in a nice form based on some ramblings
> of mine in the distant past. It could probably use s/link/monitor/, but
> the general principle holds.
>

I'll take a look, thanks.

>>
>> Just out of interest, is this enforced by epmd or internally?
>
> epmd has no role in inter-node communication once the connection has
> been established.

Thats what I thought.

> TCP enforces "cannot be used for ...". The
> VM/net_kernel will not make a new connection until it has decided that
> the old one isn't working any more, and at that point it will generate
> the node_down/'DOWN'/exit messages.

So given that intermediate kit could be proxying the connection, it is /possible/ that the node was reset but the network layer didn't notice? I mention that because if it is the case, people should bear it in mind when designing their infrastructure.

> there is a 25% possibility of the new node instance
> getting the same "creation" value.

I wasn't fully aware of this node generation tracking stuff. We see a lot of stale pids (discarding messages from
Old incarnations of the node) in some deployments, but then we actually store pids in mnesia which makes things a bit more complicated viz getting the timing right between node down messages,  updating the db and actually using the pids.

Cheers
Tim

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Jonathan Leivent
In reply to this post by Per Hedeland-4
On 07/05/2013 05:44 PM, Per Hedeland wrote:

> Jonathan Leivent <jleivent> wrote:
>>
>> On 07/05/2013 04:31 PM, Per Hedeland wrote:
>>> Jonathan Leivent <jleivent> wrote:
>>>>
>>>> OK - so it is TCP that saves the day.  I'm not sure I want to be that
>>>> dependent on TCP, though.
>>>
>>> I'm not sure how to interpret that statement. If you mean
>>>
>>> - "I want to be able to use something other than TCP" - fine, as long as
>>> it provides the same guarantees, and you can figure out how to hook it
>>> into the Erlang distribution.
>>
>> I might want to use UDP in the future.  Or, something that is just
>> slightly more than UDP - such as providing traffic control, but not
>> reliability. My app is supposed to be able to handle
>> lost/duplicate/unordered messages on is own - so much of what TCP
>> provides is redundant.
>
> So use UDP to send Erlang terms as binaries over plain sockets, gen_udp
> is a fine module. But don't try to implement Erlang distribution on top
> of UDP - some number of years from now you will at best have
> re-implemented TCP with characteristics that are significantly inferior
> to the real thing. Or you may have something that *isn't* Erlang
> distribution.
>
> --Per
>

The Erlang implementation is a reference implementation - so it will be
translated elsewhere, including to places where TCP has too much
overhead.  I'm not trying to re-invent Erlang's distributed
infrastructure for Erlang.  But, I might want to demonstrate that the
reference implementation, although in Erlang, can be ported elsewhere
without also having to port much of Erlang's infrastructure with it.

-- Jonathan


Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Per Hedeland-4
In reply to this post by Tim Watson
Tim Watson <watson.timothy> wrote:
>
>On 5 Jul 2013, at 21:07, Per Hedeland <per> wrote:
>
>> TCP enforces "cannot be used for ...". The
>> VM/net_kernel will not make a new connection until it has decided that
>> the old one isn't working any more, and at that point it will generate
>> the node_down/'DOWN'/exit messages.
>
>So given that intermediate kit could be proxying the connection, it is /possible/ that the node was reset but the network layer didn't notice? I mention that because if it is the case, people should bear it in mind when designing their infrastructure.

I'm actually not sure what you're talking about here - AFAIK it's not
common or even easily doable to "proxy" Erlang distribution connections.
But if you somehow manage to do it for some reason, you need to make
sure that the resulting connection has the same end-to-end semantics as
TCP at least in the reliability department. I.e. if your proxy "hides"
the fact that a remote node restarted, it is basically broken.

--Per

Reply | Threaded
Open this post in threaded view
|

can nodes fail/recover too fast to be seen?

Tim Watson
In reply to this post by Per Hedeland-4
On 5 Jul 2013, at 21:07, Per Hedeland wrote:

> Tim Watson <watson.timothy> wrote:
>>
>> On 5 Jul 2013, at 20:23, Per Hedeland wrote:
>>>>> On Jul 5, 2013, at 5:22 PM, Tim Watson wrote:
>>>>>
>>>>>> As i understand it, this can and does happen, because erlang does automatic reconnect in order to provide reliable communications.
>>>
>>> No.
>>
>> So is the Svensson and Frelund paper (viz [2] from my earlier post) incorrect in its assertion that messages between nodes can be dropped in the face of rapid node reconnects?
>
> No, of course they can be dropped - if the destination node goes down,
> it's impossible to know whether a given, sent message was a) received at
> the remote *host*, b) received by the remote Erlang node, c) received by
> the remote Erlang process, d) processed by the remote Erlang process, or
> e) none of the above. But if you monitor/link (e.g. use
> gen_server:call()), you will know that "badness happened", and can take
> corrective action.

Right, that makes sense. The bit about "signal loss" that I was missing there is that monitoring (or linking) *is* guaranteed in the situation where a node disconnects and reconnects quickly.

> "Re-sending only messages that need to be re-sent" is
> not possible in general, and this is not specific to Erlang distribution.
>

That's not what I was talking about, since we know sending is meant to be asynchronous and never fail - I was thinking about both ends, not just sender or receiver. The point I was making is that even if monitor_nodes can guarantee that you'll see nodedown before any other traffic over a new connection, you can only guarantee that for a process that actually called monitor_nodes (or is monitoring the remote process), since if you rely on some *other* process to do monitoring, then there's no ordering guarantee to be had. In other words (as the paper puts it) you have to monitor each interaction to be sure. And this holds, more importantly, for receipt as well. So if you aren't monitoring a remote process (or node) are you're holding on to a remote pid, then you've got to have your wits about you.

Anyway, it's good to know that the monitoring guarantees are absolute.

Cheers,
Tim



12