Non-overlapping Application Distribution Node Sets

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
Fetch here:

   git fetch git://github.com/vances/otp.git non_overlap_application_distribution

Browse here:

   https://github.com/vances/otp/commit/61f4da70e32bf745d96455b6d2f2ca42c4e4a3a7

Commit message:

Support non-overlapping application distribution nodes

Currently all known nodes should have the same value for the
kernel application's 'distributed' environment variable.  It
is not expected that any application will be distributed on
on more than one set of nodes.

It should be possible to distributed an application between
multiple non-overlapping sets of nodes.  For example with this
system configuration file on nodes a and b:

   [{kernel,
      [{distributed, [{app_no, [a, b]}]},
       {sync_nodes_optional, [a, b]},
       {sync_nodes_timeout, 5000}]}].

... and this system configuration file on nodes c and d:

   [{kernel,
      [{distributed, [{app_no, [c, d]}]},
       {sync_nodes_optional, [c, d]},
       {sync_nodes_timeout, 5000}]}].

Other applications may be distributed involving some other
combination of these nodes without interference.

This patch adds checks in dist_ac to ignore DAC protocol
messages of an application from nodes not included in that
application's distribution specification locally.


Rationale:

We often want to have active/standby pairs for applications
while also having multiple instances of the application running
on different nodes.  When nodes within the cluster are communicating
in order to, for example, distribute mnesia tables, suddenly there
is a potential conflict between these otherwise unrelated node pairs.

Currently there is a window of time during node (re)starts where a
conflict may occur.  This patch simply corrects this error case.

The documentation is somewaht unclear as to whether the configuration
above is legal or not.  The fact that, other than in the race condition
noted above, this distribution does work as expected allows one to use
the more liberal interpretation that when it says:

   "All involved nodes must have the same value for distributed and
    sync_nodes_timeout, or the behaviour of the system is undefined."

Involved nodes refers to nodes involved in tthis application's distribution.
With this patch that interpretation holds true.

Tests:

The existing tests are extended to support the following application
distribution configuration:

   cp1:
      [{kernel,
         [{sync_nodes_optional, [cp2, cp3]},
          {distributed,
             [{app1, [cp1, cp2, cp3]},
              {app2, 2000, [cp1, cp2, cp3]},
              {app_sp, 1000, [{cp1, cp2}, cp3]},"
              {app_no, 1000, [cp1, cp2]}]}]}].
'
   cp2:
      [{kernel,
         [{sync_nodes_optional, [cp1, cp3]},
          {distributed,
             [{app1, [cp1, cp2, cp3]},
              {app2, 2000, [cp1, cp2, cp3]},
              {app_sp, 1000, [{cp1, cp2}, cp3]},"
              {app_no, 1000, [cp1, cp2]}]}]}].

   cp3:
      [{kernel,
         [{sync_nodes_optional, [cp1, cp2, cp4]},
          {distributed,
             [{app1, [cp1, cp2, cp3]},
              {app2, 2000, [cp1, cp2, cp3]},
              {app_sp, 1000, [{cp1, cp2}, cp3]},"
              {app_no, 1000, [cp3, cp4]}]}]}].

   cp4:
      [{kernel,
         [{sync_nodes_optional, [cp2]},
          {distributed,
              {app_no, 1000, [cp3, cp4]}]}]}].

The Cp4 node is added along with the app_no application which is
distributed in two active/standby pairs on Cp1/Cp2 and Cp3/Cp4.
The tests check that these pairs are unaffected by the other applications'
starts, stops, failovers and takeovers.  And that they do not affect each
other's.

--
        -Vance

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Fredrik
On 05/03/2013 10:07 AM, Vance Shipley wrote:

> Fetch here:
>
>     git fetch git://github.com/vances/otp.git non_overlap_application_distribution
>
> Browse here:
>
>     https://github.com/vances/otp/commit/61f4da70e32bf745d96455b6d2f2ca42c4e4a3a7
>
> Commit message:
>
> Support non-overlapping application distribution nodes
>
> Currently all known nodes should have the same value for the
> kernel application's 'distributed' environment variable.  It
> is not expected that any application will be distributed on
> on more than one set of nodes.
>
> It should be possible to distributed an application between
> multiple non-overlapping sets of nodes.  For example with this
> system configuration file on nodes a and b:
>
>     [{kernel,
>        [{distributed, [{app_no, [a, b]}]},
>         {sync_nodes_optional, [a, b]},
>         {sync_nodes_timeout, 5000}]}].
>
> ... and this system configuration file on nodes c and d:
>
>     [{kernel,
>        [{distributed, [{app_no, [c, d]}]},
>         {sync_nodes_optional, [c, d]},
>         {sync_nodes_timeout, 5000}]}].
>
> Other applications may be distributed involving some other
> combination of these nodes without interference.
>
> This patch adds checks in dist_ac to ignore DAC protocol
> messages of an application from nodes not included in that
> application's distribution specification locally.
>
>
> Rationale:
>
> We often want to have active/standby pairs for applications
> while also having multiple instances of the application running
> on different nodes.  When nodes within the cluster are communicating
> in order to, for example, distribute mnesia tables, suddenly there
> is a potential conflict between these otherwise unrelated node pairs.
>
> Currently there is a window of time during node (re)starts where a
> conflict may occur.  This patch simply corrects this error case.
>
> The documentation is somewaht unclear as to whether the configuration
> above is legal or not.  The fact that, other than in the race condition
> noted above, this distribution does work as expected allows one to use
> the more liberal interpretation that when it says:
>
>     "All involved nodes must have the same value for distributed and
>      sync_nodes_timeout, or the behaviour of the system is undefined."
>
> Involved nodes refers to nodes involved in tthis application's distribution.
> With this patch that interpretation holds true.
>
> Tests:
>
> The existing tests are extended to support the following application
> distribution configuration:
>
>     cp1:
>        [{kernel,
>           [{sync_nodes_optional, [cp2, cp3]},
>            {distributed,
>               [{app1, [cp1, cp2, cp3]},
>                {app2, 2000, [cp1, cp2, cp3]},
>                {app_sp, 1000, [{cp1, cp2}, cp3]},"
>                {app_no, 1000, [cp1, cp2]}]}]}].
> '
>     cp2:
>        [{kernel,
>           [{sync_nodes_optional, [cp1, cp3]},
>            {distributed,
>               [{app1, [cp1, cp2, cp3]},
>                {app2, 2000, [cp1, cp2, cp3]},
>                {app_sp, 1000, [{cp1, cp2}, cp3]},"
>                {app_no, 1000, [cp1, cp2]}]}]}].
>
>     cp3:
>        [{kernel,
>           [{sync_nodes_optional, [cp1, cp2, cp4]},
>            {distributed,
>               [{app1, [cp1, cp2, cp3]},
>                {app2, 2000, [cp1, cp2, cp3]},
>                {app_sp, 1000, [{cp1, cp2}, cp3]},"
>                {app_no, 1000, [cp3, cp4]}]}]}].
>
>     cp4:
>        [{kernel,
>           [{sync_nodes_optional, [cp2]},
>            {distributed,
>                {app_no, 1000, [cp3, cp4]}]}]}].
>
> The Cp4 node is added along with the app_no application which is
> distributed in two active/standby pairs on Cp1/Cp2 and Cp3/Cp4.
> The tests check that these pairs are unaffected by the other applications'
> starts, stops, failovers and takeovers.  And that they do not affect each
> other's.
>
Hello Vance,
Could you please rebase this patch upon the current maint branch.
Thanks,

--

BR Fredrik Gustafsson
Erlang OTP Team


Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
On Fri, May 03, 2013 at 10:42:57AM +0200, Fredrik wrote:
}  Could you please rebase this patch upon the current maint branch.

Apparently my repo was messed up.  I rebuilt it and started over so
it is now a completely different commit (dee343d5c3).

Fetch here:

  git fetch git://github.com/vances/otp.git non_overlap_application_distribution

Browse here:

  https://github.com/vances/otp/commit/dee343d5c3505dba7422bf023ee11120c3c73e47


--
        -Vance

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Fredrik
On 05/03/2013 03:43 PM, Vance Shipley wrote:

> On Fri, May 03, 2013 at 10:42:57AM +0200, Fredrik wrote:
> }  Could you please rebase this patch upon the current maint branch.
>
> Apparently my repo was messed up.  I rebuilt it and started over so
> it is now a completely different commit (dee343d5c3).
>
> Fetch here:
>
>    git fetch git://github.com/vances/otp.git non_overlap_application_distribution
>
> Browse here:
>
>    https://github.com/vances/otp/commit/dee343d5c3505dba7422bf023ee11120c3c73e47
>
>
Re-fetched.
It is now in the 'pu' branch.
Thanks,

--

BR Fredrik Gustafsson
Erlang OTP Team


Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Fredrik
In reply to this post by Vance Shipley-2
On 05/03/2013 03:43 PM, Vance Shipley wrote:

> On Fri, May 03, 2013 at 10:42:57AM +0200, Fredrik wrote:
> }  Could you please rebase this patch upon the current maint branch.
>
> Apparently my repo was messed up.  I rebuilt it and started over so
> it is now a completely different commit (dee343d5c3).
>
> Fetch here:
>
>    git fetch git://github.com/vances/otp.git non_overlap_application_distribution
>
> Browse here:
>
>    https://github.com/vances/otp/commit/dee343d5c3505dba7422bf023ee11120c3c73e47
>
>
Hello Vance,
Your patch seems to fail some testcases, for certain I know that it is
failing application_SUITE:distr_changed_tc2.
And I have seen some indications that it is also is failing
application_SUITE:script_start.
Could you please look into this matter as I am removing it from pu until
further notice from you.
Thanks,

--

BR Fredrik Gustafsson
Erlang OTP Team


Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
On Tue, May 07, 2013 at 10:40:27AM +0200, Fredrik wrote:
}  Your patch seems to fail some testcases, for certain I know that it
}  is failing application_SUITE:distr_changed_tc2.
}  And I have seen some indications that it is also is failing
}  application_SUITE:script_start.
}  Could you please look into this matter as I am removing it from pu
}  until further notice from you.

Frederik,

I have pulled the pu branch, fetched my non_overlap_application_distribution
branch, built and run the application_SUITE test suite on my MacBook Pro
with the same results as before.  The only case failing is otp_3002 which
failed before my changes.

I'll look at distr_changed_tc2 and try and reason why it might fail.

Any further information you can give me on the failure you saw would help.
What environment was it run on?

--
        -Vance

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Fredrik
On 05/08/2013 03:30 PM, Vance Shipley wrote:

> On Tue, May 07, 2013 at 10:40:27AM +0200, Fredrik wrote:
> }  Your patch seems to fail some testcases, for certain I know that it
> }  is failing application_SUITE:distr_changed_tc2.
> }  And I have seen some indications that it is also is failing
> }  application_SUITE:script_start.
> }  Could you please look into this matter as I am removing it from pu
> }  until further notice from you.
>
> Frederik,
>
> I have pulled the pu branch, fetched my non_overlap_application_distribution
> branch, built and run the application_SUITE test suite on my MacBook Pro
> with the same results as before.  The only case failing is otp_3002 which
> failed before my changes.
>
> I'll look at distr_changed_tc2 and try and reason why it might fail.
>
> Any further information you can give me on the failure you saw would help.
> What environment was it run on?
>
This happened on both linux and darwin.
The testcases are not failing without your patch in pu so that was the
cause, I know that for certain.
The script_start testcase is only failing on linux though.
distr_changed_tc2 is failing with reason:

=== reason = {suite_failed,"distribution error: Cp2 [app2,app3,app6,app8,kernel,stdlib] "}


And script_start with reason:

=== reason = {suite_failed,not_valid_start_type}



--

BR Fredrik Gustafsson
Erlang OTP Team

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20130508/8bd19c16/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Siri Hansen-3
Hi Vance, I have looked at your patch and discussed with my team and here
is what we have so far:

We do think that this type of functionality is interesting. We would,
however, like to look at it in a bigger picture and possibly investigate
the functionality for distributed applications a bit more before jumping to
any conclusion. Would it be possible for you to provide us with some more
information about the exact problem that you want to solve and maybe other
use cases?

Any input and ideas from other readers of the list would of course also be
very welcome :)

While reviewing, I also found that your patch is a bit incomplete as it
does not introduce any new handling of #state.remote_started. This causes
(at least) a hanging in the following scenario:
1. Start all nodes in first group
2. On first node in first group: application:start(MyDistApp).
3. Start all nodes in second group
4. On first node in second group: application:start(MyDistApp).
5. On other node in first group: application:start(MyDistApp).
=> hangs, since #state.remote_started contains two elements and only the
first (which by chance is not the correct one) is considered.

- This is just for information, and I don't suggest that you spend a lot of
time solving this, since we don't yet know if we will accept the patch or
not.

Best regards
/siri


2013/5/8 Fredrik <fredrik>

>  On 05/08/2013 03:30 PM, Vance Shipley wrote:
>
> On Tue, May 07, 2013 at 10:40:27AM +0200, Fredrik wrote:
> }  Your patch seems to fail some testcases, for certain I know that it
> }  is failing application_SUITE:distr_changed_tc2.
> }  And I have seen some indications that it is also is failing
> }  application_SUITE:script_start.
> }  Could you please look into this matter as I am removing it from pu
> }  until further notice from you.
>
> Frederik,
>
> I have pulled the pu branch, fetched my non_overlap_application_distribution
> branch, built and run the application_SUITE test suite on my MacBook Pro
> with the same results as before.  The only case failing is otp_3002 which
> failed before my changes.
>
> I'll look at distr_changed_tc2 and try and reason why it might fail.
>
> Any further information you can give me on the failure you saw would help.
> What environment was it run on?
>
>
>  This happened on both linux and darwin.
> The testcases are not failing without your patch in pu so that was the
> cause, I know that for certain.
> The script_start testcase is only failing on linux though.
> distr_changed_tc2 is failing with reason:
>
> === reason = {suite_failed,"distribution error: Cp2 [app2,app3,app6,app8,kernel,stdlib] "}
>
>
> And script_start with reason:
>
> === reason = {suite_failed,not_valid_start_type}
>
>
>
> --
>
> BR Fredrik Gustafsson
> Erlang OTP Team
>
>
> _______________________________________________
> erlang-patches mailing list
> erlang-patches
> http://erlang.org/mailman/listinfo/erlang-patches
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20130612/8bdc1b7e/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
On Wed, Jun 12, 2013 at 05:07:38PM +0200, Siri Hansen wrote:
}  We do think that this type of functionality is interesting. We would,
}  however, like to look at it in a bigger picture and possibly investigate
}  the functionality for distributed applications a bit more before jumping to
}  any conclusion. Would it be possible for you to provide us with some more
}  information about the exact problem that you want to solve and maybe other
}  use cases?
 
The use case is simply that we run the same application on each node in
a distributed Erlang cluster and want to designate a standby node for
each as depicted below:

   +-----------+          +-----------+
   |serverA    |          |serverB    |
   | +-------+ |          | +-------+ |
   | | node1 | |          | | node2 | |
   | +-------+ |          | +-------+ |
   | +-------+ |          | +-------+ |
   | | node3 | |          | | node4 | |
   | +-------+ |          | +-------+ |
   +-----------+          +-----------+

   node1:
      [{kernel,
               [{distributed, [{app1, [node1, node2]}]}]},
                {sync_nodes_optional, [node2]},
                {sync_nodes_timeout, 5000}]}].
   node2:
      [{kernel,
               [{distributed, [{app1, [node1, node2]}]}]},
                {sync_nodes_optional, [node1]},
                {sync_nodes_timeout, 5000}]}].
   node3:
      [{kernel,
               [{distributed, [{app1, [node4, node3]}]}]},
                {sync_nodes_optional, [node4]},
                {sync_nodes_timeout, 5000}]}].
   node4:
      [{kernel,
               [{distributed, [{app1, [node4, node3]}]}]},
                {sync_nodes_optional, [node3]},
                {sync_nodes_timeout, 5000}]}].

}  While reviewing, I also found that your patch is a bit incomplete as it
}  does not introduce any new handling of #state.remote_started. This causes
}  (at least) a hanging in the following scenario:
}  1. Start all nodes in first group
}  2. On first node in first group: application:start(MyDistApp).
}  3. Start all nodes in second group
}  4. On first node in second group: application:start(MyDistApp).
}  5. On other node in first group: application:start(MyDistApp).
}  => hangs, since #state.remote_started contains two elements and only the
}  first (which by chance is not the correct one) is considered.
 
Yes, I have since discovered this issue.  I did intend to update the patch
with a permanent solution ...

}  - This is just for information, and I don't suggest that you spend a lot of
}  time solving this, since we don't yet know if we will accept the patch or
}  not.

If it can be done by simply anticipating that applications may run on
non-overlapping nodes and handling it than I sincerely hope that you
would.  For us it was not obvious at all that there was any reason that
we couldn't configure as above.  The documentation only says:

   "All involved nodes must have the same value for distributed and
    sync_nodes_timeout, or the behaviour of the system is undefined."

... which we read to say that the {App, [Node, ...]} tuple should be
consistent within those nodes.

Beyond this, hopefully simple, enhancement I plan to implement a new
distributed application controller to accomplish N+M redundancy as well.
That seems to be something which we can do without much change to OTP.
Ulf has shared his previous work on design with me, he seems to have
recognized the same requirement.  I wonder if anyone else on the list
has done any work on this sort of thing or has thoughts on requirements
or design?


--
        -Vance

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
On Thu, Jun 13, 2013 at 02:46:38PM +0530, Vance Shipley wrote:
}  The use case is simply that we run the same application on each node in
}  a distributed Erlang cluster and want to designate a standby node for
}  each as depicted below:

Siri,

For further clarification I should add that the node pairs need to
know about each other because they use distributed mnesia.  The node
pairs which may run an instance of an application each maintain a copy
of the mnesia tables which they need to run.  If serverA fails node2
will take over it's app1 instance and continue operation with the current
tables.  The problem comes in with the fact that the active nodes all
use mnesia distribution and pg2 betwen them (e.g. node & node4) as well.

--
        -Vance


Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Siri Hansen-3
Hi Vance,

we have decided that we would like to take in this functionality if the bug
I mentioned in my last mail is corrected and some more extensive tests are
added. We also need an update of the documentation, of course.

Thanks for your contribution!
/siri


2013/6/14 Vance Shipley <vances>

> On Thu, Jun 13, 2013 at 02:46:38PM +0530, Vance Shipley wrote:
> }  The use case is simply that we run the same application on each node in
> }  a distributed Erlang cluster and want to designate a standby node for
> }  each as depicted below:
>
> Siri,
>
> For further clarification I should add that the node pairs need to
> know about each other because they use distributed mnesia.  The node
> pairs which may run an instance of an application each maintain a copy
> of the mnesia tables which they need to run.  If serverA fails node2
> will take over it's app1 instance and continue operation with the current
> tables.  The problem comes in with the fact that the active nodes all
> use mnesia distribution and pg2 betwen them (e.g. node & node4) as well.
>
> --
>         -Vance
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20130619/5849503d/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Henrik Nord-3
Ping!



On 2013-06-19 16:01, Siri Hansen wrote:

> Hi Vance,
>
> we have decided that we would like to take in this functionality if
> the bug I mentioned in my last mail is corrected and some more
> extensive tests are added. We also need an update of the
> documentation, of course.
>
> Thanks for your contribution!
> /siri
>
>
> 2013/6/14 Vance Shipley <vances <mailto:vances>>
>
>     On Thu, Jun 13, 2013 at 02:46:38PM +0530, Vance Shipley wrote:
>     }  The use case is simply that we run the same application on each
>     node in
>     }  a distributed Erlang cluster and want to designate a standby
>     node for
>     }  each as depicted below:
>
>     Siri,
>
>     For further clarification I should add that the node pairs need to
>     know about each other because they use distributed mnesia.  The node
>     pairs which may run an instance of an application each maintain a copy
>     of the mnesia tables which they need to run.  If serverA fails node2
>     will take over it's app1 instance and continue operation with the
>     current
>     tables.  The problem comes in with the fact that the active nodes all
>     use mnesia distribution and pg2 betwen them (e.g. node & node4) as
>     well.
>
>     --
>             -Vance
>
>
>
>
> _______________________________________________
> erlang-patches mailing list
> erlang-patches
> http://erlang.org/mailman/listinfo/erlang-patches

--
/Henrik Nord Erlang/OTP

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-patches/attachments/20140217/f8bb82f1/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Non-overlapping Application Distribution Node Sets

Vance Shipley-2
Henrik,

Yes, I haven't forgotten about this.  It will get the required attention
in the next few weeks.

--
        -Vance

On Mon, Feb 17, 2014 at 10:00:06AM +0100, Henrik Nord wrote:
}  Ping!
}  
}  
}  
}  On 2013-06-19 16:01, Siri Hansen wrote:
}  >Hi Vance,
}  >
}  >we have decided that we would like to take in this functionality
}  >if the bug I mentioned in my last mail is corrected and some more
}  >extensive tests are added. We also need an update of the
}  >documentation, of course.
}  >
}  >Thanks for your contribution!
}  >/siri
}  >
}  >
}  >2013/6/14 Vance Shipley <vances <mailto:vances>>
}  >
}  >    On Thu, Jun 13, 2013 at 02:46:38PM +0530, Vance Shipley wrote:
}  >    }  The use case is simply that we run the same application on each
}  >    node in
}  >    }  a distributed Erlang cluster and want to designate a standby
}  >    node for
}  >    }  each as depicted below:
}  >
}  >    Siri,
}  >
}  >    For further clarification I should add that the node pairs need to
}  >    know about each other because they use distributed mnesia.  The node
}  >    pairs which may run an instance of an application each maintain a copy
}  >    of the mnesia tables which they need to run.  If serverA fails node2
}  >    will take over it's app1 instance and continue operation with the
}  >    current
}  >    tables.  The problem comes in with the fact that the active nodes all
}  >    use mnesia distribution and pg2 betwen them (e.g. node & node4) as
}  >    well.
}  >
}  >    --
}  >            -Vance
}  >
}  >
}  >
}  >
}  >_______________________________________________
}  >erlang-patches mailing list
}  >erlang-patches
}  >http://erlang.org/mailman/listinfo/erlang-patches
}  
}  --
}  /Henrik Nord Erlang/OTP
}  

--
        -Vance

Reply | Threaded
Open this post in threaded view
|

Re: Non-overlapping Application Distribution Node Sets

Henrik Nord-2
Hi again!

I might have missed some kind of update to this?
Or are we in really long weeks mode =)

I would suggest you open a Pull Request on github for this patch
when(if) you decide to work on it again.

Thank you


On 2014-02-18 09:39, Vance Shipley wrote:
> Henrik,
>
> Yes, I haven't forgotten about this.  It will get the required attention
> in the next few weeks.
>

--
/Henrik Nord Erlang/OTP

_______________________________________________
erlang-patches mailing list
[hidden email]
http://erlang.org/mailman/listinfo/erlang-patches