The Erlang way - dynamic upgrade of a server and UBF extensions

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

The Erlang way - dynamic upgrade of a server and UBF extensions

Joe Williams-2

  I  want to  add some  things to  UBF so  we can  make fault-tolerent
systems. This can  be done with only a little change  to UBF. I havn't
thought out all the details so I thought I'd try the idea out first.

  All comments are welcome.

  The UBF  extensions provide an alternative solution  to the problem
of  dynamically upgrading  or migrating  a server  or  of dynamically
upgrading software.

  Firstly a bit of philosophy

  The Erlang way
  ==============

  1) everything is a process
  2) processes communicate by message passing
  3) processes obey protocols
  4) errors are handled non locally

  Should I add more points?

  UBF
  ===

  UBF describes (specifies) 3) above
   
  Extensions to UBF
  =================

  A) ? (a undefined type) - to denote absence of an argument

  B) The protocol versioning tag

  C) The next IP word.

  A) Undefined Type
  =================

  Just use ? in UBF(A) to denote a missing value (easy)

  B) + C)
  =======

  Client UBF packets to a server should begin

  +---------------+---------------------+
  | VersionNumber | ... rest of packet  |
  +---------------+---------------------+

  And server replies should begin

  +---------------+---------+---------------------+
  | VersionNumber | NextIP  | ... rest of packet  |
  +---------------+---------+---------------------+

  B) Protocol versioning
  ======================

  The version number is an integer 1 2 3 ... etc.

  This is to allow dynamic upgrade of services
 
  I have not thought out all the details but the idea is this:

  Assume that a client has versions 1,2,3,6,8,10,11 of some software
  The server speaks versions 1,2,3,4,5,9,10,14

  At the start of the session we enter "protocol negotiation mode"

  Both sides agree that version 10 is the highest common version of the
protocol that they both understand.

  A session starts at level 10.

  The client S/W crashes - this is detected by the client SW
  T
  The next message it sends to the server is a version 8 message

  The server realizes that something has gone wrong - it cannot handle
version 8 messages - so they go back into protocol negotiation mode.
They agree on version 3 and continue.

  This should allow "dynamic introduction of new services" with fallback to
previous versions if things go wrong.

 C) Next IP
 ==========

 This is to allow dynamic migration of services.

  There are two basic approaches  to making things reliable. I'd like
to suggest a third.

 The two common methods are:

 1) Fixed server IP

 The server has a fixed IP. To make things fault-tolerant the fixed IP is the
IP of a switch - the switch dispatches the request to a back-end.

  (This  is  the common  cluster  solution -  the  switch  is big  and
expensive and can act as a front-end to many back-end servers)
 
 2) Two fixed IP's

 This is the DNS solution. My machine has two addresses for DNS - I try the
first, if it is broken I try the second.

 I propose a third method: Each packet contains a NextIP field. This is
the address where the Next message should be sent.

 Example. I have a server at 127.45.67.223

 a) The client connects to 127.45.67.223

 b) Client and server exchange messages

 The server becomes heavily loaded, or, the operator wants to take the
server out of operation.

 At some point the client receives a rpc reply with a NEW IP address
(not 127.45.67.223) but something different.

 The client closes the connection to 127.45.67.223 connects to the new
address and carries on.

 I think that B) + C) above would solve a lot of problems.

 Comments?

 /Joe





Reply | Threaded
Open this post in threaded view
|

The Erlang way - dynamic upgrade of a server and UBF extensions

Erik Pearson-2
Hi Joe,

A. great
B. don't know enough about this problem domain.
C. Hmm, I'm worried about what happens when the server experiences
difficulties that degrade the quality of service to the point it has
problems even telling clients to reroute to the next ip address.
Perhaps, tying into B, there could be an explicit set of configuration
negotiations between client and server (perhaps relating to versions?),
such that a  server can tell a client what the fallback measures are
for degradation of services from that server -- e.g. if latency falls
below .25 sec, then here is a set of other ip addresses to try, etc.
This policy could be dynamically adjusted, of course, and having it be
persistent would be great so it doesn't have be be renegotiated each
time (the version neg. protocol would handle that...). How would this
fit into the current client->server model? Perhaps "out of band"
messages could be sent as replies to clients -- this would ensure
serialization as well (e.g. if all clients were servers and servers
clients, a server could broadcast reconfiguration information, but
there would be no guarantee that clients would see it before blasting
through another few messages -- if a server is trying to calm a traffic
storm, it might be a good idea to get the immediate attention of all
clients who are actually interacting with the server...)

Another area that has concerned me is error signaling. Although this
could fall into the application realm, experience informs me that at
least a framework for the formatting of error messages (e.g. predefined
types) would be very helpful. A communication mode (ala out of band
messages, or rpc style.) I don't know if this would be best as a hard
art of UBF(C), or perhaps a set of "recommended practices".

Erik.

On Tuesday, April 29, 2003, at 01:44 AM, Joe Armstrong wrote:

>
>   I  want to  add some  things to  UBF so  we can  make fault-tolerent
> systems. This can  be done with only a little change  to UBF. I havn't
> thought out all the details so I thought I'd try the idea out first.
>
>   All comments are welcome.
>
>   The UBF  extensions provide an alternative solution  to the problem
> of  dynamically upgrading  or migrating  a server  or  of dynamically
> upgrading software.
>
>   Firstly a bit of philosophy
>
>   The Erlang way
>   ==============
>
>   1) everything is a process
>   2) processes communicate by message passing
>   3) processes obey protocols
>   4) errors are handled non locally
>
>   Should I add more points?
>
>   UBF
>   ===
>
>   UBF describes (specifies) 3) above
>
>   Extensions to UBF
>   =================
>
>   A) ? (a undefined type) - to denote absence of an argument
>
>   B) The protocol versioning tag
>
>   C) The next IP word.
>
>   A) Undefined Type
>   =================
>
>   Just use ? in UBF(A) to denote a missing value (easy)
>
>   B) + C)
>   =======
>
>   Client UBF packets to a server should begin
>
>   +---------------+---------------------+
>   | VersionNumber | ... rest of packet  |
>   +---------------+---------------------+
>
>   And server replies should begin
>
>   +---------------+---------+---------------------+
>   | VersionNumber | NextIP  | ... rest of packet  |
>   +---------------+---------+---------------------+
>
>   B) Protocol versioning
>   ======================
>
>   The version number is an integer 1 2 3 ... etc.
>
>   This is to allow dynamic upgrade of services
>
>   I have not thought out all the details but the idea is this:
>
>   Assume that a client has versions 1,2,3,6,8,10,11 of some software
>   The server speaks versions 1,2,3,4,5,9,10,14
>
>   At the start of the session we enter "protocol negotiation mode"
>
>   Both sides agree that version 10 is the highest common version of the
> protocol that they both understand.
>
>   A session starts at level 10.
>
>   The client S/W crashes - this is detected by the client SW
>   T
>   The next message it sends to the server is a version 8 message
>
>   The server realizes that something has gone wrong - it cannot handle
> version 8 messages - so they go back into protocol negotiation mode.
> They agree on version 3 and continue.
>
>   This should allow "dynamic introduction of new services" with
> fallback to
> previous versions if things go wrong.
>
>  C) Next IP
>  ==========
>
>  This is to allow dynamic migration of services.
>
>   There are two basic approaches  to making things reliable. I'd like
> to suggest a third.
>
>  The two common methods are:
>
>  1) Fixed server IP
>
>  The server has a fixed IP. To make things fault-tolerant the fixed IP
> is the
> IP of a switch - the switch dispatches the request to a back-end.
>
>   (This  is  the common  cluster  solution -  the  switch  is big  and
> expensive and can act as a front-end to many back-end servers)
>
>  2) Two fixed IP's
>
>  This is the DNS solution. My machine has two addresses for DNS - I
> try the
> first, if it is broken I try the second.
>
>  I propose a third method: Each packet contains a NextIP field. This is
> the address where the Next message should be sent.
>
>  Example. I have a server at 127.45.67.223
>
>  a) The client connects to 127.45.67.223
>
>  b) Client and server exchange messages
>
>  The server becomes heavily loaded, or, the operator wants to take the
> server out of operation.
>
>  At some point the client receives a rpc reply with a NEW IP address
> (not 127.45.67.223) but something different.
>
>  The client closes the connection to 127.45.67.223 connects to the new
> address and carries on.
>
>  I think that B) + C) above would solve a lot of problems.
>
>  Comments?
>
>  /Joe
>
>
>
>
Erik Pearson
Adaptations
desk +1 510 527 5437
cell +1 510 517 3122