[Twg] Protocol interoperation issues

[Eric Barton]

Lustre support for transparent rolling upgrade has been an assumption for some
time now.  However it comes at a cost.  I'd like to explain how it is
implemented so that we can all share a better understanding of this cost and
let that guide OpenSFS development decisions.

* LNET protocol version negotiation

  LNET has 2 levels of protocol - a generic LNET protocol is layered over an
  LND-specific protocol.  LNDs exchange protocol version on the first
  communication between any pair of NIDs and use the earlier version.  The
  generic LNET protocol version negotiation is implicit in the LND version
  negotiation.

* LNET rolling upgrade

  Peer-to-peer communication state in LNET and its LNDs is discarded on any
  error and re-established on subsequent communication attempts.  This allows
  LNET to be restarted or upgraded at any time, at the potential cost of a few
  dropped communications.  Protocols layered over LNET (e.g. Lustre) must be
  coded to allow for this.

  LND protocol changes require at least 1 interim version of the LND to support
  both protocol versions.  All LND instances must be upgraded to one of these
  interim versions before any can be upgraded to subsequent versions that
  remove support for the old protocol version.

  Changes to the LNET protocol affect all LNDs - i.e. all LNDs must be coded to
  support both old and new protocol versions.

* Lustre protocol version negotiation

  Lustre peers negotiate protocol versions on connection establishment.
  Protocol features supported by either peer are represented using a bitmap so
  that only features supported by both peers are used.

* Lustre Rolling Upgrade

  Lustre's recovery protocols make server restart/failover transparent to
  client applications.  This can be exploited to allow rolling upgrades - where
  the server is upgraded before it is restarted.

  Much of recovery is implemented simply by replaying RPCs that hadn't yet been
  committed to disk by the previous instance of the server.  This is a problem
  if there is a major server version change because the client's replay RPCs
  are now in an incompatible format.

  The solution to this is "Simplified Interoperation" which will provide an
  administrative server shutdown feature to force clients to minimize cached
  state.  This reduces the set of RPCs required for recovery to a tiny fraction
  which are regenerated when the new server's protocol version is known and
  avoids the need to replay uncommitted RPCs entirely.

  1.8/2.x upgrade

  The Lustre 1.8.5 client can talk the 2.x protocol, therefore servers may be
  upgraded to 2.x without requiring clients to upgrade immediately.  Until
  simplified interoperation is implemented the upgrade process cannot be
  transparent to clients.  They will be evicted and therefore applications
  running on them may see I/O errors during the server upgrade.

The biggest protocol change on the horizon is to support IPv6.  This cannot be
put off indefinitely, so when it comes, should we try to support transparent
rolling upgrade for it, or require a site-wide upgrade?

* IPv6

  Support for IPv6 requires a change in the NID format to accommodate a 128bit
  IPv6 address in the address-within-network field.  This affects all protocol
  levels; LNDs, LNET and Lustre.

  Rolling upgrade will require versions of LNET (and all its LNDs) and Lustre
  that can communicate in both small and large NID LND/LNET/Lustre protocol
  versions.  The version of Lustre will also have to be able to read and
  convert the small NID on-disk representation.  Transparent rolling upgrade
  will require Simplified Interoperation support.

  However, at the cost of a 1-time site-wide upgrade most of this cost can be
  avoided.  If IPv6 protocol version interoperation is not required, changing
  the protocols (LNET and Lustre) to use a large NID is relatively trivial since
  all that remains to be done after recompiling with the new NID format definition
  is to allow Lustre to read and convert any use of the binary small NID
  representation on disk.

          Cheers,
                   Eric

Eric Barton
CTO Whamcloud, Inc.
Tel: +44 (117) 330 1575
Mob: +44 (7920) 797 273