[Twg] TWG meeting minutes for 2012-03-29

Wed Apr 4 07:09:31 UTC 2012

Attached are the minutes from our 3/29/2012 meeting.   We had a few new requirements introduced this call.  Please take a look at the notes and post corrections/questions/comments to discuss at lists.opensfs.org<mailto:discuss at lists.opensfs.org>.

Our next meeting is this Thursday, 4/5/2012 @ 9:30a PDT/12:30p EDT
   Dial-in numbers are 1-715-726-4994 or 1-866-304-8294
   Meeting ID and password are 72090

Thanks,

--jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120404/8a353c36/attachment.html>
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 2012-03-29

Concall: start : 9:30a PT, adjourn : 10:30a PT

Next meeting: Thursday, 04/05/2012 @ 9:30a PDT/12:30p EDT
   Dial-in numbers are 1-715-726-4994 or 1-866-304-8294
   Meeting ID and password are 72090

Attending: 

   Name                 Organization   email
   -----------------    -------------- ----------------------------
   Diego Moreno         Bull           Diego.Moreno-Lazaro at bull.net
   Cory Spitz           Cray           spitzcor at cray.com
   John Carrier         Cray           carrier at cray.com
   Alex Kulyavtsev      FNAL           aik at fnal.gov
   Joshua Walgenbach    IU             jjw at indiana.edu
   Chris Morrone        LLNL           morrone2 at llnl.gov
   Shay Seager          OpenSFS        shay at opensfs.org
   Eric Barton          Whamcloud      eeb at whamcloud.com
   Bryon Neitzel        Whamcloud      bryon at whamcloud.com

Agenda:
   discuss requirements

Status:
   These are the requirements we've accumulated so far this year:
      - improve single client performance 
      - improve storage pool management
                to address :        OST rebalancing
                                    dynamic layout
                                    async mirroring, etc
      - dynamic LNET configuration
      - extend POSIX for Lustre access patterns
                 to address :       file sets
                                    scalable opens
      - reduce Lustre's dependence on timeouts
                  to address :      health networks

LUG F2F:
   At the end of the meeting we discussed the process for getting a
   requirements list completed by our F2F meeting at LUG (4/25). We
   agreed that meeting weekly is that right way to move forward so that
   we complete the process by late before LUG.  John and Dave to update
   the requirements doc with new features.

   Xyratex, Cray, LLNL, Whamcloud planning to attend the F2F meetings on
   Wednesday.  A number of us will also be attending the benchmarking WG
   on Sunday.  Please remember that DDN & Whamcloud are having a
   reception Sunday night before LUG starts.

Discussion:

* Shared file performance
   - Henry Newman (via email)
   - N-1 I/O is much less than N-N

* Administrative shutdown
   - EricB raised this issue
   - Lustre needs a safe server shutdown to ensure that clients flush
      state to disk.
   - such a capability will be necessary for transparent server version
      upgrades

* Improve configuration robustness
   - Need to make file system configuration robust in the face failures
   - ChrisM related problems he recently had at LLNL trying to add OSTs
      to an existing file system.  The failure to make the addition
      required that he hand-edit the MGS log files.
   - EricB said we should anticipate similar errors with DNE when we
      expand the number of MDTs in the namespace. For example, consider
      what might happen when allowing directories on MDTs not
      rooted to MDT0.  
   - ChrisM believes there are timebombs remaining in the code and we'll
      need to pay the technical debt to remove them.
   - We also discussed how Lustre could benefit from dynamic
      configuration.  For example, imperative recovery introduces the
      need for dynamic registration with MGS.  We could use this to
      remove static configuration and use the registration to define the
      config.

* code stability  
   - With all the architectural changes landing on Lustre in the next
      year (HSM, DNE, OSD, etc), there is concern we may be changing too
      much and should spend time on clean-up and stability
   - We also know that the layout locks introduce with the HSM will need
      to be reworked so that they can be re-used for OST rebalancing and
      other migration tasks for files not at rest.
   - Lustre reliance on timeouts also contribute significantly to system
      instability or, at least, the perception that they system is
      unstable because of periodic pauses. The health network should be
      a high priority to remove timeouts and make client/server
      interactions more deterministic. Eric's current thinking is that a
      spanning tree with a bullet-proof root would allow resilient,
      scalable distribution of global information.   This project will
      require a formal architecture and prototype before completion.
   - ChrisM mentioned that /proc doesn't have sufficient locks.  Cat'ing
      /proc can crash the node.  He isn't sure where the problem is,
      but bugs like this make Lustre appear to be unstable.

* LNET credit configuration
   - ChrisM suggested that we need to revisit LNET credits.  His
      experience is that transmit credits complicate LNET traffic
      between networks.  He thinks LNET router nodes should never have
      transmit credit limits since the limited resource is the buffers.
      The problem is espeically pronounced when there are multiple
      network levels.
   - Eric suggested having high and low priority message queues.  This
      could be part of a project to improve network robustness.  His
      original design used multiple routers to guarantee connectivity,
      but performance tanks when there are large numbers of routers.
   - Improving this aspect of the LNET network interface is separate
      from and orthogonal to the previous discussions about LNET (ie 
      dynamic LNET configuration and LNET channel bonding). Chris thinks
      that improving LNET robustness may require a wire protocol change.
   - We also should identify poorly behaving routers so that they can be
      removed from the network to improve performance.

* other performance improvements
   - As mentioned, there are a number of MDS improvements in the works:
      pdirops     - single directory
      smp scaling - fatter servers
      DNE         - horizontal scalability
      stat ahead / glimpse ahead
   - Discussed but not yet implemented is size on metadata which has an
      issue with evictions caused when caching data on OSTs when evicted
      from the MDS. 

* storage pool management : standard target
   - A standard target would remove the distinction between the MDT and
      OSTs. One outcome is that the same target could be used by both
      the MDS and OSS.  As a result, could put small directories on an
      MDT to improve performance since I/O could complete in a single
      roundtrip.
   - Putting small I/O on the MDT would improve performance and dynamic
      layout would allow migration from MDT to OST.
   - Changing striping for the MDT would create giga+ sytle directories
   - Eric says he has a 2-pater for storage pool management describes
      the benefits, components and phases for the work.  TWG could use
      this as basis for its report to the OpenSFS board.

adjourn: 10:30a PT