[Twg] TWG meeting minutes for 2012-03-01
John Carrier
carrier at cray.com
Thu Mar 8 14:32:01 UTC 2012
Attached are the meeting minutes from last week's TWG call. We spent this meeting introducing features that the TWG should consider as we update our requirements and roadmaps document.
We will continue this discussion this week (3/8).
Dial: 1-715-726-4994, 1-866-304-8294
ID & Password: 72090
Please post corrections/questions/comments to discuss at lists.opensfs.org<mailto:discuss at lists.opensfs.org>
My apologies for getting these out so late.
--jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120308/f4d43b2a/attachment.html>
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 2012-03-01
Concall: start : 9:30a PT, adjourn : 10:44a PT
Next meeting: Thursday, 03/08/2011 @ 9:30a PDT/12:30p EDT
Dial-in numbers are 1-715-726-4994 or 1-866-304-8294
Meeting ID and password are 72090
Attending :
Name Organization email
----------------- -------------- ----------------------------
Diego Moreno Bull Diego.Moreno-Lazaro at bull.net
Cory Spitz Cray spitzcor at cray.com
John Carrier Cray carrier at cray.com
Alex Kulyavtsev FNAL aik at fnal.gov
Gene Oleynik FNAL oleynik at fnal.gov
Joshua Walgenbach IU jjw at indiana.edu
Chris Morrone LLNL morrone2 at llnl.gov
Norm Morse OpenSFS normanrmorse at gmail.com
Shay Seager OpenSFS shay at opensfs.org
Dave Dillow ORNL dillowda at ornl.gov
Andreas Dilger Whamcloud adilger at whamcloud.com
Agenda
- Requirements & Roadmap
* update community requirements
* identify features in need of development
* funneling projects outside OpenSFS into CDWG
We started discussing the list of requirements the TWG gathered a year
ago (http://goo.gl/cZSWG) [The topic list is copied at the end of these
minutes]. We quickly focused on the new items below.
= single client performance improvements
DaveD said that single client gets CPU bound which limits bandwidth
to 500 MB/s.
AndreasD said the new multi-thread RPC code, which has been funded by
ORNL and LLNL, improves single threaded reader/writer performance.
The new code should land in Lustre 2.2.
ChrisM said Livermore is seeing 2x improvements with the new code
(1000 MB/s using 8 threads). The CPU is not limited and the
bottleneck beyond GB/s is elsewhere.
ORNL's point is still valid: there is a bottleneck limiting single
client performance. Though CPU affinity for the MDS may propagate
to the clients, the client needs to improve SMP scalability and
efficiency.
= improved MDS rates
JohnC asked if the debug feature Whamcloud uses to increase the MDS
rate for testing could be used to improve performance by
increasing the number of MDS RPCs in flight.
Andreas said the debug symbol he mentioned last week is useful for
testing but breaks recovery. Nonetheless, there is a need to
increase the number of file-modifying RPCs in flight for DNE,
which will must support multiple threads on a single client and
many RPCs in flight from one MDS to another. This feature will
appear as a debug patch in 2.2. If testing with the patch works,
then there is work to deliver the feature to the client.
= online data migration
Andreas suggested that this feature was needed to empty OSTs,
rebalance space between OSTs and to rebalance between OST pools.
The first phase is tied to the HSM since a migration tool would
need layout locks, a policy engine and the IOCTL to tell MDS to
change the layout and flush the client cache.
Apparently, the layout locks in the current implementation of the
HSM are sufficient for quiescent files, but not for active files.
Hence, online data migration will require fixing layout locks in
the HSM first.
DiegoM said that the HSM is fully operational, but not robust. He
said Bull seems to be the only one interested in testing and the
code needs more groups using it to make progress. OpenSFS could
ask CEA how OpenSFS could help.
AlexK said FNAL was very interested in data migration. He is
interested in inactive files. Andreas said lfs_migrate already
does this and would need a layout lock patch. The problem is that
the script is active all the time and today has a performance
impact.
Fixing the layout locks is a first step to migration, replication,
and other enterprise features. For example, a policy manager
could move copies to other sites after the I/O quiesces using a
similar mechanism to migration.
John asked if the layout locks and migration could be used for
snapshots. Andreas thought this needs to be supported in the
backend file system. ldiskfs can't do it, but ZFS can. ChrisM
said LLNL is not yet looking at this ZFS feature in their work.
= improved security
Alex said his site needs kerberos and x509. The open science grid
needs x509 for data transfers.
JoshW described the work IU plans to do to improve security for the
LustreWAN project. He agreed to talk to Alex on how x509 could be
done within the new infrastructure.
= dynamic LNET configuration
Andreas introduced the idea of providing dynamic LNET configuration
so that changes to the LNET configuration would not require
stopping the file system for the changes to be propagated. The
group enthusiastically supported any attempt to remove LNET's
current dependence on modprobe.
The configuration also has to allow flexibility in some of the LNET
parameters. Cory mentioned that tuning peer credites on o2iblnd
has to agree across all LNET tiers. He has found that the only
way to fix the mismatch is to shutdown all clusters.
ORNL currently uses scripts to modify lNET configuration by hand.
Dave said he thinks this effort would have to fix ptlrpc to remove
stale entries. He thinks this is a long-term solution, but it
wouldn't really help him at ORNL.
Andreas said the effort for dynamic LNET config isn't gigantic, but
it would touch lots of code (LNET, ptlrpc)
At this point, we had run over our scheduled meeting time. The following
items were presented without much discussion:
= IPv6
This item was on our list last year. Andreas said adding support
for IPv6 will change the network protocol so the impact is more
involved than just changing internal data structures. He
suggested a phased approach that could make the feature available
in the code base, but would not be used until a site upgraded to
IPv6. Andreas said the lack of IPv6 will hit WAN users first.
= Network Request Scheduler (NRS)
The core infrastructure of NRS has been prototyped and Xyratex has
done some work recently. What is needed to complete the project
and have it land in the canonical tree?
= complex file layouts
Andreas suggested adding support to allow different file layouts
for different file types (eg HDF5). A simple example would let a
file start with one stripe definition then move to different
striping after the file grew.
Dave ended the meeting by asking the TWG to consider projects that
affected Lustre's long term roadmap. Should OpenSFS fund research
projects?
-------------------------
OpenSFS Requirements (3/22/2011)
* Performance Requirements
- Metadata server performance [FUNDED]
- Metadata server scalability [FUNDED]
- Single file performance
- Quality of service
- Locality and scalability
- LNET channel bonding
* Foundational Requirements
- Support for alternate backend file systems [FUNDED]
- Backend storage investigation
- Scalable fault management
* Manageability and Administrative Requirements
- Better support for newer kernels
- Improved configuration of Lustre
- Allowing for controlled partial-system maintenance
- Balancing storage use
- Adaptive storage layout
- Arbitrary OST assignment
- Better userspace tools
- File system consistency checks [FUNDED]
- Snapshots
- User Identity Mapping [FUNDED]
* Application Interface Requirements
- Improved storage semantics/interfaces
- Better user tool API
* Other Requirements
- Varying page-sizes
- Mixed endian support
- IPv6
More information about the Twg
mailing list