[Twg] TWG meeting minutes for 2012-03-15
John Carrier
carrier at cray.com
Mon Mar 19 05:54:12 UTC 2012
Attached are the meeting minutes from last week's TWG call. This meeting continued the discussion of features that the TWG should consider as we update our requirements and roadmaps document. The following is a list of features we have discussed this year:
- single client performance improvements
- migration (OST rebalancing, dynamic layout, async mirroring, etc)
- small I/O performance improvements
- dynamic LNET configuration
- POSIX extensions : file sets
- POSIX extensions : scalable opens
- health networks
In addition we reviewed the impact of EMC's efforts to move the Lustre client into the Linux kernel. The community needs to come to a consensus on the importance of this effort versus maintaining OS interoperability without forking the code.
We will continue this discussion this week:
Date/Time : 03/22/2012 @ 9:30a PDT/12:30p EDT
Dial: 1-715-726-4994, 1-866-304-8294
ID & Password: 72090
If there are no new features to discuss, we need to begin converting features to requirements in preparation for making a recommendation to the OpenSFS board.
Note that minutes from past meetings are available in the archives of both the twg and discuss lists at http://lists.opensfs.org.
Please post corrections/questions/comments to discuss at lists.opensfs.org<mailto:discuss at lists.opensfs.org>.
--jc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120319/5ee2410c/attachment.html>
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 2012-03-15
Concall: start : 9:30a PT, adjourn : 10:45a PT
Next meeting: Thursday, 03/22/2012 @ 9:30a PDT/12:30p EDT
Dial-in numbers are 1-715-726-4994 or 1-866-304-8294
Meeting ID and password are 72090
Attending:
Name Organization email
------------------- ------------- ---------------------------------
Diego Moreno Bull Diego.Moreno-Lazaro at bull.net
Cory Spitz Cray spitzcor at cray.com
John Carrier Cray carrier at cray.com
Alex Kulyavtsev FNAL aik at fnal.gov
Joshua Walgenbach IU jjw at indiana.edu
Norm Morse OpenSFS normanrmorse at gmail.com
Dave Dillow ORNL dillowda at ornl.gov
Eric Barton Whamcloud eeb at whamcloud.com
Andreas Dilger Whamcloud adilger at whamcloud.com
Nathan Rutman Xyratex nathan_rutman at xyratex.com
Agenda:
continue requirements discussion for 2013 roadmap
Notes outline
corrections
requirements
- single client performance improvements
- migration / replication
- dynamic LNET configuration
- POSIX extensions : file sets
- POSIX extensions : scalable opens
- health networks
EMC to submit client upstream to kernel.org
corrections:
Dave was asked to make the following correction: the test cluster
that OpenSFS is placing at IU will be used for verifying the
functionality of SMP affinity and the online lfsck not just the
distributed namespace code.
Requirements:
Last week we made progress identifying the features and requirements
that have been in the works for some time (eg NRS, HSM) do not need
any OpenSFS funding to land. Xyratex is moving NRS forward and has
started large scale testing with different policies. CEA has HSM in
hand with landings planned for 2.2, 2.3, and 2.4
New features we have discussed in the last few weeks that remain on
the list are:
- single client performance improvements
- migration / replication
- dynamic LNET configuration
* single client performance improvements
Whamcloud has an experimental option to increase the number of
RPCs from a single client that needs funding to complete testing
and integration.
* migration and replication
We spent a good bit of time defining terms:
Replication is the process of copying data from one file system to
another. One-way replication is possible today with Lustre
rsync. There is no coherency between the remote copy back to
the original, but remote replication is sufficient as nearline
backup for disaster recovery.
The current rsync walks the namespace to look for changes that
require files to be copied to the remote destination. Xyratex
has developed a tool that does the copy more intelligently
using the change logs to copy only files that have changed.
Migration moves file objects within the same file system. It
relies on the layout lock feature added to Lustre for the HSM
interface to ensure that object migration maintains file
coherency. The current layout lock implementation will require
work to manage files that are not at rest.
There are several use cases described that are driving the need
for dynamic layous and file migration:
- OST rebalancing
move objects between OSTs to more evenly distribute free
space among the OSTs; distribute objects to new OSTs
added to expand the file system.
- OST emptying
move objects off of an OST before it is replaced or
removed from the file system
- dynamic layouts
allow file layouts to change as the file grows or access
patterns change
- async mirroring
create multiple copies of a file within the same file
system namespace (ala Hadoop). This use case requires
addition consistency checks to enable real-time
mirroring.
There were several architectural changes mentioned during this
discussion.
- client aware layout changes
currently the client is unware of layout changes
and relies on cascading timeouts to acknowledge
layout changes. need to make the client aware of the
changes as they happen.
- persistent storage pools
Andreas calls this making storage pools first class
citizens. The current OST pool definition is an
ephemeral name for a collection of OSTs. The pool needs
to be peristent to enable, for example, quotas by pool.
- unified storage target
Stop thinkg of storage as MDTs or OSTs. Intead think of
storage in terms of performance for different workloads.
Improve small I/O performance by using storage tuned for
this access pattern. Then restripe the file to different
storage as the file gets larger. (Eric mentioned the
"giga+" concept, but I'm not sure how Giga+ applies.)
- dynamic layout for subset of a file
CEA will restore a part of a file, rather than the
whole, from an archive to speed access to the critical
data.
Migration will involve a number of inter-related features. Instead
of stating specific features, we should identify the user
requirements. For example, a unified storage target might be the
feature tracked during development, but it could just as well be
motivated by improving small file performance as facilitating
dynamic layouts for migration.
* dynamic LNET configuration
The goal is to make LNET configuration more like IP configuration
using admin commands to add/remove interfaces dynamically. Current
routes are static and requires LNET (ie the file system) to
restart to capture configuration changes.
New features mentioned during this meeting:
* POSIX extensions : file sets
Alex mentioned the need for file sets to ease migration of files
from tape to disk. GPFS has this today. Eric implied that this is
a new data structure, not just a new tag on the file namespace. If
done right, the set will appear in the directory and quotas can be
allocated on the set not just the component files. Andreas
thought changing a file set via single command, rather than
traversing the directory tree could lead to snapshots.
* POSIX extensions : scalable opens
Andreas said the kernel just got an "open by handle" system call.
It now needs to get wired into Lustre.
* health networks
We discussed this feature last year as a means of providing lower
latency fault detection and improved error handling. It is a
scalability feature that by providing an active, deterministic
mechanism for communicating system status will avoid the sequence
of cascading timeouts that limits Lustre at scale. Cray has found
that the current timeout mechanism using pings is very difficult
to tune at scale.
Other Business : EMC to put client into upstream kernel
Andreas sent an email to the lustre-discuss and twg reflectors
describing EMC's efforts to move the Lustre client upstream into the
Linux kernel in order to reduce Lustre's barrier to entry. We had a
lively discussion about the impact this change would have on keeping
the client interoperable with other operating systems.
Putting the client into the kernel drew analogies with OFED. It is
significant to note that the windows OFA stack is completely
different from the Linux OFA stack (in this respect, OFED has two
separate stacks). Lustre currently has a single client and a
portability layer to map the client to different operating systems.
Removing Lustre's portability layer (libcfs) will negatively impact
the Lustre client on MacOS. The windows native client would also be
affected, but that code is tied up at Oracle. Unless the Lustre is
being actively developed on those platforms, it is hard to convince
the Linux kernel to accept portability layers.
We discussed using liblustre on FUSE to create a compatbility layer
for the MAC and Windows clients, but performance under FUSE is not
optimal. This would cripple Lustre on these OSs.
Including the client in the kernel will reduce the barrier of entry
to use Lustre. It will also clean up the client code and remove
accumulated cruft. On the other hand, in addition to the loss of OS
interoperability, such a change will increase the support window for
any given Lustre version since a given kernel version is available on
the market longer.
Whamcloud is seeking community feedback on what parts of the client
code people are using. Can EMC pull-out old code (such as liblustre)
to make a linux client acceptable to the kernel?
The goal will be to keep the upstream client in sync with the
upstream tree. Once the code lands, however, it is not clear who
will be the upstream maintainer. It is a concern that changes made
for a kernel landing won't get pushed back to the Lustre tree.
[meeting adjourned : 10:45a PT]
More information about the Twg
mailing list