[Twg] meeting minutes fro 2011-03-10
John Carrier
carrier at cray.com
Wed Mar 16 21:12:51 UTC 2011
Attached are my summary notes from last week's meeting. Dave and I are still working on our recommendation for the board for review at our meeting tomorrow.
Please send any corrections to the list.
Thanks,
--jc
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 03/10/2011
Concall: start : 9:30a PT, adjourn : 11:20a PT
Next meeting: Thursday, 3/17/2011 @ 9:30a PT/12:30p ET
Dial-in numbers are 715-726-4994 or 866-304-8294.
Meeting ID and password are 7012090.
Attending :
Name Organization email
----------------- -------------- ----------------------------
Cory Spitz Cray spitzcor at cray.com
John Carrier Cray carrier at cray.com
Steve Simms IU ssimms at indiana.edu
Chris Morrone LLNL morrone2 at llnl.gov
Shay Seager OpenSFS shay at opensfs.org
Dave Dillow ORNL dillowda at ornl.gov
Sarp Oral ORNL oralhs at ornl.gov
Ercan Kamber RAID, Inc. ercan_kamber at raidinc.com
Andreas Dilger Whamcloud adilger at whamcloud.com
Eric Barton Whamcloud eeb at whamcloud.com
Agenda :
* resolve requirement priorities
Requirements :
We received the following ordered lists of the general requirements
discussed during the last few meetings. (See 2/12 email from Dave
Dillow to discuss at lists.opensfs.org for list of requirements.)
ORNL
- metadata scaling (ls -l, du, file opens for write)
- failover and recovery
- backend data system (expansion, larger luns, data protection)
LLNL
- complete OSD
- scalable fault management
- LNET channel bonding to provide more than one pipe from one box
- balancing storage usage and stripe allocation for even object
distribution
Indiana
- improved back-end
- better fault management
- metadata performance for WAN users
Cray
- single MDS performance
- compatibility/interoperability [withdrawn during meeeting]
- back-end improvements (>16TB LUNs, resiliency, rebuilds)
- scalable fault management
- HSM
- scalable MDS performance
- on-line file scrubber
- end-to-end data integrity
Discussion :
Several expressed the hope that OpenSFS would take a broader view by
considering work that needs to be done to pave the way for future
functionality. Yes, there are feature/performance issues that have a
clear target with immediate benefits, but foundational work will
resolve some of the remaining technical debt in the Lustre code and
set the stage for feature deliveries in 2013/2014.
By the end of the meeting, we agreed to recommend to the board that
OpenSFS pursue requirements for 1-2 performance features for 2012
delivery and 1-2 infrastructure projects that would lead to
deliverables after 2012:
Foundational projects :
OSD restructuring to provide basis for alternate backends
scalable fault management
btrfs investigation
Performance features :
speed up MDS (ls -l, du) to use avail HW capabilities
increase MDS capacity (distributed namespaces)
Dave and John agreed to prepare a draft document describing these
requirements for the next meeting. We also noted that OpenSFS needs
to move quickly. There are several institutions waiting to invest in
Lustre development in light of OpenSFS's decisions.
The following is a summary by topic of the projects discussed during
the call :
Technical debt :
What we describe as foundational features will address some of the
technical debt remaining in the current Lustre 2.x code. We
resolved that TWG requirements for any proposal should have
vendors show how the deliverable not only did not add to technical
debt, but paid off some of it too. It was suggested that the RFPs
include boiler plate asking how new feature development would
remove technical debt. Documentation and other release collateral
should also be part of the RFP boiler plate.
OSD :
Much of our meeting time focused on the OSD restructuring needed
to enable alternate file systems. This project will encounter
significant technical debt. For example, the 2.0 stack includes
1.8 client side stack for the server to talk to OSTs. This is
"Frankenstein" code that has been left behind and should not be
carried forward. It is a technical debt that needs to be repaid
(and fixed) during the OSD restructuring.
OSD restructuring is a feature; the following are the requirements
behind this feature :
(1) The need for an interchangable backend.
(2) Code reorganization to facilitate interop with different
backend devices
Funding partnerships :
OSD is a major project. LLNL thinks it could easily consume all
of OpenSFS's development budget. If OpenSFS is interested in OSD,
then we need to consider how to partner with other companies and
members on the RFPs.
Another option is to ask the RFP responders to split a task into
pieces that OpenSFS can fund. For example, 1/3 of the namespace
ops use the OSD API. It could be a separate task to add the API to
the other 2/3s (change logs, recover, OST, MGS, quotas).
Btrfs :
The requirements for an improved backend acknowledge that ldiskfs
is a short-term solution. The OSD restructuring provides a
mechanism for using other backend file systems, but it does not
actually specify the ldiskfs replacement. For this, we would have
to begin investigating alternatives, such as btrfs. The
investigation should have detailed requirements to be sure that we
understand stability, layering, and performance of the new file
system. We need to complete the baseline investigation before
starting on a btrfs OSD implementation.
Lustre administrtion :
Simple changes to improve administration (eg change protocol to
include jobid) will require restructuring beyond the scope of the
actual deliverable. This is another example a project that
pays off technical debt.
Error reporting (scalable fault management) :
Foundational work is needed to make Lustre tractable to standard
admin infrastructure. We cannot continue with error logs as being
used today. Need instead to define how do errors in underlying HW
get collected and made amenable to some other automated
administration tools. There are two phases:
- detection : collect and parse faults for notification
- notification : scalable network to distribute errors for
admin action
The health network is the scalable infrastructure for distributing
the notifications. The health network also used to detect nodes
participating in the cluster.
Distributed Namespace (aka CMD) :
CMD is a solution to requirements for distributed namespaces or
horizontal scalability of MDSs. The effort can be identified
easily into achievable pieces. We can structure requirements to
be aligned with two phases of development efforts.
eg, - namespace distribution for phase 1
- distributed directories for phase 2
Note that failover with CMD is an additional discussion not
covered by the phases above. Need non-blocking transactions for
full server failure. Until then, have async replication of data
and metadata. Don't confuse the issue.
It was suggested that, in this context, distributed namespaces are
not really foundational but performance related.
The following projects were considered to be out of scope for OpenSFS
TWG :
HSM:
There are technical issues being finalized in JC's group at CEA
(immutable layouts, layout locking). Though its implementation
is close to complete, it is believed that there is significant
amount of work to get it tested for release. On larger systems,
for example, expect HSM will run into performance problems related
to change-log scalability issues. Therefore, expect considerable
effort for productization of the proof of concept.
Lustre documentation :
There is currently no provision for Lustre documentation. The text
is in the creative commons, but there is work to create an open
source release. This effort, though, falls in the domain of the
release planning working group.
LUN size for ldiskfs :
Belief is that this will get done. It is not core to OpenSFS's
mission.
Interoperability / Compatability :
Rolling upgrades from Lustre 1.8 to Lustre 2.x will not be
possible. It will require admin shutdown of the servers
and client remount after the upgrade because of changes in the
LNET protocol.
More information about the Twg
mailing list