[Twg] 2010-12-10 meeting minutes
John Carrier
carrier at cray.com
Thu Dec 16 00:36:37 UTC 2010
Hi all,
Attached are my notes from last week's meeting. Please let me know if you have any corrections or additions.
This email is also a reminder of our meeting tomorrow 12/16 at 9:30a PT / 12:30 ET. Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and password are 7012090.
We had two homework items from our last meeting:
* review the whitepaper
* get roadmap priorities from our home institutions
I was told by a member of the board that the whitepaper needs to be our priority. This is our proposal for the goals of the working group and the process we will follow in pursuing those goals. We need to submit a draft of this document before we can pursue the roadmap and initiate any development discussions.
Please be prepared to discuss the draft whitepaper at tomorrow's meeting. You can find a copy in the email archive:
<http://lists.opensfs.org/pipermail/twg-opensfs.org/attachments/20101202/46e7fccb/attachment-0001.pdf>
Thanks,
--jc
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 12/10/2010
Concall: start : 9:30a PT, end 10:32a PT
Next meeting: Thursday, 12/16/2010 @ 9:30a PT/12:30p ET
Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and
password are 7012090.
Attending:
Name Organization email
----------------- -------------- ----------------------------
John Carrier Cray carrier at cray.com
Kit Westneat DDN kwestneat at ddn.com
Justin Miller Indiana Univ. jupmille at indiana.edu
Josh Walgenbach Indiana Univ. jjw at indiana.edu
Damian Hazen LBL dhazen at lbl.gov
Chris Morrone LLNL morrone2 at llnl.gov
Ken Hornstein NRL kenh at cmf.nrl.navy.mil
Shay Seager OpenSFS ssseager at gmail.com
Andreas Dilger Oracle andreas.dilger at oracle.com
Dave Dillow ORNL dillowda at ornl.gov
Ercan Kamber RAID, Inc. ercan_kamber at raidinc.com
Eric Barton Whamcloud eeb at whamcloud.com
Robert Reed Whamcloud rread at whamcloud.com
Agenda:
* whitepaper reminder
* continue review Eric's slides "Lustre Development Roadmap"
<http://www.opensfs.org/wp-content/uploads/2010/11/Lustre-development-roadmap1.pdf>
Whitepaper
- John sent a draft of the whitepaper to the group on 12/1:
Desc: OpenSFS TWG white paper_ver1.pdf
URL: <http://lists.opensfs.org/pipermail/twg-opensfs.org/attachments/20101202/46e7fccb/attachment-0001.pdf>
- John wants to send a draft of the document to the other working
groups but needs feedback from the other members of the working
group. Please read it next week. Send comments to the list.
Roadmap slides
section 3 : Development tasks (continued from 12/3/2010 meeting)
------------------------------
15 * fault management
= fault reporting
- improve lustre's reporting of errors and faults
- some work already done to classify errors
- originally any error used C macro to report to
syslog (c debug/event tracker)
- problem is that errors are in system logs and no way
to plug errors out of lustre into wider admin
infrastructure
- need to define report architecture
- vendor neutral
- flag fault class, severity, etc
- may initially go to system event logs, but if
trying to integrate lustre into vendor
management infratstructure, the vendor could
take feed of errors from lustre (collected on
servers and clients), then channel them into
someplace for fault logging or automatic
administrative action based on faults reported
- should anticipate health networks
- recognize that health networks will be major
stepping stone for scalablity of fault
management of Lustre
- need to anticipate how these networks should be
checked
discussion:
Q: AD: newer mechanisms in kernel to avoid using syslogs
- can use more than printk
- work done at TACC to add different messages out of
kernels
- other channels: ex netlink -- should do something
like it
EB: major issue is to classify the errors consistently
thinking ahead to what classes and severity of errors
will be necessary in the future.
require serious thought to get right classes / severities
- or is there a different architecutre
- need to feed into automatic tool, not parsing
text messages
essentially, the issue is how to put stethescope on
side of lustre
AD: pass messages over the network?
EB: don't want to invent something that will be useless
once these networks are available
AD: classes of errors will report nodes as dead to the
network?
EB: or just how to use network for reporting errors.
health networks respond to faults, this is how to
report the faults
Q: CM: specific admin tools?
EB: no particular message in mind. don't want to second
guess what vendors will use
CM: work with vendors for what are the inputs they need
EB: satisfy requirements, but not mandate the tool used
AD: use standard tools (eg ganglia)
= Online FSCK / scrubber
- want to avoid lfsck, it doesn't scale
- need mechanism to validate the file system and find
inconsistencies in the FS introduced by bugs and crashes
- sync CMD relying on orphan clean-up for failures in the
middle of distributed metadata operation
- need something to crawl over the entire file system based
on policy to ensure anything damaged is detected and
fixed
- also want to do failover without doing full FS check
- as long as we know underlying storage is consistent
to crashes
- guarantee that crawl of FS is in finite time
- allow failover without lengthy check before hand
- design work already done by the lustre group for Cray HPCS
- but need to address OSD specific scanning
- OSD-specific resilvering
16 * administration
= improved configuration
- move away from config files, use scripts instead
- ex can configure LNET from userspace with lctrl
- work is to allow network drivers to be brought up and
down on individual interfaces
- result puts all name handling into user space
- currently have complex config with multiple
networks
- clumsy to configure the networking in one way, then
bringup servers just to get nids in right places
- need a simpler, text-driven config system.
- configure file system as single text file on MGS
discussion:
Q: KH: config of clients via mod parameters too?
EB: anything to do with file system itself should be part of
textual file system configuration (eg global
parameters that affect all clients).
networking interfaces for clients would be LNET configs
- do it with scripts like IP is configured
- bring up interfaces etc, (more admin friendly)
but client-specific parameters : need those in tunables
set on clients, not part of file system
Q: KH: keep those as mod params or put in user space?
EB: want to overcome non-obviousness of creating file system
and binding the server addresses. eg, currently some
config information disappears after file system starts
= OST/MDT pool management
- OST pools are ways to name collections of OSTs (type of
shorthand)
- To make them more useful, have to control which users are
using which pools
- reserve pools for high priority projects
- permissions would increase usability
- rebalance pools based on OST usage , adding new OST
- allow migration between peers
- HSM allows migration between OST and tape. Want to extend
to allow migration between different qualities of storage
= global snapshots
- use underlying OSD model
- simple case: block, then copy
- with epochs, could make it non-blocking
- make snapshots as easy as it is with ZFS
- consider standards to warn storage that snapshot is desired
- need consistency of storage
- but want to warn apps using the file system so they
leave the correct state
- any architecture doing global snapshots need to
inform applications to save state
discussion:
AD: an ioctl allows application to initiate snapshot.
could export to lustre to let it receive the ioctl
EB: second guessing admin, requiring all apps to be crash
recoverable is a burden on the apps
AD: what are the APIs?
EB: standards for doing it
AD: unsynchronized snapshots is good
EB: two cases:
- app initiated snapshot
- admin initiated snapshot
for (2), if app is crash recoverable, go to previous
state and roll forward or
- have IOCTLs to tell application about freeze/thaw
events
- for freeze message, get in good state on file system
for application consistency.
- means recovery is easier later on
DD: 3rd option to have apps tell FS that the app is busy and
not to snapshot
EB: yes, but need to consider range of apps running on FS
simultaneously
RR: need snapshot done first
[end of slides]
Discussion - next steps
The working group needs to digest the information in Eric's slides.
- take the roadmap back to home institutions for review
- prioritize the featurs (and find features not yet discussed)
- identify short-term projects for OpenSFS to pursue
Andreas mentioned the process Sun followed for Cray's HPCS
program. There were two RFPs: one for design followed by one for
implementation.
- the HPCS designs are linked here:
<http://wiki.lustre.org/index.php/Lustre_HPCS_Activities>
Eric suggested one short-term project for OpenSFS would be a btrfs
evaluation. BtrFS has similar properties to ZFS and is a contender
for the Lustre backing store. This project would be a survey of the
code and the stability of its current implementation. The result
would be an understanding of the suitability of btrfs for Lustre and
the scope of the development needed to create a btrfs OSD.
More information about the Twg
mailing list