[Twg] 2010-12-03 meeting minutes
John Carrier
carrier at cray.com
Thu Dec 9 23:47:26 UTC 2010
Hi all,
Attached are the minutes from last week's meeting. Please let me know if you have any corrections or additions (eg, I may not have caught everyone who was on the call).
This is also a reminder that we are continuing the discussion tomorrow at 9:30a PT / 12:30 ET. Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and password are 7012090.
--jc
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 12/03/2010
Concall: start : 9:30a PT, end 10:42a PT
Next meeting: Friday, 12/10/2010 @ 9:30a PT/12:30p ET
Dial-in numbers are 715-726-4994 or 866-304-8294.
The meeting ID and password are 7012090.
[Note: the original meeting was to have been on 12/2, but due to a
schedule mix-up, we were forced to delay the meeting 24hrs.]
Attending:
Name Organization email
----------------- -------------- ----------------------------
John Carrier Cray carrier at cray.com
Justin Miller Indiana Univ. ?
Steve Simms Indiana Univ. ssimms at indiana.edu
Josh Walgenbach Indiana Univ. jjw at indiana.edu
Chris Moronne LLNL morrone2 at llnl.gov
Andreas Dilger Oracle andreas.dilger at oracle.com
Dave Dillow ORNL dillowda at ornl.gov
Sarp Oral ORNL oralhs at ornl.gov
Galen Shipman ORNL ghsipman at ornl.gov
Ercan Kamber RAID, Inc. ercan_kamber at raidinc.com
Eric Barton Whamcloud eeb at whamcloud.com
Agenda:
Review Eric's slides "Lustre Development Roadmap"
[http://www.opensfs.org/wp-content/uploads/2010/11/Lustre-development-roadmap1.pdf]
[Editor's note: I have broken Eric's presentation into three sections --
intro, building blocks, development tasks -- that flow from the general
principles to more specific items. Also, just a reminder, these are
notes and not a stand-alone description of Eric's slides.]
03 Engineering principles
- Lustre has lots of code and want to keep it scaling.
- not just customers experiencing instability will try something new
but cannot continue development without stability
= therefore, stability must be at heart of the strategy
- to do this, must develop small steps and must have a clear
direction where the development is going
= we must think ahead to where we want to be in 5yrs so development
today is progression to this goal
- ALSO, fundamental to a project like lustre, must modify subsystems
- always fighting schedules, then try to take shortcuts
- interruption in middle of development of new structures, leads
to keeping old structures in place
- challenge for developer to work on new things while keeping old
things in mind
- technical debt: anytime you keep old code in place, then are
incurring debt and there is interest accrued over time for
keeping the old code around
example: Sun's purchase of CFS resulted in a 3yr hiatus to
remove the technical debut accrued by CFS
04 agenda
far attractors = long term directions
future technologies = game-changing building blocks
development choices = what needs to be done now while anticipating
the evolution of Lustre
section 1 : long-term goals
---------------------------
05 * major requirements
= extreme system size
- more clients (nodes, process/node)
- fault management is an issue at scale of these devices
- failing of lustre today is that advertised fault tolerance
works at tiny scale, but breaks down at scales that
Lustre claims to attract
- fan out ratio is huge (all clients to all servers)
- need fault management in all-to-all
- well bound time to detect failure and respond to it
=> if not, then spend time responding to faults
= throughput of servers
- 1000's of servers (OST, MDT)
- HPC loads : barriers lead to "denial of service"
- exascale = barriers will be evil
- will find that apps remove barriers and have
constant I/O loads
- consistent with enterprise (cloud) goals
- MDS performance
- start addressing issue of multiple MDS
- make lustre more general purpose
- in absence of contention, it should perform like a
local file system
= usability
- central/global file system
- share with other clusters
- need QoS, SLAs
- security is an issue for cross campus i/o
- data management
- HSM is just a start
- better ways to exploit different types of storage
- cheap for capacity
- expensive for speed
- structure files and data sets
- what other layouts are possible?
- administration
- existing tools not developed with mind of admin in mind
- how to cope with scale of devices,
- how to present the status of the devices
section 2: Building Blocks (future technologies)
------------------------------------------------------
06 * health network
- separate fault detection from the inband protocol
- currently, all client/server failures by inband communications
- but server load is so high that one client among 100Ks
has very high latency
- hence, using latency as indication of component failure
isn't very useful
- thus finding out status of peer is a facility needed to
cope with systems at enormous scale
how?
- logically separate network implementing spanning tree over
all nodes
- tree rooted where all data is aggregated
- make global decisions
- root must be fault tolerant (eg paxos)
but other nodes must be redundant
- result is that communication scales latency as O(log n)
- how to embed this inside LNET?
- LNET has diff message priorities
- embed the spanning tree in LNET and give guaranteed
latency across system
- use for
- prompt failure notifications
- collectives (census)
- distributed transactions
07 * distributed transactions (epochs)
- have had specific adhoc solutions to distributed transactions
- hard to implement CMD because don't have distr.trans,
- has adhoc solution: synchronous sequential operations
- ensures that always safe
- okay for a start, but not general
- need cleanup after failure
- need to sync transactions to disk
- eeb is not keen on two-phase commits
- epochs implement global transactions
- system wide transaction number
- servers keep roll-back logs
- clients keep roll-forward logs
- in case of any failure of a client
- server can roll-back
- in case of server failure
- system can roll-back to stable point
- roll-forward from clients to capture new data
this tool lets you do CMD, but also
- aggregate RPCs
- RAID files
- replication
08 * caching
- example = MDS caching
- need both health network and distributed transactions
to enable scalable caching
- allows Lustre to aggregate clients on single node
- with SDDs, offer home directories as fast as local
storage
- need to aggregate i/o and lock access
- what are policies for exploiting the features efficiently
09 * layouts
- currently only have RAID0 (ie striping across OSTs)
- want more complex, dynamic layouts
layout locks from HSM are the key
currently, Lustre has objects and layout stays with object
- allow files to be on MDT when small
- move to objects (raid0) when larger
change layout incrementally
- then do seamless replication and migration
- for example, if know OSTs will fail regularly, then
could mirror OSTs or files within the file
also enable non-posix access
Discussion :
Q: order of slides
A: health network (6) enables distributed transactions (7)
distributed transactions (7) enables a lot of caching (8)
layouts (9) is independent of the other three technologies
for example, distributed transactions are important, but
need scalable collectives
section 3 : Development tasks
------------------------------
10 * current issues
lustre 1.8
- MD performance
- failover latency
- ldiskfs scalability
discussion:
Q: AD: lustre 1.8 fixes?
A: EB: not taking plunge to 2.x until see someone else to it
AD: but adding features
EB: strict separation of feature and bugfix releases
AD: 1.10? ldiskfs 16TB
EB: 1.9 would be a feature release
depends on
AD: new 1.x could fragment the development effort
EB: any development for 1.x has to port forward for 2.x
DD: problems for 1.8 also exist in 2.0
lustre 2.0
- performance regressions
- smp scaling (numio)
- zfs / e2e integrity
- hsm
measurement & test
11 * ZFS alternatives
BTRFS
- consider BTRFS
- does it fit with OSD model
ldisfkfs
- overcome 16TB limits
- 64bit capacity
- t10diff
- hardening for future proofing
- look at data structure to see fi they scale
12 * technical debt
= distributed resources
- existing quota and grant are fragile
quota for resource allocation
grant = space reservation on OST so that when
cache cleaned there is space for data
- more robust generic implementation of distributed
resource management could do both as well as other
features,
eg total RPCs in flight
- if only one client, could increase the RPCs in
flight, but when all clients are active, then only
want one RPC in flight
= security
- secure RPC is implemented, capabilities are not
- ensure operations on one server granted by another,
will continue
- for example, connect on client across country
- all operations will be authenticated and authorized
- but OSTs and which particular user is using it, not
authenticated
- can subvert the protection of the namespace by
accessing objects directly
= capabilities complete the implementation
- returned with file handle to allow user to access
file objects
13 * performance
= multirail networking
- aggregate IB for greater per node performance
- failover
= server side request scheduling
- service thread pools in Lustre created to avoid
deadlocks
- request handling has been elaborated so that requests
inspected and scheduled to threads for handling
- have the beginnings: know some are high priority and
moved to front of queue
- but to do it properly need general purpose request
scheduling (QoS, priority,) would allow better use of
threads, gang-scheduling (avoid starvation)
eg, allow viz cluster to work while doing crash dump
from compute cluster
14 * CMD
- full BW for all types of servers requires lots of development
- requires health network and epochs
- working on version of CMD based on write through and
sequenced operations
- if have ops on three diff servers, then order the ops so
that, if there is a failure, the worst that can happen is
that the link counts will not
- increment link counts before creating reference
- destroy references before decrementing link counts
- worst case is an orphan reference (objects exist but are
unused by namespace)
- can be scanned with background utility
- development in two phases
phase1: use admin function to create directories with
distrubed functions on different MDTs
- access directories with 1-1 access as it is today
- allow home directories on different MDT than the
directories for the scratch i/o
- created independence of applications using
- similare to mounting differerent File system that
happen to use the same OSTs
phase2 : stripe directories
- have files in directories on diff targets
- hash filename, tells where to get MDTs for the
- also start out with admin function, but thereafter,
all i/o in directory is local to MDT
- stripe directories give order and speed-up for file
create in single directory and speed-up for large
directories
phase1 - reasonable to handle multiple links with xdev
phase2 - less reasonable to say that rename and link will
give xdev because moving files between MDTs
discussion:
AD: code exists already
- but have implementation to get into production state
EB: rough estimate
- non-trivial to get into production
[meeting adjourned at 10:42a PT -- two slides, fault management and
administration yet to be discussed]
More information about the Twg
mailing list