[Twg] 2011-02-10 meeting minutes
John Carrier
carrier at cray.com
Wed Feb 16 05:43:08 UTC 2011
Attached are the minutes from last Thursday's meeting when we reviewed Fujitsu's requirements. Dave sent our updated requirements to <discuss at lists.opensfs.org> on Saturday. We were to have sent new requirements to the list by EOB Monday.
Our next meeting is Thursday, 2/17/2011 @ 9:30a PT/12:30p ET. (Dial-in numbers are 715-726-4994 or 866-304-8294, meeting ID and password are 7012090.) The agenda for the meeting is to discuss requirement priorities and prepare for our future architecture meetings. Please review the requirements list and send your priorities _before_ the meeting. (Dave would prefer EOB Wednesday :)
Send requirements and priorities to <discuss at lists.opensfs.org> and any process concerns or questions to <twg at lists.opensfs.org>.
Thanks,
--jc
-------------- next part --------------
OpenSFS Technical Working Group
Meeting minutes : 02/10/2011
Concall: start : 9:30a PT, end : 11:30a PT
Next meeting: Thursday, 2/17/2011 @ 9:30a PT/12:30p ET
Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and
password are 7012090.
Deadline
- requirements by 5pm EST Monday
- priorities by 5pm EST
attending
Name Organization email
----------------- -------------- ----------------------------
Cory Spitz Cray spitzcor at cray.com
John Carrier Cray carrier at cray.com
Justin Miller IU jupmille at indiana.edu
Steve Simms IU ssimms at indiana.edu
Chris Morrone LLNL morrone2 at llnl.gov
Marc Stearman LLNL marc at llnl.gov
Shay Seager OpenSFS shay at opensfs.org
Dave Dillow ORNL dillowda at ornl.gov
Galen Shipman ORNL gshipman at ornl.gov
Eric Barton Whamcloud eeb at whamcloud.com
Agenda
requirement discussion and prioritization
Discussion
Eric forwarded a document he received from Fujitsu "Fujitsu Lustre
Requirements.pdf". John forwarded the doc to the reflector and we
spent the remainder of the meeting discussing the items they listed.
Note the meeting ran for an additional, unplanned, hour in order to
complete the review of Fujitsu's requirements.
In the notes below, the requirement is on one line and the discussion
appears, indented, on the lines below it. We went through Fujitsu's
doc page by page, but topics on each page may not have been covered
in order.
* 512KB block size for ldiskfs
This is a backend requirement. Not sure if ldiskfs is right
target, which is an implementation detail. A larger blocksize
gives more performance and gets by scaling requirements, but there
may be other backend solutions that do what Fujitsu really needs.
* 100 PB file system size
We expect file system size to scale with the memory size of the
mainframe. A typical rule of thumb is to track node memory,
size of normal checkpoint data, and the number of checkpoints
to store during a run.
LLNL expects to have 50 PB in 2011, 100 PB in 2012. ORNL has 10PB
today. They bought spindles to get the bandwidth needed and then
sized the capacity to meet storage requirements. Since capacity
of drives doubles every 18 months, agreed that 100PB is a
reasonable target for 2012.
* 10K OSS nodes
2K is achievable in near term, but 10K is aggressive and will
require changing derived attributes. To scale the OSS beyond 2K,
Lustre will need fast health detection to avoid problems caused by
distributed communications between clients and servers.
We need to track scaling of OSS as another feature that depends on
the health network. Nonetheless, we need to understand how many
OpenSFS customers will build systems of this size.
* 10K OSTs
Max # of OSTs with today's architecture will be a stripe-size of
1500, with minor format changes. There will be a significant
amount of work to go beyond this limit.
10K OSTs and 10K OSS implies ratio of 1 OST per OSS. Can single
OST use all the bandwidth from a single OSS. Not clear what the
architectural requirements are. How much bandwidth is needed? How
many rails of communication are there?
Consensus is that the goal should be 1K OSSs. The # of OSTs will
vary to get max BW from the OSS. One OST/OSS would reduce admin
cost.
* 100K clients
ORNL is moving to fewer, fatter nodes. BlueGene (LLNL) uses i/o
forwarding to funnel node requests to fewer clients. If Fujitsu
needs high node counts, they could also use I/O forwarding to
mitigate the client requirements.
Consensus that 64K clients in 2012 and 128K in 2014 is reasonable.
* 100TB OST size
This is a back-end storage requirement. Consensus that 32TB should
be the near-term target and 64TB is reasonable long-term (2014).
Beyond this, need new backend file system.
* 100K max subdirectories
Group believes the limit is not the file system itself but at the
VFS layer. Lustre is constrained by the Linux architecture.
* big-endian support
Lustre servers will be little-endian. Big-endian support is
available in Lustre 2.x for clients, but we will need to
test the code to be sure it works.
* varying page-sizes
This topic arose during the big-endian discussion. Lustre must
allow clients and servers to use different page sizes. (Expect
smaller on client, larger on server.)
* advisory file locking
This is currently supported, though no one was sure it was fully
operational. Also not sure how it was used, though one
person suggested the adio drive for MPI-IO might.
* server cache
Read cache exists. Write cache is messy. Agreed not to require
it.
* arbitrary OST assignment
Goal is to assign specific stripes to specific OSTs rather than
heuristically. This has a potential impact on WAN operations.
* 200K file creates/sec
This is an aggregate requirement for creates from all nodes in
the system. We looked at LLNL's 1.5 million core Sequoia system
to put this in context. They expect 100K file create/s. If each
rank creates its own file, then the checkpoint will need 15
seconds to create all files from all cores.
Someone then asked the question of how this time compared to the
time each core spends writing its file. There is 1 GB per core
and 512 GB/s is the bandwidth target. At 15 secs, LLNL estimates
opens will take ~4% of the write time.
200K creates/s may be a reasonable short-term goal. Some thought
we should expect this to quadruple by 2014.
* 100K metadata ops
There is a difference between 1.5 million cores creating files and
a single process running "ls -l" on the files created by the 1.5
million cores.
We need to separate aggregate metadata ops from requirements for
small numbers of processes. This may lead to two requirements:
one for aggregate stats, one for single client.
* assured disk i/o
The requirement should be for zero down-time while checking the
file system. We discussed separting this requirement for a
Lustre-level fsck and a low-level fsck. Both systems should have
online checking and scrubbing. Failover doesn't work if doing
off-line fsck.
* 1PB file size
This is a consistent with shared-file requirements from others.
* dynamic OST addition/deletion
Ceph provides a model for adding storage dynamically. It uses the
CRUSH algorithm to determine striping. For Lustre, we need to
consider the locality of the new OSTs while enabling dynamic
striping. Could create metadata to create heirarchical map of
storage. With equivalent choices, get random stripe assignments;
with unequivalent choices, get directed i/o.
other topics mentioned during this discussion:
* space-determined migration,
migration can be driven by policy or command line
* adaptive object layouts
needs research: CRUSH is not spectacularly good or bad
* replicas
not RAID across OSTs, but policy-driven replication
- for certain files, need to get copied somewhere else
- if complete backend disaster, then have recovery mechanism
* Exascale and POSIX
Fujitsu's exascale numbers are really extreme and show how
exascale is stretching POSIX. Instead of exposing POSIX to such
extreme scales, we could explore architecture changes.
Applications must still see POSIX APIs and write to file objects
as expected, but without overloading the namespace held by the
metadata server to show all files.
An example would be the use of data sets on the MDS, where instead
of exposing all of the files in the namespace, there would a
single file with metadata for all of the other subordinate objects
in the data set.
The challenge is expressing this in terms of requirements. We
agreed to explore the goal of alternate access methods and
namespaces. For example, look to support multiple namespaces and
consider a generalized object model vs posix interface.
* MDS scaling
We all know that Sun had worked on SMP scaling to improve
MDS performance. If Lustre can scale on SMPs, then there will be
wider range of systems to run on. If can't exploit additional
cores, esp on servers, then limiting ourselves.
The requirement is that we expect Lustre to run on reasonable
hardware and that we expect the software system to scale and
perform with underlying hardware.
* patchless servers
Should be required for better support for new Linux, security
patches etc
* scalability
The Fujitsu system appears to be large enough that there should be
locality of reference to the OSSs to avoid contention. Having all
clients talking to all servers will cause turbulence on the
interconnection network.
ORNL attempted to resolve this problem with LNET fine grain
routing. The routing forced clients to access the router connected
to specific servers in order to avoid contention on their IB I/O
network.
Locality is part of the story for avoiding network congestion
between clients and servers.
Clients spend time checking connectivity of all servers in the
file system. Localization would also reduce the number of servers
mapped by the clients and avoid using loads of memory to keep
connection state.
Caching proxy servers will also alleviate congestion problems.
Long-term, procxies will be needed to reach 1 million clients.
* snapshots
Replicas and snapshots will be needed to do Lustre backups. The
requirement is for consistent use of file system at point in time.
Next Meeting (2/17)
continue discussion on <discuss at lists.opensfs.org>
- send additional requirements by 5pm EST Monday
- send priorities by 5pm EST Wednesday
More information about the Twg
mailing list