[Twg] Rough requirements

[David Dillow]

Extremely rough draft, I'm sure I didn't capture everything for our
discussion Thursday, and didn't get to incorporate discussion on the
lists. My comments/unfinished text is in [[ ]] pairs. We need to divide
things up by area, and try to polish enough to send to the open
reflector by late Monday.



Performance requirements

Metadata Performance
[[ mainly an import of John's list ]]

 * metadata performance

    GOAL: improve file system scalability and interactive
          performance

    requirements:                  Q2 2012         Q1 2014
                                --------------  -----------------
    -  # files in file system    100 billion       1 trillion
    -  # files in directory       50 million      10 billion
    -  aggregate file creates/s  100 thousand       ?
     - single file creates/s         ?            30 thousand
    -  directory listings/s          ?              ?
    -  open files per client        -           100 thousand
    -  file system capacity       30 PB          100 PB
    -  # clients                  30 thousand       ?
    - ... 

[[ There is a question as to what's driving 10 billion files in a directory?
   How does one expect ls to work in that directory? Is that expected to
   be POSIX compliant? ]]
[[ open files per client instead of open files per process? or is that
   total open files per system or job? We've already passed that it seems... ]]
             
 * backend storage

    GOAL: provide reliable, scalable backing store for 
          Lustre servers

    requirements: 
    -  large LUNs (min?, max?)
    -  end-to-end data integrity 
       (ie provide resiliency that T10 PI gives local file systems)
    -  low performance impact for file system repair

[[ suggest 32 TB for minimum max LUN size supported near term, 64 TB for
   mid-term ]]


[[ Eric notes that find has a similar set of requirements as ls ]]
[[ full bandwidth and IOPs for a single shared files -- this is wide stripe ]]
[[ Improved small file IO performance ]]


Quality of service

	Existing deployments currently have no mechanism to balance the
performance needs of interactive users against the needs of large-scale
compute jobs -- it is possible and likely that directory listings will
encounter absurdly long execution times when competing against a 200,000
core checkpoint operation. To maintain usability in such scenarios, Lustre
must be able to allocate a {job,cluster,user} a share of IOPS and bandwidth
consumate with the priority levels assigned by an administrator.



Scalable fault management

	While it is already the case that today's supercomputers have a
marked dependence on their file systems for productive use, this dependency
will continue to rise as we see more and more center-wide file systems. To
minimize the downtime for the entire center, reliability must increase and
recovery from faults must be bounded in time. Lustre must be able to recovery
in O(log n) time or better as a mid-term goal to meet this requirement.
[[ Replication to achieve fault tolerance was mentioned --
   requirement vs implementation detail? ]]


Improved configuration of Lustre

	The current mechanism of using module parameters is relatively
inflexible and can constrain large, complex Lustre configurations. As a
short- to mid-term requirement, Lustre must have the ability to allow
dynamic configuration of the LNET interfaces and routes. This ability should
include the ability to bind targets (OST/MDT/etc) to specific LNET interfaces,
and to hot-add or hot-remove LNET interfaces.


Allowing for controlled partial-system maintenance

	Currently, to upgrade a Lustre installation or perform maintenance
on a subset of the comprising hardware, one must unmount the filesystem from
all clients or risk hanging processes until the hardware is back online
(maintenance) or other odd, undefined client behavior once the upgrade
completes. To allow more flexible administration, the filesystem must be able
to handle these situations gracefully, and allow the clients to avoid
attempting to use hardware known to be down.
[[ AKA Administrative Server Shutdown ]]


Balancing storage use

	Currently, ensuring a balanced use of the storage space available
to Lustre relies on a haphazard set of setting default stripping, storage
pools, and manual rebalancing of overfull OSTs. As a mid-term goal, Lustre
must be able to allow automatic emptying of an OST, migrating the data
to other devices in the filesystem. Similarly, Lustre must be able to
rebalance the storage load over new OSTs as they are added. Additionally,
Lustre must be able require authorization for use of specific storage pools.

more management
[[
	how to handle integrity check requirements, split online/offline?

	Improved analysis and visualization tools?

	Better userspace tools
		-- redo llapi
		-- lctl
]]

Improved storage semantics/interfaces
[[ need verbiage, possible requirements:
	ADIO/MPI-IO improvements
		collective open?

	Better ability for user to specify expected access patterns
		-- fadvise() to say "I will use this range, get the
			appropriate locks"

	Support Linux AIO to make O_DIRECT more useful

	readdir+/statlite/collective open from POSIX HECEWG
]]

Misc
[[
	Improved LNET bandwidth and reliability
		-- channel bonding

	Improved security
		-- user name mapping
		-- propagating authorization from MDT to OST through
			trusted channels rather than untrusted clients

	Patchless server and better support for upstream kernels
]]


-- 
Dave Dillow
National Center for Computational Science
Oak Ridge National Laboratory
(865) 241-6602 office