[Twg] Rough requirements

[Andreas Dilger]

I'm just adding some notes for each item to give an idea where it stands today.

idea = just something that was discussed
design = a written design exists
code = some code exists


Also, I won't be able to attend the Thursday meeting, possibly Eric will be able to attend.

On 2011-02-04, at 16:01, David Dillow wrote:
> Performance requirements
> 
> Metadata Performance
> [[ mainly an import of John's list ]]
> 
> * metadata performance
> 
>    GOAL: improve file system scalability and interactive
>          performance
> 
>    requirements:                  Q2 2012         Q1 2014
>                                --------------  -----------------
>    -  # files in file system    100 billion       1 trillion
>    -  # files in directory       50 million      10 billion

The 2012 numbers could be attainable with CMD + ldiskfs, if one considers 1-2B inodes/fs, 0.5-1M inodes/dir and 50-100 MDTs.  I don't think the 2014 numbers are achievable with ldiskfs, unless we really push the number of MDTs to the ~500 range.  It is definitely higher than the number we were thinking about for CMD.

>    -  aggregate file creates/s  100 thousand       ?
>     - single file creates/s         ?            30 thousand
>    -  directory listings/s          ?              ?
>    -  open files per client        -           100 thousand
>    -  file system capacity       30 PB          100 PB
>    -  # clients                  30 thousand       ?

I would put most of these requirements  in the "idea/code" stage.  Some CMD code exists, some needs to be written and has only been discussed.

> [[ There is a question as to what's driving 10 billion files in a directory?
>   How does one expect ls to work in that directory? Is that expected to
>   be POSIX compliant? ]]
> [[ open files per client instead of open files per process? or is that
>   total open files per system or job? We've already passed that it seems... ]]
> 
> * backend storage
> 
>    GOAL: provide reliable, scalable backing store for 
>          Lustre servers
> 
>    requirements: 
>    -  large LUNs (min?, max?)
> [[ suggest 32 TB for minimum max LUN size supported near term, 64 TB for
>   mid-term ]]

code exists in e2fsprogs, ext4, needs testing, integrated testing with Lustre.  Also needs additional fixes to support large objects (> 2TB)

>    -  end-to-end data integrity 
>       (ie provide resiliency that T10 PI gives local file systems)

idea - T10 for ldiskfs
design - end-to-end checksum design for HPCS

>    -  low performance impact for file system repair

idea/code - slow OST avoidance, improved MDS object allocation algorithm

> [[ Eric notes that find has a similar set of requirements as ls ]]
> [[ full bandwidth and IOPs for a single shared files -- this is wide stripe ]]
> [[ Improved small file IO performance ]]
> 
> 
> Quality of service
> 
> 	Existing deployments currently have no mechanism to balance the
> performance needs of interactive users against the needs of large-scale
> compute jobs -- it is possible and likely that directory listings will
> encounter absurdly long execution times when competing against a 200,000
> core checkpoint operation. To maintain usability in such scenarios, Lustre
> must be able to allocate a {job,cluster,user} a share of IOPS and bandwidth
> consumate with the priority levels assigned by an administrator.

code - prototype NRS code exists, needs review, testing

> Scalable fault management
> 
> 	While it is already the case that today's supercomputers have a
> marked dependence on their file systems for productive use, this dependency
> will continue to rise as we see more and more center-wide file systems. To
> minimize the downtime for the entire center, reliability must increase and
> recovery from faults must be bounded in time. Lustre must be able to recovery
> in O(log n) time or better as a mid-term goal to meet this requirement.
> [[ Replication to achieve fault tolerance was mentioned --
>   requirement vs implementation detail? ]]

idea

> Improved configuration of Lustre
> 
> 	The current mechanism of using module parameters is relatively
> inflexible and can constrain large, complex Lustre configurations. As a
> short- to mid-term requirement, Lustre must have the ability to allow
> dynamic configuration of the LNET interfaces and routes. This ability should
> include the ability to bind targets (OST/MDT/etc) to specific LNET interfaces,
> and to hot-add or hot-remove LNET interfaces.

idea

> Allowing for controlled partial-system maintenance
> 
> 	Currently, to upgrade a Lustre installation or perform maintenance
> on a subset of the comprising hardware, one must unmount the filesystem from
> all clients or risk hanging processes until the hardware is back online
> (maintenance) or other odd, undefined client behavior once the upgrade
> completes. To allow more flexible administration, the filesystem must be able
> to handle these situations gracefully, and allow the clients to avoid
> attempting to use hardware known to be down.
> [[ AKA Administrative Server Shutdown ]]

idea

> Balancing storage use
> 
> 	Currently, ensuring a balanced use of the storage space available
> to Lustre relies on a haphazard set of setting default stripping, storage
> pools, and manual rebalancing of overfull OSTs. As a mid-term goal, Lustre
> must be able to allow automatic emptying of an OST, migrating the data
> to other devices in the filesystem. Similarly, Lustre must be able to
> rebalance the storage load over new OSTs as they are added. Additionally,
> Lustre must be able require authorization for use of specific storage pools.

idea - object migration
idea - improved MDS object allocation

> more management
> [[
> 	how to handle integrity check requirements, split online/offline?
> 
> 	Improved analysis and visualization tools?

code - some work has been done on this to convert Lustre RPCTRACE logs to OTF for viewing in VampirTrace.  Is that an open-source tool?

> 	Better userspace tools
> 		-- redo llapi
> 		-- lctl
> ]]
> 
> Improved storage semantics/interfaces
> [[ need verbiage, possible requirements:
> 	ADIO/MPI-IO improvements
> 		collective open?

idea

> 	Better ability for user to specify expected access patterns
> 		-- fadvise() to say "I will use this range, get the
> 			appropriate locks"

idea

> 	Support Linux AIO to make O_DIRECT more useful

idea

> 	readdir+/statlite/collective open from POSIX HECEWG

idea/code - some code proposed to kernel developers in the past, got bogged down in a morass of feature bloat.

> Misc
> [[
> 	Improved LNET bandwidth and reliability
> 		-- channel bonding

idea

> 	Improved security
> 		-- user name mapping

code for 1.8.  Needs efficient NID matching, per discussions with Eric in the distant past.

> 		-- propagating authorization from MDT to OST through
> 			trusted channels rather than untrusted clients

idea

> 	Patchless server and better support for upstream kernels

idea/design - detailed descriptions of how to remove each patch, some work done for RHEL6/Lustre 2.1



Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.