[Twg] Lustre Requirements and Roadmap

[Andreas Dilger]

On 2011-01-27, at 14:56, Christopher J. Morrone wrote:
> I think that for the metadata performance, we should use labels other than "min" and "max".  After all, we would never say "no, we won't accept a filesystem that is capable of more than than this". :)
> 
> Also, the requirements are only useful if we can give target dates for the requirements.  So maybe the "min" column becomes Q3/2012, and the max column Q1/2014.  I am just pulling those dates out of thin air, they could be something else.

The other thing that is always difficult about specifying performance requirements for Lustre, is that these numbers cannot be specified in the absence of some idea about the system on which the tests are expected to run.

> I think we should split the aggregate and single client file creates/sec requirements into two separate requirement lines.
> 
> For backend storage:
> 
> I think that we should take out the "framework to enable alternatives to ldiskfs" line.  That is a design decision rather than a requirement.  I think it is certainly necessary given the requirements, but if we are trying to distinguish between requirements and architectural decisions, that line should be removed.
> 
> Rather than "no performance impact", say "low performance impact".  Zero impact is probably impossible. :)   At some point we need to be clear on exactly what quilifies as "low".

I totally agree with this, and have never liked the "no performance impact" term.  It essentially means that you are throwing away performance in the non-degraded case, just so that the degraded case does not appear to have any performance impact.

> One part of that might be:
> 
>  - The solution must perform filesystem integrity checking and repair
>    on-line.
> 
> I.E., There may be no required downtime just for consistence checking and repair.
> 
> What is the rationale for the "direct I/O mode" requirement?  Not that I am opposed to it...but is that really a requirement, or a solution?
> 
> For end-to-end integrity, I would suggest that T10 PI is not sufficient, since it only concerns itself with the path between the HBA and the backend storage (disks/flash).

That is not necessarily true.  T10 PI could likely be integrated into the Lustre integrity checking, in the same manner that was proposed for the ZFS checksums in the HPCS design, so that it goes all the way to the memory of the client.

One major issue that has been observed with T10 PI in Linux (even for local filesystems) is that the VM/VFS allow data blocks to be modified while IO is in flight (in particular for MMAP files), and that can cause spurious data integrity failures.

Lustre works around this today in its network checksum implementation, but the only sure-fire way to handle it is to either copy the data before it is sent over the network, or to block writes to pages that are currently in RPCs, both of which can have significant performance impact.  Fortunately, this is not a common use case for HPC applications.

>  I think that we need to define where the "ends" are, and they should really extend up into memory on the lustre servers.



> On 01/27/2011 09:24 AM, John Carrier wrote:
> 
>> The following is an incomplete list of requirements to motivate further
>> discussion :
>> 
>>    * metadata performance
>> 
>>       GOAL: improve file system scalability and interactive
>>             performance
>> 
>>       requirements:                    min            max
>>       -  # files in file system    100 billion       1 trillion
>>       -  # files in directory       50 million      10 billion
>>       -  file creates / sec        100 thousand     30 thousand
>>                                      (aggregate)      (single client)
>>       -  directory lisings / sec
>>       -  open files per process        -           100 thousand
>>       -  file system capacity       30 PB          100 PB
>>       -  # clients                  30 thousand    ?00 thousand
>> 
>> 
>>    * backend storage
>> 
>>       GOAL: provide reliable, scalable backing store for
>>             Lustre servers
>> 
>>       requirements:
>>       -  large LUNs (min 32 TB)
>>       -  end-to-end data integrity (T10 PI or equivalent)
>>       -  no performance impact for file system repair
>>       -  framework to enable alternatives to ldiskfs
>>       -  direct I/O mode
>>       -  ??
>> 
>> 
>> 
>> 
>> _______________________________________________
>> twg mailing list
>> twg at lists.opensfs.org
>> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org
>> .
>> 
> 
> _______________________________________________
> twg mailing list
> twg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org


Cheers, Andreas
--
Andreas Dilger 
Principal Engineer
Whamcloud, Inc.