[Twg] T10 End-to-End Data Integrity HLD

[Andreas Dilger]

On 2012-04-18, at 15:47, "Nathan Rutman" <Nathan_Rutman at xyratex.com> wrote:
> Perhaps we can talk about this further at LUG, but I did want to answer a couple of things.
> 
> On Apr 18, 2012, at 2:55 PM, Andreas Dilger wrote:
>> Sure, but the Merkle tree ends up being just a different checksum algorithm supported by the OST, that doesn't need any specific knowledge of the backing filesystem (blocksize, sector size, etc), and it doesn't impose any significant overhead on the protocol.  The only potential change outside of adding the algorithm would be to send a full 256-bit checksum instead of only a 32-bit one today.
>> 
>> Exposing a low-level detail like the disk sector size to the Lustre client (or application, should it get that far) to compute the GRD tag is a terrible layering violation, IMHO.  This flies in the face of code abstraction and proper layering/isolation of different components in the storage stack, and is really contrary to the principles of Lustre that try to isolate the clients from internal details of the storage.  It doesn't make sense that a Lustre client doesn't need to know the block size or even the type of filesystem on the OST, yet it needs to know the sector size of the disks living some layers below each OST?
>> 
>> Perhaps if there was a simple transformation for merging eight 512-byte GRDs into a single 4kB GRD, then at least the Lustre client could always compute and send the GRD tags for the smallest sector size for any OST it is connected to (512 bytes, if there are mixed sector sizes, or 4kB if all OSTs have 4kB AF sector drives), and then the OST could efficiently merge them if it has a larger sector size locally.
> 
> You prompted me to go on a search here -- turns out the 4k sectors are bumped up to 4160 bytes for T10 in Marten Peterson's original Linux patches, so 8 bytes of PI per 512 bytes, fixed, no matter if either sector size is used in the backend.
> 
> So complete layering horribleness is avoided, and only minor violation allowed to sneak in.  But at least Lustre doesn't have to care about it.

Excellent news. I have absolutely no problem with this, since a fixed 512-byte CRC size is just a parameter of the checksum algorithm.

It has the less serious drawback that it is a very small unit of currency for Lustre, so it will add the maximum overhead for every RPC without giving a strong checksum. It has the benefit that T10 is a standard that users (and hardware) understand. 

If we are going to the length of sending the GRD and REF tags for every sector, we may as well go whole-hog and allow for the APP tags to be sent as well (or at least conditionally pass them and set a flag on the RPC if the RPC growth seems prohibitive) so that if userspace ever gets its act together we don't have to change the protocol again in the future. I don't recall anymore if this was covered in your original design. 

Cheers, Andreas