[Twg] T10 End-to-End Data Integrity HLD

[Nathan Rutman]

Perhaps we can talk about this further at LUG, but I did want to answer a couple of things.

On Apr 18, 2012, at 2:55 PM, Andreas Dilger wrote:

	On 2012-04-17, at 11:02 AM, Nathan Rutman wrote:
	

		On Apr 17, 2012, at 12:06 AM, Andreas Dilger wrote:
		

			On 2012-04-13, at 4:26 PM, Nathan Rutman wrote:
			

				On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote:
				

					I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago.  
					


				I view this as different solutions to the problem, but I see no reason why they can't coexist.
				


			The difficulty with having two independent, but closely related features that solve the same general problem is that this introduces additional code that needs to be maintained indefinitely for the future.  I think that having separate checksum algorithms is fine, and possibly different transports for the checksums (if that is really needed, more below), but there should definitely only be a single framework for the checksum calculation and in-core storage that varies only by the selection of the algorithm.
			


		As far as Lustre is concerned, I view the T10 method as more of a out-of-band data transfer mechanism.  When you say single framework, you imply that the checksum is performed at the same point in the code path, but I think that is fundamentally different between Merkle tree/ZFS (in core) and T10 (in hardware).
		


	Sure, but the Merkle tree ends up being just a different checksum algorithm supported by the OST, that doesn't need any specific knowledge of the backing filesystem (blocksize, sector size, etc), and it doesn't impose any significant overhead on the protocol.  The only potential change outside of adding the algorithm would be to send a full 256-bit checksum instead of only a 32-bit one today.
	
	Exposing a low-level detail like the disk sector size to the Lustre client (or application, should it get that far) to compute the GRD tag is a terrible layering violation, IMHO.  This flies in the face of code abstraction and proper layering/isolation of different components in the storage stack, and is really contrary to the principles of Lustre that try to isolate the clients from internal details of the storage.  It doesn't make sense that a Lustre client doesn't need to know the block size or even the type of filesystem on the OST, yet it needs to know the sector size of the disks living some layers below each OST?
	
	Perhaps if there was a simple transformation for merging eight 512-byte GRDs into a single 4kB GRD, then at least the Lustre client could always compute and send the GRD tags for the smallest sector size for any OST it is connected to (512 bytes, if there are mixed sector sizes, or 4kB if all OSTs have 4kB AF sector drives), and then the OST could efficiently merge them if it has a larger sector size locally.
	


You prompted me to go on a search here -- turns out the 4k sectors are bumped up to 4160 bytes for T10 in Marten Peterson's original Linux patches, so 8 bytes of PI per 512 bytes, fixed, no matter if either sector size is used in the backend.

So complete layering horribleness is avoided, and only minor violation allowed to sneak in.  But at least Lustre doesn't have to care about it.


			
			

			It would be a strange software bug that generated new DIF checksums that were accepted by the hardware (for possibly corrupt data?) but also generated a valid hashed checksum to match the RPC.  Since the RPC hash is a direct hash of the DIF checksums, the only way I can see this happening (for any reasonable RPC hash function, with a 32-bit or 256-bit RPC hash) is if the 16-bit DIF hash on the corrupt data matched the original value, in which case DIF would have missed the corruption as well...
			


		I can give you a trivial example: bad code in the Merkle tree implementation always generates a checksum of 0xDEADBEEF.  Client computes it, server computes it, hey they match, and individual sector recomputed CRC's can still be correct, but the data do not match.
		


	Can you give me a non-trivial example? :-) 

Sure: bad Merkle tree implementation provides poor coverage of the available hash space, leading to potential missed detection of lower-level T10 errors. 


	 This same argument (though admittedly a strange bug :-) could be used for the non-Merkle checksums, and we are no further ahead.  

To me, this is the point of transporting the unmolested checksum all the way to disk -- presumably the hardware calcs are simple and correct, and the user app is responsible for the other side, and nobody else's software has to be trusted at all.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120418/1e112476/attachment.html>