<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Verification on multiple levels can help isolate where faults occur, but that's more for debugging/fixing purposes than end-to-end proof of correctness. "Calculated as early as possible, verified as late as possible" catches anything in-between. Of course, the verification must happen before the original data buffers are released, so the data can be re-sent.<div><br></div><div>From Martin Peterson's original T10 Linux patches:</div><div><pre style="color: rgb(17, 17, 17); background-color: rgb(242, 242, 242); font-family: 'bitstream vera sans mono', monaco, monospace; line-height: 1.4em; padding-top: 20px; padding-right: 20px; padding-bottom: 20px; padding-left: 20px; border-top-width: 1px; border-right-width: 1px; border-bottom-width: 1px; border-left-width: 1px; border-top-style: solid; border-right-style: solid; border-bottom-style: solid; border-left-style: solid; border-top-color: rgb(85, 85, 85); border-right-color: rgb(85, 85, 85); border-bottom-color: rgb(85, 85, 85); border-left-color: rgb(85, 85, 85); border-image: initial; letter-spacing: normal; font-size: 0.9em; font-style: normal; font-variant: normal; font-weight: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">The main idea of DIF/DIX is to allow integrity metadata to be
generated as close to the original data as possible.</pre><div><br></div></div><div><br><div><div>On Apr 13, 2012, at 10:06 PM, John Carrier wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Nathan,<br><br>I'm confused by your comment that "the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk."<br><br>I thought the appeal of T10 is that the intervening layers can verify the checksums as the data moves through them. The idea is to catch errors _before_ they land on the disk. <br><br>--jc<br><br>-----Original Message-----<br>From: <a href="mailto:twg-bounces@lists.opensfs.org">twg-bounces@lists.opensfs.org</a> [mailto:twg-bounces@lists.opensfs.org] On Behalf Of Nathan Rutman<br>Sent: Friday, April 13, 2012 4:27 PM<br>To: Andreas Dilger<br>Cc: Andrew Perepechko; <a href="mailto:twg@lists.opensfs.org">twg@lists.opensfs.org</a><br>Subject: Re: [Twg] T10 End-to-End Data Integrity HLD<br><br><br>On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote:<br><br><blockquote type="cite">On 2012-04-10, at 1:53 PM, Nathan Rutman wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Nathan,<br></blockquote><blockquote type="cite">thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly.<br></blockquote>I appreciate your time!<br><blockquote type="cite"><br></blockquote><blockquote type="cite">I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. <br></blockquote>I view this as different solutions to the problem, but I see no reason why they can't coexist.<br><br><blockquote type="cite">The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server.<br></blockquote>There is little consensus on what/who gets to use the APP tag; we were going to use it for our own purposes at the MDRAID layer.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors).<br></blockquote>From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely.<br><br>From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk.<br><br>Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired).<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size.<br></blockquote>This is a good point; I think a reasonable solution would require a common sector size within an OST.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Some other comments on the design:<br></blockquote><blockquote type="cite">6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the<br></blockquote><blockquote type="cite"> PG_writeback/page_mkwrite() support in newer kernels. It should<br></blockquote><blockquote type="cite"> no longer be possible for pages to change after calculating the<br></blockquote><blockquote type="cite"> checksum but before the RPC is sent. See LU-884 for details.<br></blockquote>Excellent!<br><br><blockquote type="cite">8.5.2: why not compute the CRC in the correct endianness in the first<br></blockquote><blockquote type="cite"> place? The kernel already has e.g. crc32_le() and crc32_be()<br></blockquote><blockquote type="cite"> routines, so it may also have the needed routines for T10-CRC.<br></blockquote><blockquote type="cite"> It also isn't clear whether CRC-then-swab is the same as doing<br></blockquote><blockquote type="cite"> the CRC in the other endianness in the first place? Typically,<br></blockquote><blockquote type="cite"> RPCs are in "receiver makes right" format, which reduces overhead<br></blockquote><blockquote type="cite"> in the common case that both ends are the same. However, if<br></blockquote><blockquote type="cite"> the underlying DIF API requires the CRC to always be big endian,<br></blockquote><blockquote type="cite"> it makes sense to swab it on the client already.<br></blockquote>Our conclusion as well - big endian everywhere.<br><br><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers, Andreas<br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite">Andreas Dilger Whamcloud, Inc.<br></blockquote><blockquote type="cite">Principal Lustre Engineer <a href="http://www.whamcloud.com/">http://www.whamcloud.com/</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote>_______________________________________________<br>twg mailing list<br><a href="mailto:twg@lists.opensfs.org">twg@lists.opensfs.org</a><br>http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org<br></div></blockquote></div><br></div></body></html>