From chris at whamcloud.com Tue Apr 3 00:26:57 2012 From: chris at whamcloud.com (Chris Gearing) Date: Tue, 03 Apr 2012 01:26:57 +0100 Subject: [Twg] Lustre-devel Digest, Vol 72, Issue 17 In-Reply-To: References: Message-ID: <4F7A43D1.5000209@whamcloud.com> Hi Roman, > Problem 1 > > Currently Lustre and test are living in one code space and build in one > time,and often have specific dependencies between test and code. > > This situation directly affect > > 1) interoperability testing between different version. (because testing > is started from client which have different test framework then server > and client remotely execute test framework as their own. just copying > tests for equalization could not work with big difference between versions) > > 2) it is not simple execute(especially in automation) testing for test. > F.e. a bug is fixed, the test on it added. Executing the test on an old > revision(probably on a previous release) should show failed test result. > But with big difference between versions where fixed and where execute > test-framework can fail to start. > > Possible solution: split Lustre and lustre tests in code and build > levels. It means that lustre and tests will not be connected on code > revision, only by logic, f.e. via keywords. Also should be added in same > time an abstraction level in test framework which allows to execute > lustre utils from different version of lustre. The situation here is exactly the same as exists with the source code. When we run interop tests the test system runs test scripts belonging to the server version against those belonging to the client version. So we might use 1.8.7 client scripts against 2.2 server scripts. These scripts need to inter-operate in exactly the same way that the Lustre source code itself needs to interoperate. If people find cases where this does not happen then they raise bugs against Lustre and these are fixed using the same processes as any other Lustre bug. This means people across the community investing effort to rectify the incompatibilities and then posting patches for review, test and inclusion in the product. Of course in a perfect world we would be back at the point where 1.8.7 and 2.x forked and never allow the interoperability issues in the test scripts to creep in, but the reality is the issues do exist and effort does needs to be spent resolving them. I am 100% sure that the resolution of the relatively few incompatibilities is infinitely easier than attempting to take the 1.8, 2.1 and 2.2 test scripts and produce an independent one size fits all solution. We currently track each failing test in Jira, see here http://jira.whamcloud.com/browse/LU-1193 for an example. If others find issues in their testing then they should create Jira issues to track them and if possible post patches to resolve them. > > > Problem 2 > > (to avoid term problems, I call there: sanity = test suite, 130 = test, > 130c and 130a = test cases) > > Different test cases, ended with letter(f.e. 130c), have an different > idea of dependencies. Some test cases have dependences to previous test > cases, some don't have. > > All they now can be executed with "ONLY" parameter and all they have > separated item in result yaml file as just separated tests( which > doesn't have test cases ended with letter, f.e. sanity 129). Also, tests > which have testcases and don't have their own body can be execute with > ONLY parameter( but doesn't have their special result). > > So, logically, all test which can be execute via using ONLY must be not > depended to other tests. But we have test which depended. Moreover, some > developers prefer to consider testcases as step of full one test. > > What is entities which I call "testcases" and "test" from your point of > view? > > Answer of this question affect automated test execution and test > development, and maybe ask some test-framework changes. > I think you highlight a very good point here that we don't really know enough about the test contents, their prerequisites or other dependencies. I would suggest that many attempts have been made over the years to use naming conventions, numeric ordering or other similar mechanisms to track such behaviour. What we need to make sense of the 1000+ test cases we have is an extensible knowledge-base of information that can grow overtime to become a rich source of information that can be used to allow automated systems as well as developers to confidently use the tests in the most flexible way possible. Because of the nature of Lustre we need to find a way that keeps this knowledge within the public domain, provides for use to expand the range of things we store about each test and provides for both people and machines to access it with equal ease. One reasonable proposal is to add a comment block at the start of each test script and subtest within that script that lists the test name, short and long description that includes what the test is supposed to be doing, what bug (if any) it was originally added for, what part of the code it is intended to cover, prerequisites (filesystem initialization, min/max number of clients, OSTs, MDTs it can test with, etc) in a machine readable format that it not only documents the test today but that can be expanded in the future. Once we have an agreement on an initial format for this comment block, the development community can work to populate it for each subtest and improve the understanding and usefulness of all existing tests. Thanks Chris Gearing Sr. Software Engineer Quality Engineering Whamcloud Inc From Roman_Grigoryev at xyratex.com Tue Apr 3 06:07:00 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Tue, 03 Apr 2012 10:07:00 +0400 Subject: [Twg] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7A43D1.5000209@whamcloud.com> References: <4F7A43D1.5000209@whamcloud.com> Message-ID: <4F7A9384.80707@xyratex.com> Hi Chris, Thank you for answer ( I have cut part of my original message): >> Problem 1 >> >> Currently Lustre and test are living in one code space and build in one >> time,and often have specific dependencies between test and code. ...... >> Possible solution: split Lustre and lustre tests in code and build >> levels. It means that lustre and tests will not be connected on code >> revision, only by logic, f.e. via keywords. Also should be added in same >> time an abstraction level in test framework which allows to execute >> lustre utils from different version of lustre. > The situation here is exactly the same as exists with the source code. > When we run interop tests the test system runs test scripts belonging to > the server version against those belonging to the client version. So we > might use 1.8.7 client scripts against 2.2 server scripts. These scripts > need to inter-operate in exactly the same way that the Lustre source > code itself needs to interoperate. Yes, it is. But I don't see why we should use old test base for interoperability testing? Between 1.8.7 and 2.x tests was fixed and also as test framework was changed. For getting same test coverage for old features we should backport new fixes in test to old (maybe already frozen) code. Also, as results, we have different tests sets for compatibility testing. For 1.8.7 it will one, for 2.1 - other. Only a part of differences shows difference between code base for one feature set. (F.e. we see on special 1.8.7 branch failures which already fixed in 2.x code.) > > If people find cases where this does not happen then they raise bugs > against Lustre and these are fixed using the same processes as any other > Lustre bug. This means people across the community investing effort to > rectify the incompatibilities and then posting patches for review, test > and inclusion in the product. Of course in a perfect world we would be > back at the point where 1.8.7 and 2.x forked and never allow the > interoperability issues in the test scripts to creep in, but the reality > is the issues do exist and effort does needs to be spent resolving them. > I am 100% sure that the resolution of the relatively few > incompatibilities is infinitely easier than attempting to take the 1.8, > 2.1 and 2.2 test scripts and produce an independent one size fits all > solution. > I agree that there is a lot of work on test framework abstraction layer also on adding simple filtering procedure and setting keywords. But having one test base give us good possibility just setup an actual test package on all nodes and have somehow predict executed test set. Also, in this letter below, we discuss test/test framework changes. If we will have separated tests between version package then benefits from new tests descriptors we be accessible for all previous system too. In other case we could create incompatible improvements. > We currently track each failing test in Jira, see here > http://jira.whamcloud.com/browse/LU-1193 for an example. If others find > issues in their testing then they should create Jira issues to track > them and if possible post patches to resolve them. Thanks for this link. Do you have any keyword or attribute for interoperability failures? > >> >> >> Problem 2 >> >> (to avoid term problems, I call there: sanity = test suite, 130 = test, >> 130c and 130a = test cases) >> ... >> Answer of this question affect automated test execution and test >> development, and maybe ask some test-framework changes. >> > I think you highlight a very good point here that we don't really know > enough about the test contents, their prerequisites or other > dependencies. I would suggest that many attempts have been made over the > years to use naming conventions, numeric ordering or other similar > mechanisms to track such behaviour. > > What we need to make sense of the 1000+ test cases we have is an > extensible knowledge-base of information that can grow overtime to > become a rich source of information that can be used to allow automated > systems as well as developers to confidently use the tests in the most > flexible way possible. > > Because of the nature of Lustre we need to find a way that keeps this > knowledge within the public domain, provides for use to expand the range > of things we store about each test and provides for both people and > machines to access it with equal ease. > > One reasonable proposal is to add a comment block at the start of each > test script and subtest within that script that lists the test name, > short and long description that includes what the test is supposed to be > doing, what bug (if any) it was originally added for, what part of the > code it is intended to cover, prerequisites (filesystem initialization, > min/max number of clients, OSTs, MDTs it can test with, etc) in a > machine readable format that it not only documents the test today but > that can be expanded in the future. I agree, it is very important to separating meta information and test body. Internally in Xyratex, we use external scripts and descriptors which somehow add same possibility(per-test timeouts, keywords...). > > Once we have an agreement on an initial format for this comment block, > the development community can work to populate it for each subtest and > improve the understanding and usefulness of all existing tests. I absolutely agree that we need agreement to start any work on test improvements. How can we initiate this process? Maybe good first step is creating glossary to use and terms and based on these terms fix tests? Also, what do you think about a possible simple solutions for decreasing dependence problem which is currently pretty painful for us: 1) test(test scenario) must have only number name (1,2,3..110...999) 2) test cases (test step) must have number+char index (1f,2,b...99c) Only Test can be executed via ONLY. Test cases can be execute only as part of test. Thanks, Roman From chris at whamcloud.com Tue Apr 3 14:21:36 2012 From: chris at whamcloud.com (Chris Gearing) Date: Tue, 03 Apr 2012 15:21:36 +0100 Subject: [Twg] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7A9384.80707@xyratex.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> Message-ID: <4F7B0770.8050308@whamcloud.com> On 03/04/2012 07:07, Roman Grigoryev wrote: > Hi Chris, > Thank you for answer ( I have cut part of my original message): >> When we run interop tests the test system runs test scripts belonging to >> the server version against those belonging to the client version. So we >> might use 1.8.7 client scripts against 2.2 server scripts. These scripts >> need to inter-operate in exactly the same way that the Lustre source >> code itself needs to interoperate. > Yes, it is. But I don't see why we should use old test base for > interoperability testing? Between 1.8.7 and 2.x tests was fixed and also > as test framework was changed. For getting same test coverage for old > features we should backport new fixes in test to old (maybe already > frozen) code. > Also, as results, we have different tests sets for compatibility > testing. For 1.8.7 it will one, for 2.1 - other. Only a part of > differences shows difference between code base for one feature set. > (F.e. we see on special 1.8.7 branch failures which already fixed in 2.x > code.) > We don't have a single script because the tests are at times very tightly coupled to the Lustre version. There were a lot of changes between 1.8.x and 2.x and a lot of corresponding changes to the test scripts. Where the tests are the same and bugs were found in the 2.x test scripts these should have been backported to the 1.8.x test scripts if this was not done then we should do it for inclusion into the 1.8.8 release. The notion of making 'master' scripts work with with all versions is obviously possible but it is a very significant task and given that the scripts themselves are written in a language (sic) that does not provide structure a single script strategy is likely to create many more 'interoperability issues' than it fixes. Also it's worth considering that we have best part of a 1000 discrete changes, whenever a test is re-engineered the test itself must be proven to detect failure as well as success. i.e. If someone produced a version independent test set that passed all versions we would not know that the process was a success, we would need to check that each re-engineered test 'failed' appropriately for each Lustre version, this is a big task that I doubt can be properly achieved in bash. So in summary the best solution given what we have today is to back port fixes to the test scripts as we back port fixes to the code. This is an investment in time and requires the same discipline to test as we have for coding. A single set of scripts that caters for all versions appears I believe like an easy solution but actually would require huge investment that would be better spent developing a modern test framework and infrastructure that can support Lustre for the next ten years. > >>> >>> Problem 2 >>> >>> (to avoid term problems, I call there: sanity = test suite, 130 = test, >>> 130c and 130a = test cases) >>> > ... > >>> Answer of this question affect automated test execution and test >>> development, and maybe ask some test-framework changes. >>> >> I think you highlight a very good point here that we don't really know >> enough about the test contents, their prerequisites or other >> dependencies. I would suggest that many attempts have been made over the >> years to use naming conventions, numeric ordering or other similar >> mechanisms to track such behaviour. >> ... >> One reasonable proposal is to add a comment block at the start of each >> test script and subtest within that script that lists the test name, >> short and long description that includes what the test is supposed to be >> doing, what bug (if any) it was originally added for, what part of the >> code it is intended to cover, prerequisites (filesystem initialization, >> min/max number of clients, OSTs, MDTs it can test with, etc) in a >> machine readable format that it not only documents the test today but >> that can be expanded in the future. > I agree, it is very important to separating meta information and test body. > Internally in Xyratex, we use external scripts and descriptors which > somehow add same possibility(per-test timeouts, keywords...). > >> Once we have an agreement on an initial format for this comment block, >> the development community can work to populate it for each subtest and >> improve the understanding and usefulness of all existing tests. > I absolutely agree that we need agreement to start any work on test > improvements. How can we initiate this process? Maybe good first step is > creating glossary to use and terms and based on these terms fix tests? > > Also, what do you think about a possible simple solutions for decreasing > dependence problem which is currently pretty painful for us: > > 1) test(test scenario) must have only number name (1,2,3..110...999) > 2) test cases (test step) must have number+char index (1f,2,b...99c) > > Only Test can be executed via ONLY. > Test cases can be execute only as part of test. I don't think there is a problem with this simple solution in that it does no harm as long as you applied any changes to all the branches that are applicable. At the same time I will draft a possible meta data format that includes the extensible metadata within the source in a way that maximizes its value both today and in the future, we can then review, revise and then agree that format on Lustre-Devel, although I'll mail you privately so you can have input before that. It may actually be the case that some work has occurred on this topic previously and if so we can leverage that. Thanks Chris Gearing Sr. Software Engineer Quality Engineering Whamcloud Inc From carrier at cray.com Wed Apr 4 07:09:31 2012 From: carrier at cray.com (John Carrier) Date: Wed, 4 Apr 2012 07:09:31 +0000 Subject: [Twg] TWG meeting minutes for 2012-03-29 Message-ID: Attached are the minutes from our 3/29/2012 meeting. We had a few new requirements introduced this call. Please take a look at the notes and post corrections/questions/comments to discuss at lists.opensfs.org. Our next meeting is this Thursday, 4/5/2012 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 2012-03-29 Concall: start : 9:30a PT, adjourn : 10:30a PT Next meeting: Thursday, 04/05/2012 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Attending: Name Organization email ----------------- -------------- ---------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Joshua Walgenbach IU jjw at indiana.edu Chris Morrone LLNL morrone2 at llnl.gov Shay Seager OpenSFS shay at opensfs.org Eric Barton Whamcloud eeb at whamcloud.com Bryon Neitzel Whamcloud bryon at whamcloud.com Agenda: discuss requirements Status: These are the requirements we've accumulated so far this year: - improve single client performance - improve storage pool management to address : OST rebalancing dynamic layout async mirroring, etc - dynamic LNET configuration - extend POSIX for Lustre access patterns to address : file sets scalable opens - reduce Lustre's dependence on timeouts to address : health networks LUG F2F: At the end of the meeting we discussed the process for getting a requirements list completed by our F2F meeting at LUG (4/25). We agreed that meeting weekly is that right way to move forward so that we complete the process by late before LUG. John and Dave to update the requirements doc with new features. Xyratex, Cray, LLNL, Whamcloud planning to attend the F2F meetings on Wednesday. A number of us will also be attending the benchmarking WG on Sunday. Please remember that DDN & Whamcloud are having a reception Sunday night before LUG starts. Discussion: * Shared file performance - Henry Newman (via email) - N-1 I/O is much less than N-N * Administrative shutdown - EricB raised this issue - Lustre needs a safe server shutdown to ensure that clients flush state to disk. - such a capability will be necessary for transparent server version upgrades * Improve configuration robustness - Need to make file system configuration robust in the face failures - ChrisM related problems he recently had at LLNL trying to add OSTs to an existing file system. The failure to make the addition required that he hand-edit the MGS log files. - EricB said we should anticipate similar errors with DNE when we expand the number of MDTs in the namespace. For example, consider what might happen when allowing directories on MDTs not rooted to MDT0. - ChrisM believes there are timebombs remaining in the code and we'll need to pay the technical debt to remove them. - We also discussed how Lustre could benefit from dynamic configuration. For example, imperative recovery introduces the need for dynamic registration with MGS. We could use this to remove static configuration and use the registration to define the config. * code stability - With all the architectural changes landing on Lustre in the next year (HSM, DNE, OSD, etc), there is concern we may be changing too much and should spend time on clean-up and stability - We also know that the layout locks introduce with the HSM will need to be reworked so that they can be re-used for OST rebalancing and other migration tasks for files not at rest. - Lustre reliance on timeouts also contribute significantly to system instability or, at least, the perception that they system is unstable because of periodic pauses. The health network should be a high priority to remove timeouts and make client/server interactions more deterministic. Eric's current thinking is that a spanning tree with a bullet-proof root would allow resilient, scalable distribution of global information. This project will require a formal architecture and prototype before completion. - ChrisM mentioned that /proc doesn't have sufficient locks. Cat'ing /proc can crash the node. He isn't sure where the problem is, but bugs like this make Lustre appear to be unstable. * LNET credit configuration - ChrisM suggested that we need to revisit LNET credits. His experience is that transmit credits complicate LNET traffic between networks. He thinks LNET router nodes should never have transmit credit limits since the limited resource is the buffers. The problem is espeically pronounced when there are multiple network levels. - Eric suggested having high and low priority message queues. This could be part of a project to improve network robustness. His original design used multiple routers to guarantee connectivity, but performance tanks when there are large numbers of routers. - Improving this aspect of the LNET network interface is separate from and orthogonal to the previous discussions about LNET (ie dynamic LNET configuration and LNET channel bonding). Chris thinks that improving LNET robustness may require a wire protocol change. - We also should identify poorly behaving routers so that they can be removed from the network to improve performance. * other performance improvements - As mentioned, there are a number of MDS improvements in the works: pdirops - single directory smp scaling - fatter servers DNE - horizontal scalability stat ahead / glimpse ahead - Discussed but not yet implemented is size on metadata which has an issue with evictions caused when caching data on OSTs when evicted from the MDS. * storage pool management : standard target - A standard target would remove the distinction between the MDT and OSTs. One outcome is that the same target could be used by both the MDS and OSS. As a result, could put small directories on an MDT to improve performance since I/O could complete in a single roundtrip. - Putting small I/O on the MDT would improve performance and dynamic layout would allow migration from MDT to OST. - Changing striping for the MDT would create giga+ sytle directories - Eric says he has a 2-pater for storage pool management describes the benefits, components and phases for the work. TWG could use this as basis for its report to the OpenSFS board. adjourn: 10:30a PT From Nathan_Rutman at xyratex.com Wed Apr 4 21:24:01 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 4 Apr 2012 14:24:01 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7B0770.8050308@whamcloud.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1091C@XYUS-EX22.xyus.xyratex.com> On Apr 3, 2012, at 7:21 AM, Chris Gearing wrote: > On 03/04/2012 07:07, Roman Grigoryev wrote: >> Hi Chris, >> Thank you for answer ( I have cut part of my original message): >>> When we run interop tests the test system runs test scripts belonging to >>> the server version against those belonging to the client version. So we >>> might use 1.8.7 client scripts against 2.2 server scripts. These scripts >>> need to inter-operate in exactly the same way that the Lustre source >>> code itself needs to interoperate. >> Yes, it is. But I don't see why we should use old test base for >> interoperability testing? Between 1.8.7 and 2.x tests was fixed and also >> as test framework was changed. For getting same test coverage for old >> features we should backport new fixes in test to old (maybe already >> frozen) code. >> Also, as results, we have different tests sets for compatibility >> testing. For 1.8.7 it will one, for 2.1 - other. Only a part of >> differences shows difference between code base for one feature set. >> (F.e. we see on special 1.8.7 branch failures which already fixed in 2.x >> code.) >> > We don't have a single script because the tests are at times very > tightly coupled to the Lustre version. There were a lot of changes > between 1.8.x and 2.x and a lot of corresponding changes to the test > scripts. Where the tests are the same and bugs were found in the 2.x > test scripts these should have been backported to the 1.8.x test scripts > if this was not done then we should do it for inclusion into the 1.8.8 > release. > > The notion of making 'master' scripts work with with all versions is > obviously possible but it is a very significant task and given that the > scripts themselves are written in a language (sic) that does not provide > structure a single script strategy is likely to create many more > 'interoperability issues' than it fixes. > > Also it's worth considering that we have best part of a 1000 discrete > changes, whenever a test is re-engineered the test itself must be proven > to detect failure as well as success. i.e. If someone produced a version > independent test set that passed all versions we would not know that the > process was a success, we would need to check that each re-engineered > test 'failed' appropriately for each Lustre version, this is a big task > that I doubt can be properly achieved in bash. > > So in summary the best solution given what we have today is to back port > fixes to the test scripts as we back port fixes to the code. This is an > investment in time and requires the same discipline to test as we have > for coding. A single set of scripts that caters for all versions appears > I believe like an easy solution but actually would require huge > investment that would be better spent developing a modern test framework > and infrastructure that can support Lustre for the next ten years. I agree on this last point -- is that something that OpenSFS should spearhead? Roman has pointed out some of the limitations with the current test framework, Robert Read has pointed out the poor/redundant coverage of many of the existing tests; is it time to start from scratch? Is there a more evolutionary approach we can/should use? > >> >>>> >>>> Problem 2 >>>> >>>> (to avoid term problems, I call there: sanity = test suite, 130 = test, >>>> 130c and 130a = test cases) >>>> >> ... >> >>>> Answer of this question affect automated test execution and test >>>> development, and maybe ask some test-framework changes. >>>> >>> I think you highlight a very good point here that we don't really know >>> enough about the test contents, their prerequisites or other >>> dependencies. I would suggest that many attempts have been made over the >>> years to use naming conventions, numeric ordering or other similar >>> mechanisms to track such behaviour. >>> ... >>> One reasonable proposal is to add a comment block at the start of each >>> test script and subtest within that script that lists the test name, >>> short and long description that includes what the test is supposed to be >>> doing, what bug (if any) it was originally added for, what part of the >>> code it is intended to cover, prerequisites (filesystem initialization, >>> min/max number of clients, OSTs, MDTs it can test with, etc) in a >>> machine readable format that it not only documents the test today but >>> that can be expanded in the future. >> I agree, it is very important to separating meta information and test body. >> Internally in Xyratex, we use external scripts and descriptors which >> somehow add same possibility(per-test timeouts, keywords...). >> >>> Once we have an agreement on an initial format for this comment block, >>> the development community can work to populate it for each subtest and >>> improve the understanding and usefulness of all existing tests. >> I absolutely agree that we need agreement to start any work on test >> improvements. How can we initiate this process? Maybe good first step is >> creating glossary to use and terms and based on these terms fix tests? >> >> Also, what do you think about a possible simple solutions for decreasing >> dependence problem which is currently pretty painful for us: >> >> 1) test(test scenario) must have only number name (1,2,3..110...999) >> 2) test cases (test step) must have number+char index (1f,2,b...99c) >> >> Only Test can be executed via ONLY. >> Test cases can be execute only as part of test. > I don't think there is a problem with this simple solution in that it > does no harm as long as you applied any changes to all the branches that > are applicable. At the same time I will draft a possible meta data > format that includes the extensible metadata within the source in a way > that maximizes its value both today and in the future, we can then > review, revise and then agree that format on Lustre-Devel, although I'll > mail you privately so you can have input before that. It may actually be > the case that some work has occurred on this topic previously and if so > we can leverage that. > > Thanks > > Chris Gearing > Sr. Software Engineer > Quality Engineering > Whamcloud Inc > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From Alexander_Lezhoev at xyratex.com Thu Apr 5 08:40:16 2012 From: Alexander_Lezhoev at xyratex.com (Alexander Lezhoev) Date: Thu, 05 Apr 2012 12:40:16 +0400 Subject: [Twg] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7B0770.8050308@whamcloud.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> Message-ID: <4F7D5A70.8040006@xyratex.com> Hi Chris, I completely agree that the test-framework must be significantly revised. We have some plans to improve it and I think, it would be useful to share our ideas and visions of this task. Really we considered the separation of tests and Lustre code as a part of the framework improvement. So I think, we need to share our approaches and develop a conception which would be satisfactory for all. What do you think about opening a discussion about new test-framework? -- Alexander Lezhoev. Morpheus test team. Xyratex. On 04/03/2012 06:21 PM, Chris Gearing wrote: > > We don't have a single script because the tests are at times very > tightly coupled to the Lustre version. There were a lot of changes > between 1.8.x and 2.x and a lot of corresponding changes to the test > scripts. Where the tests are the same and bugs were found in the 2.x > test scripts these should have been backported to the 1.8.x test > scripts if this was not done then we should do it for inclusion into > the 1.8.8 release. > > The notion of making 'master' scripts work with with all versions is > obviously possible but it is a very significant task and given that > the scripts themselves are written in a language (sic) that does not > provide structure a single script strategy is likely to create many > more 'interoperability issues' than it fixes. > > Also it's worth considering that we have best part of a 1000 discrete > changes, whenever a test is re-engineered the test itself must be > proven to detect failure as well as success. i.e. If someone produced > a version independent test set that passed all versions we would not > know that the process was a success, we would need to check that each > re-engineered test 'failed' appropriately for each Lustre version, > this is a big task that I doubt can be properly achieved in bash. > > So in summary the best solution given what we have today is to back > port fixes to the test scripts as we back port fixes to the code. This > is an investment in time and requires the same discipline to test as > we have for coding. A single set of scripts that caters for all > versions appears I believe like an easy solution but actually would > require huge investment that would be better spent developing a modern > test framework and infrastructure that can support Lustre for the next > ten years. > From bzzz.tomas at gmail.com Thu Apr 5 10:38:30 2012 From: bzzz.tomas at gmail.com (Alex Tomas) Date: Thu, 5 Apr 2012 14:38:30 +0400 Subject: [Twg] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7D5A70.8040006@xyratex.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> <4F7D5A70.8040006@xyratex.com> Message-ID: while technically this is possible, the framework will have to support different versions of Lustre because in some cases they implement functionality different ways with different side effects (often used in t-f). which kind of contradicts with the original idea ? thanks, Alex On Thu, Apr 5, 2012 at 12:40 PM, Alexander Lezhoev wrote: > Hi Chris, > > I completely agree that the test-framework must be significantly revised. We > have some plans to improve it and I think, it would be useful to share our > ideas and visions of this task. Really we considered the separation of tests > and Lustre code as a part of the framework improvement. So I think, we need > to share our approaches and develop a conception which would be satisfactory > for all. What do you think about opening a discussion about new > test-framework? > > -- > Alexander Lezhoev. > Morpheus test team. > Xyratex. > > > > > On 04/03/2012 06:21 PM, Chris Gearing wrote: >> >> >> We don't have a single script because the tests are at times very tightly >> coupled to the Lustre version. There were a lot of changes between 1.8.x and >> 2.x and a lot of corresponding changes to the test scripts. Where the tests >> are the same and bugs were found in the 2.x test scripts these should have >> been backported to the 1.8.x test scripts if this was not done then we >> should do it for inclusion into the 1.8.8 release. >> >> The notion of making 'master' scripts work with with all versions is >> obviously possible but it is a very significant task and given that the >> scripts themselves are written in a language (sic) that does not provide >> structure a single script strategy is likely to create many more >> 'interoperability issues' than it fixes. >> >> Also it's worth considering that we have best part of a 1000 discrete >> changes, whenever a test is re-engineered the test itself must be proven to >> detect failure as well as success. i.e. If someone produced a version >> independent test set that passed all versions we would not know that the >> process was a success, we would need to check that each re-engineered test >> 'failed' appropriately for each Lustre version, this is a big task that I >> doubt can be properly achieved in bash. >> >> So in summary the best solution given what we have today is to back port >> fixes to the test scripts as we back port fixes to the code. This is an >> investment in time and requires the same discipline to test as we have for >> coding. A single set of scripts that caters for all versions appears I >> believe like an easy solution but actually would require huge investment >> that would be better spent developing a modern test framework and >> infrastructure that can support Lustre for the next ten years. >> > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From chris at whamcloud.com Mon Apr 9 00:04:09 2012 From: chris at whamcloud.com (Chris Gearing) Date: Mon, 9 Apr 2012 01:04:09 +0100 Subject: [Twg] [Lustre-devel] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <73AED5C780AE05478241DB067651A92102F1091C@XYUS-EX22.xyus.xyratex.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1091C@XYUS-EX22.xyus.xyratex.com> Message-ID: Hi Nathan, Please excuse the lack of included context but I think it's fair to say that the current test-framework and scripts are at the end of their evolutionary life and that whilst they will be required to fulfil a role in test for the foreseeable future what is required is a vastly more capable and scalable approach to testing Lustre in particular and massively parallel exascale file systems in general. Thanks Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Mon Apr 9 22:54:13 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 9 Apr 2012 15:54:13 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1091C@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10939@XYUS-EX22.xyus.xyratex.com> Agreed. We started a conversation on the last TWG call about this; I was waiting for John Carrier's meeting minutes to carry on that conversation. On Apr 8, 2012, at 5:04 PM, Chris Gearing wrote: Hi Nathan, Please excuse the lack of included context but I think it's fair to say that the current test-framework and scripts are at the end of their evolutionary life and that whilst they will be required to fulfil a role in test for the foreseeable future what is required is a vastly more capable and scalable approach to testing Lustre in particular and massively parallel exascale file systems in general. Thanks Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Tue Apr 10 19:53:43 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 10 Apr 2012 12:53:43 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD Message-ID: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> Hi all, I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: HighleveldesignofLustreEnd-to-enddataintegrity.pdf Type: application/pdf Size: 624097 bytes Desc: HighleveldesignofLustreEnd-to-enddataintegrity.pdf URL: From morrone2 at llnl.gov Wed Apr 11 02:32:17 2012 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 10 Apr 2012 19:32:17 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> Message-ID: <4F84ED31.6050804@llnl.gov> How will this work with OSD? How does this solution compare to the long standing plan to use merkle trees for end-to-end checksums? On 04/10/2012 12:53 PM, Nathan Rutman wrote: > Hi all, > I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. > > From carrier at cray.com Wed Apr 11 14:03:56 2012 From: carrier at cray.com (John Carrier) Date: Wed, 11 Apr 2012 14:03:56 +0000 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <4F84ED31.6050804@llnl.gov> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <4F84ED31.6050804@llnl.gov> Message-ID: I thought Sun's Merkle tree solution was tied to ZFS. I don't see any requirement of the backend file system in this doc. Given the OSD work being done at LLNL to separate the Lustre servers from the details of the backend storage, we will need to add DIX to OSD so that there is a mechanism for passing the protection information between Lustre and the disk file system. In fact, adding OSD support will be required for whatever end-to-end data integrity solution we devise, whether it is T10PI or Sun's Merkle tree or something else. Otherwise we violate the clean layering OSD will provide. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Christopher J. Morrone Sent: Tuesday, April 10, 2012 7:32 PM To: Nathan Rutman Cc: twg at lists.opensfs.org; Andrew Perepechko Subject: Re: [Twg] T10 End-to-End Data Integrity HLD How will this work with OSD? How does this solution compare to the long standing plan to use merkle trees for end-to-end checksums? On 04/10/2012 12:53 PM, Nathan Rutman wrote: > Hi all, > I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. > > _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From carrier at cray.com Wed Apr 11 14:23:12 2012 From: carrier at cray.com (John Carrier) Date: Wed, 11 Apr 2012 14:23:12 +0000 Subject: [Twg] TWG meeting minutes for 2012-04-05 In-Reply-To: References: Message-ID: Attached are the minutes from our 4/5/2012 meeting.   We discussed requirements for re-evaluating the Lustre test suite during the call.  Please take a look at the notes and post corrections/questions/comments to discuss at lists.opensfs.org.  Our next meeting is this Thursday, 4/12/2012 @ 9:30a PDT/12:30p EDT    Dial-in numbers are 1-715-726-4994 or 1-866-304-8294    Meeting ID and password are 72090 Thanks, --jc -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 2012-04-05 Concall: start : 9:30a PT, adjourn : 10:30a PT Next meeting: Thursday, 04/12/2012 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Attending: Name Organization email ----------------- ------------ ------------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Joshua Walgenbach IU jjw at indiana.edu Chris Morrone LLNL morrone2 at llnl.gov Mahmoud Hanafi NASA Ames mahmoud.hanafi at nasa.gov Andreas Dilger Whamcloud adilger at whamcloud.com Bryon Neitzel Whamcloud bryon at whamcloud.com Roman Grigoryev Xyratex roman_grigoryev at xyratex.com Alexander Lezhoev Xyratex alexander_lezhoev at xyratex.com Nathan Rutman Xyratex nathan_rutman at xyratex.com Cheng Shao Xyratex cheng_shao at xyratex.com Agenda: discuss requirements discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. 2) create single test suite Each release has its own test suite and assumes that the client and server will use the same test version. It is becoming common today to have different client/server versions. In this case, it is unclear which tests from which release package should be used. Instead, we should have a single test suite that accommodates different combinations of client and server versions. 3) extract individual tests from the scripts Many of the tests are run as a suite. There needs to be some mechanism to allow running components of the suite independently. To do this, we will need to make clear the dependencies of the subtests so that users know a test can be run only after another component of the test has executed. Also mentioned was the need for a common header on all test results to facilitate post-processing. 4) client failure shouldn't stop the tests The goal is never to have test failures, nonethelees failures currently can stall test progress. The suites need to be written to allow for client failures. 5) increase code coverage There are new tests that we should consider adding to the Lustre suite in order to increase code coverage. Suggestions were xfstest and some of the MPI tests. Going Forward We then discussed the process for moving the requirements list forward toward funding new projects. John and Dave have been trying to follow the steps we took last year. We need to update the requirements doc to include all the features addressed this year. From this list, the TWG will create a short list that we ask our organizations to rank. This list and the TWG's ranking would then be forwarded to the Board for consideration of funding. Chris suggested formalizing the process on the reflector. We agreed that discussion of the list can be done by email, but there is still value in the weekly group concalls. From Nathan_Rutman at xyratex.com Wed Apr 11 15:09:34 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 11 Apr 2012 08:09:34 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <4F84ED31.6050804@llnl.gov> Message-ID: <73AED5C780AE05478241DB067651A92102F10947@XYUS-EX22.xyus.xyratex.com> On Apr 11, 2012, at 7:03 AM, John Carrier wrote: > I thought Sun's Merkle tree solution was tied to ZFS. That's right. The ptlrpc change will be flexible enough to describe multiple schemes. > > I don't see any requirement of the backend file system in this doc. Given the OSD work being done at LLNL to separate the Lustre servers from the details of the backend storage, we will need to add DIX to OSD so that there is a mechanism for passing the protection information between Lustre and the disk file system. > > In fact, adding OSD support will be required for whatever end-to-end data integrity solution we devise, whether it is T10PI or Sun's Merkle tree or something else. Otherwise we violate the clean layering OSD will provide. This is very true. I will ask Andrew to look into the current Orion code as well and propose interface changes if needed. > > > --jc > > > -----Original Message----- > From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Christopher J. Morrone > Sent: Tuesday, April 10, 2012 7:32 PM > To: Nathan Rutman > Cc: twg at lists.opensfs.org; Andrew Perepechko > Subject: Re: [Twg] T10 End-to-End Data Integrity HLD > > How will this work with OSD? > > How does this solution compare to the long standing plan to use merkle > trees for end-to-end checksums? > > > On 04/10/2012 12:53 PM, Nathan Rutman wrote: >> Hi all, >> I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. >> >> > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From morrone2 at llnl.gov Wed Apr 11 20:57:32 2012 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Wed, 11 Apr 2012 13:57:32 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F10947@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <4F84ED31.6050804@llnl.gov> <73AED5C780AE05478241DB067651A92102F10947@XYUS-EX22.xyus.xyratex.com> Message-ID: <4F85F03C.5000302@llnl.gov> On 04/11/2012 08:09 AM, Nathan Rutman wrote: > > On Apr 11, 2012, at 7:03 AM, John Carrier wrote: > >> I thought Sun's Merkle tree solution was tied to ZFS. > That's right. The ptlrpc change will be flexible enough to describe multiple schemes. The merkle tree solution was certainly motivated by ZFS, but that doesn't mean that it can't be a generic solution, does it? Perhaps we we make the checksum algorithm negotiable, and the base leaf size as well. But the T10 solution did not sound very general on first read. It also seems to not mention dealing with things like synchronous partial block writes, which the merkle tree design seems to address. Do we really want to expose the lustre clients to the details of on-disk storage? I'm not fully up to date on T10, and quick web search didn't answer it for me. Does this HLD allow 4K sectors? The HLD only seems to talk about 512-byte sectors. It sounds like the padding for 4K sectors is enough that maybe there are just 8 512-byte checksums concatenated, but I don't know if that is the way it is done, or whether the whole 4K sector is checksummed as a single block. Chris From Nathan_Rutman at xyratex.com Wed Apr 11 21:18:14 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 11 Apr 2012 14:18:14 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <4F85F03C.5000302@llnl.gov> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <4F84ED31.6050804@llnl.gov> <73AED5C780AE05478241DB067651A92102F10947@XYUS-EX22.xyus.xyratex.com> <4F85F03C.5000302@llnl.gov> Message-ID: <73AED5C780AE05478241DB067651A92102F10950@XYUS-EX22.xyus.xyratex.com> On Apr 11, 2012, at 1:57 PM, Christopher J. Morrone wrote: > On 04/11/2012 08:09 AM, Nathan Rutman wrote: >> >> On Apr 11, 2012, at 7:03 AM, John Carrier wrote: >> >>> I thought Sun's Merkle tree solution was tied to ZFS. >> That's right. The ptlrpc change will be flexible enough to describe multiple schemes. > > The merkle tree solution was certainly motivated by ZFS, but that doesn't mean that it can't be a generic solution, does it? The major motivation was that ZFS already does this, so there's no cost to the server to do it at the Lustre level instead. Without ZFS, it's certainly still usable, but not a clear win like it is for ZFS. Incidentally, that same motivation (no cost to the server) is the one of the same reasons we're following the T10 path. There should be room for multiple options here. > Perhaps we we make the checksum algorithm negotiable, and the base leaf size as well. Yes, our intent was to allow a variety of data integrity possibilities in the ptlrpc layer. > > But the T10 solution did not sound very general on first read. It also seems to not mention dealing with things like synchronous partial block writes, which the merkle tree design seems to address. Version mirroring component addresses current RAID limitations -- I'll speak to this at the LUG. > > Do we really want to expose the lustre clients to the details of on-disk storage? Not just the lustre clients, but potentially the application level as well. This is the philosophy behind T10-DIX, anyhow: "the data I wrote and checksummed myself is the exact same data that got written to the disk sector". That said, I look at our project more as facilitating T10-DIX use, rather than tightly integrating it in the Lustre levels. Lustre more or less is just passing the info along. > > I'm not fully up to date on T10, and quick web search didn't answer it for me. Does this HLD allow 4K sectors? The HLD only seems to talk about 512-byte sectors. It sounds like the padding for 4K sectors is enough that maybe there are just 8 512-byte checksums concatenated, but I don't know if that is the way it is done, or whether the whole 4K sector is checksummed as a single block. The latter, and it results in a lower coverage of the checksum for 4k sectors, but it doesn't change the design. > > Chris From Nathan_Rutman at xyratex.com Wed Apr 11 23:00:40 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 11 Apr 2012 16:00:40 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing improvements (was Lustre-devel Digest, Vol 72, Issue 17) In-Reply-To: <4F7B0770.8050308@whamcloud.com> References: <4F7A43D1.5000209@whamcloud.com> <4F7A9384.80707@xyratex.com> <4F7B0770.8050308@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10954@XYUS-EX22.xyus.xyratex.com> On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would like to address. discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. 2) create single test suite Each release has its own test suite and assumes that the client and server will use the same test version. It is becoming common today to have different client/server versions. In this case, it is unclear which tests from which release package should be used. Instead, we should have a single test suite that accommodates different combinations of client and server versions. 3) extract individual tests from the scripts Many of the tests are run as a suite. There needs to be some mechanism to allow running components of the suite independently. To do this, we will need to make clear the dependencies of the subtests so that users know a test can be run only after another component of the test has executed. Also mentioned was the need for a common header on all test results to facilitate post-processing. 4) client failure shouldn't stop the tests The goal is never to have test failures, nonethelees failures currently can stall test progress. The suites need to be written to allow for client failures. 5) increase code coverage There are new tests that we should consider adding to the Lustre suite in order to increase code coverage. Suggestions were xfstest and some of the MPI tests. Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture? On Apr 8, 2012, at 5:04 PM, Chris Gearing wrote: Hi Nathan, Please excuse the lack of included context but I think it's fair to say that the current test-framework and scripts are at the end of their evolutionary life and that whilst they will be required to fulfil a role in test for the foreseeable future what is required is a vastly more capable and scalable approach to testing Lustre in particular and massively parallel exascale file systems in general. Thanks Chris On Apr 3, 2012, at 7:21 AM, Chris Gearing wrote: On 03/04/2012 07:07, Roman Grigoryev wrote: Hi Chris, Thank you for answer ( I have cut part of my original message): When we run interop tests the test system runs test scripts belonging to the server version against those belonging to the client version. So we might use 1.8.7 client scripts against 2.2 server scripts. These scripts need to inter-operate in exactly the same way that the Lustre source code itself needs to interoperate. Yes, it is. But I don't see why we should use old test base for interoperability testing? Between 1.8.7 and 2.x tests was fixed and also as test framework was changed. For getting same test coverage for old features we should backport new fixes in test to old (maybe already frozen) code. Also, as results, we have different tests sets for compatibility testing. For 1.8.7 it will one, for 2.1 - other. Only a part of differences shows difference between code base for one feature set. (F.e. we see on special 1.8.7 branch failures which already fixed in 2.x code.) We don't have a single script because the tests are at times very tightly coupled to the Lustre version. There were a lot of changes between 1.8.x and 2.x and a lot of corresponding changes to the test scripts. Where the tests are the same and bugs were found in the 2.x test scripts these should have been backported to the 1.8.x test scripts if this was not done then we should do it for inclusion into the 1.8.8 release. The notion of making 'master' scripts work with with all versions is obviously possible but it is a very significant task and given that the scripts themselves are written in a language (sic) that does not provide structure a single script strategy is likely to create many more 'interoperability issues' than it fixes. Also it's worth considering that we have best part of a 1000 discrete changes, whenever a test is re-engineered the test itself must be proven to detect failure as well as success. i.e. If someone produced a version independent test set that passed all versions we would not know that the process was a success, we would need to check that each re-engineered test 'failed' appropriately for each Lustre version, this is a big task that I doubt can be properly achieved in bash. So in summary the best solution given what we have today is to back port fixes to the test scripts as we back port fixes to the code. This is an investment in time and requires the same discipline to test as we have for coding. A single set of scripts that caters for all versions appears I believe like an easy solution but actually would require huge investment that would be better spent developing a modern test framework and infrastructure that can support Lustre for the next ten years. Problem 2 (to avoid term problems, I call there: sanity = test suite, 130 = test, 130c and 130a = test cases) ... Answer of this question affect automated test execution and test development, and maybe ask some test-framework changes. I think you highlight a very good point here that we don't really know enough about the test contents, their prerequisites or other dependencies. I would suggest that many attempts have been made over the years to use naming conventions, numeric ordering or other similar mechanisms to track such behaviour. ... One reasonable proposal is to add a comment block at the start of each test script and subtest within that script that lists the test name, short and long description that includes what the test is supposed to be doing, what bug (if any) it was originally added for, what part of the code it is intended to cover, prerequisites (filesystem initialization, min/max number of clients, OSTs, MDTs it can test with, etc) in a machine readable format that it not only documents the test today but that can be expanded in the future. I agree, it is very important to separating meta information and test body. Internally in Xyratex, we use external scripts and descriptors which somehow add same possibility(per-test timeouts, keywords...). Once we have an agreement on an initial format for this comment block, the development community can work to populate it for each subtest and improve the understanding and usefulness of all existing tests. I absolutely agree that we need agreement to start any work on test improvements. How can we initiate this process? Maybe good first step is creating glossary to use and terms and based on these terms fix tests? Also, what do you think about a possible simple solutions for decreasing dependence problem which is currently pretty painful for us: 1) test(test scenario) must have only number name (1,2,3..110...999) 2) test cases (test step) must have number+char index (1f,2,b...99c) Only Test can be executed via ONLY. Test cases can be execute only as part of test. I don't think there is a problem with this simple solution in that it does no harm as long as you applied any changes to all the branches that are applicable. At the same time I will draft a possible meta data format that includes the extensible metadata within the source in a way that maximizes its value both today and in the future, we can then review, revise and then agree that format on Lustre-Devel, although I'll mail you privately so you can have input before that. It may actually be the case that some work has occurred on this topic previously and if so we can leverage that. Thanks Chris Gearing Sr. Software Engineer Quality Engineering Whamcloud Inc _______________________________________________ Lustre-devel mailing list Lustre-devel at lists.lustre.org http://lists.lustre.org/mailman/listinfo/lustre-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Apr 12 06:42:23 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 12 Apr 2012 00:42:23 -0600 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> Message-ID: <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> On 2012-04-10, at 1:53 PM, Nathan Rutman wrote: > I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. Nathan, thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly. I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. Some other comments on the design: 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the PG_writeback/page_mkwrite() support in newer kernels. It should no longer be possible for pages to change after calculating the checksum but before the RPC is sent. See LU-884 for details. 8.5.2: why not compute the CRC in the correct endianness in the first place? The kernel already has e.g. crc32_le() and crc32_be() routines, so it may also have the needed routines for T10-CRC. It also isn't clear whether CRC-then-swab is the same as doing the CRC in the other endianness in the first place? Typically, RPCs are in "receiver makes right" format, which reduces overhead in the common case that both ends are the same. However, if the underlying DIF API requires the CRC to always be big endian, it makes sense to swab it on the client already. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From carrier at cray.com Thu Apr 12 06:54:54 2012 From: carrier at cray.com (John Carrier) Date: Thu, 12 Apr 2012 06:54:54 +0000 Subject: [Twg] FW: T10 End-to-End Data Integrity HLD Message-ID: Reposting Andreas' reply to include discuss at lists.opensfs.org too. -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Andreas Dilger Sent: Wednesday, April 11, 2012 11:42 PM To: Nathan Rutman Cc: twg at lists.opensfs.org; Andrew Perepechko Subject: Re: [Twg] T10 End-to-End Data Integrity HLD On 2012-04-10, at 1:53 PM, Nathan Rutman wrote: > I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. Nathan, thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly. I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. Some other comments on the design: 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the PG_writeback/page_mkwrite() support in newer kernels. It should no longer be possible for pages to change after calculating the checksum but before the RPC is sent. See LU-884 for details. 8.5.2: why not compute the CRC in the correct endianness in the first place? The kernel already has e.g. crc32_le() and crc32_be() routines, so it may also have the needed routines for T10-CRC. It also isn't clear whether CRC-then-swab is the same as doing the CRC in the other endianness in the first place? Typically, RPCs are in "receiver makes right" format, which reduces overhead in the common case that both ends are the same. However, if the underlying DIF API requires the CRC to always be big endian, it makes sense to swab it on the client already. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From carrier at cray.com Thu Apr 12 09:30:45 2012 From: carrier at cray.com (John Carrier) Date: Thu, 12 Apr 2012 09:30:45 +0000 Subject: [Twg] new requirements document Message-ID: I have published a draft of the TWG requirements on google docs: https://docs.google.com/document/d/1gXs0VPaCNEU3l9OuXYmBCyxKEm3dkIV60smCVAD5Un8/edit This is my attempt to merge the requirements we discussed this year (as recorded in the meeting minutes) with our features list from last year. Please note that this is a working document. At this point, Dave and I have the pen, but we will gladly add editors who volunteer. Otherwise, anyone with access to the link can read the document and provide comments to the reflector. As discussed at last week's TWG meeting, I would like the TWG to review this document and select a list of 5-6 requirements that we will rank and recommend to the OpenSFS board for funding. I propose reviewing the document this week and then generating the list at _next_ week's meeting so that we can discuss ranking at the F2F in Austin. Please send your comments to the discuss reflector (discuss at lists.opensfs.org). --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at whamcloud.com Thu Apr 12 13:33:21 2012 From: chris at whamcloud.com (Chris) Date: Thu, 12 Apr 2012 14:33:21 +0100 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: References: Message-ID: <4F86D9A1.8090105@whamcloud.com> Hi, > On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would > like to address. > > discussion: > Xyratex started by suggesting that the test suite included with every > Lustre release needed reworking. This led to a group conversation > that focused on the following areas: > > 1) refactor unused tests > Test scripts have grown unwieldly and do not form a coherent > test package. As a result, there are some tests that are > used regularly and others that ineffective and skipped. We > should remove the unused tests. Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass! I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed. It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled. > 2) create single test suite > Each release has its own test suite and assumes that the client > and server will use the same test version. It is becoming common > today to have different client/server versions. In this case, it > is unclear which tests from which release package should be used. > Instead, we should have a single test suite that accommodates > different combinations of client and server versions. What is the problem statement here that leads to the conclusion that a single test suite should be produced? It can't be interoperability because the suites do run together ( b1_8 with b2_x ) and where interoperability issues exist tickets are raised and the issues resolved. It's worth noting that the tests behind the link do not perform significantly worse than a straight master test cycle and that most of the issues are Lustre not test-interoperability. Perhaps a complete problem statement will allow us to understand what the ROI would be from a single test suite. > 3) extract individual tests from the scripts > Many of the tests are run as a suite. There needs to be some > mechanism to allow running components of the suite independently. > To do this, we will need to make clear the dependencies of the > subtests so that users know a test can be run only after another > component of the test has executed. Also mentioned was the need > for a common header on all test results to facilitate > post-processing. What is to be achieved here that ONLY/EXCEPT does not allow, using a mixture of ONLY and EXCEPT any combination of tests can be run. On the topic of meta data I think placing machine readable extensible data in a comments section above every test to store this data would make a lot of sense. We probably need nothing more than a set of key:value pairings. If people have seen good working solutions that should be considered it would be useful if they posted them. > 4) client failure shouldn't stop the tests > The goal is never to have test failures, nonethelees failures > currently can stall test progress. The suites need to be written > to allow for client failures. Can someone elaborate on what 'client failure' means, perhaps it is obvious but from my experience test failures do not stop the test suites. Can I also say that the goal is not to never have test failures, because if that were true we would just have no tests. The goal is to have a set of tests that detect and flag every possible short coming in the operation of Lustre, this goal would then mean that Lustre would need to be perfect to pass test. > 5) increase code coverage > There are new tests that we should consider adding to the Lustre > suite in order to increase code coverage. Suggestions were xfstest > and some of the MPI tests. Before we can increase code coverage we need to systematically understand what coverage we have today, this might well be aided by creating the appropriate metadata in 3 (above). Once we have a measurement we can objectively plan an increase of the coverage, and then measure our effectiveness in the future. > Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture? > My statement was not bold it was just a statement of fact. The current test-framework is not a suitable platform for fulfilling the requirements of tomorrow. We have to be building for the next 10 years which will include exascale, does anybody really feel that the current bash based, un-scalable and relatively unstructured test facility that we have today is that platform? Thanks Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu Apr 12 16:36:30 2012 From: carrier at cray.com (John Carrier) Date: Thu, 12 Apr 2012 16:36:30 +0000 Subject: [Twg] TWG concall problems Message-ID: I am not able to login to the concall I created for our TWG meeting today and am having trouble establishing a VPN with the Cray network to re-establish the call information. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Apr 12 16:41:35 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 12 Apr 2012 10:41:35 -0600 Subject: [Twg] [Discuss] TWG meeting minutes for 2012-04-05 In-Reply-To: References: Message-ID: <80559687-C9A5-4FF8-91A3-EA2339ED536E@whamcloud.com> On 2012-04-11, at 8:23 AM, John Carrier wrote: > Attached are the minutes from our 4/5/2012 meeting. We discussed requirements for re-evaluating the Lustre test suite during the call. Please take a look at the notes and post corrections/questions/comments to discuss at lists.opensfs.org. > > Our next meeting is this Thursday, 4/12/2012 @ 9:30a PDT/12:30p EDT > Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 > Meeting ID and password are 72090 Anyone else having trouble dialing into this call? The 72090 code isn't working for me... Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From aik at fnal.gov Thu Apr 12 16:39:49 2012 From: aik at fnal.gov (Alex Kulyavtsev) Date: Thu, 12 Apr 2012 11:39:49 -0500 Subject: [Twg] [Discuss] TWG meeting minutes for 2012-04-05 In-Reply-To: <80559687-C9A5-4FF8-91A3-EA2339ED536E@whamcloud.com> References: <80559687-C9A5-4FF8-91A3-EA2339ED536E@whamcloud.com> Message-ID: <83F5C074-5129-4D1B-9383-D6E997CE014F@fnal.gov> Same for me. Alex. On Apr 12, 2012, at 11:41 AM, Andreas Dilger wrote: > Anyone else having trouble dialing into this call? The 72090 code > isn't working for me... -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjw at iu.edu Thu Apr 12 16:46:30 2012 From: jjw at iu.edu (Joshua Walgenbach) Date: Thu, 12 Apr 2012 12:46:30 -0400 Subject: [Twg] IU conference bridge Message-ID: <4F8706E6.3060205@iu.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Simms set up an IU conference bridge if we want to use it as an alternative: 812-856-7060 22618# - -Josh -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+HBuUACgkQcqyJPuRTYp98ewCcCAjE6F9UbSi89MiXZbcEBzGW Gu8An034yxbPxYGRJnfLLNs/7RpBV/qC =hfyl -----END PGP SIGNATURE----- From carrier at cray.com Thu Apr 12 16:48:26 2012 From: carrier at cray.com (John Carrier) Date: Thu, 12 Apr 2012 16:48:26 +0000 Subject: [Twg] TWG concall problems In-Reply-To: References: Message-ID: We're back in business. If you are still available, the Cray bridge is active. Thanks all to offering the other bridges. --jc Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of John Carrier Sent: Thursday, April 12, 2012 9:37 AM To: twg at lists.opensfs.org; discuss at lists.opensfs.org Subject: [Twg] TWG concall problems I am not able to login to the concall I created for our TWG meeting today and am having trouble establishing a VPN with the Cray network to re-establish the call information. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Thu Apr 12 20:29:57 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Thu, 12 Apr 2012 13:29:57 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <4F86D9A1.8090105@whamcloud.com> References: <4F86D9A1.8090105@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> On Apr 12, 2012, at 6:33 AM, Chris wrote: Hi, On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would like to address. discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass! I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed. It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled. There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run. 2) create single test suite Each release has its own test suite and assumes that the client and server will use the same test version. It is becoming common today to have different client/server versions. In this case, it is unclear which tests from which release package should be used. Instead, we should have a single test suite that accommodates different combinations of client and server versions. What is the problem statement here that leads to the conclusion that a single test suite should be produced? The test suite from 1.8.7 is not the same as from 2.2. Therefore, when running the tests on a 1.8.7 client talking to a 2.2 server, which versions' test suite should be run? An independent test RPM, divorced from the Lustre versioning, would provide a definitive answer. Yes, I know the tests change with code changes; they should change in the "master" test repo, and a mechanism should be implemented to skip known failures with older versions of non-interop (e.g. 1.8.7-1.8.7) and interop cases. This mechanism should probably involve pairs of server-client versions and simple lists of tests. This also solves the problem of having to backport each test to the old test suite as new tests are developed. It can't be interoperability because the suites do run together ( b1_8 with b2_x ) and where interoperability issues exist tickets are raised and the issues resolved. It's worth noting that the tests behind the link do not perform significantly worse than a straight master test cycle and that most of the issues are Lustre not test-interoperability. Perhaps a complete problem statement will allow us to understand what the ROI would be from a single test suite. 3) extract individual tests from the scripts Many of the tests are run as a suite. There needs to be some mechanism to allow running components of the suite independently. To do this, we will need to make clear the dependencies of the subtests so that users know a test can be run only after another component of the test has executed. Also mentioned was the need for a common header on all test results to facilitate post-processing. What is to be achieved here that ONLY/EXCEPT does not allow, using a mixture of ONLY and EXCEPT any combination of tests can be run. That would be the ideal. Unfortunately, it simply is not the case today. Plenty of tests depend utterly on the tests before them doing some kind of setup or "first half" of a test. Definition of what constitutes a "test" and a "sub-test" muddies the issue somewhat. On the topic of meta data I think placing machine readable extensible data in a comments section above every test to store this data would make a lot of sense. We probably need nothing more than a set of key:value pairings. If people have seen good working solutions that should be considered it would be useful if they posted them. Yes, I'm a big proponent of YAML. Beyond this keyword descriptions of the test would allow automatic testing of specific areas, "acl" or "fopen" or "quotas" etc. 4) client failure shouldn't stop the tests The goal is never to have test failures, nonethelees failures currently can stall test progress. The suites need to be written to allow for client failures. Can someone elaborate on what 'client failure' means, perhaps it is obvious but from my experience test failures do not stop the test suites. Some tests hang or crash the client machine running the test script. An external mechanism is needed in that case to power cycle the machine and restart the tests. Architectural redesign of the framework deployment should recognize and avoid this problem. Can I also say that the goal is not to never have test failures, because if that were true we would just have no tests. Not test failure, client failure as above. The goal is to have a set of tests that detect and flag every possible short coming in the operation of Lustre, this goal would then mean that Lustre would need to be perfect to pass test. 5) increase code coverage There are new tests that we should consider adding to the Lustre suite in order to increase code coverage. Suggestions were xfstest and some of the MPI tests. Before we can increase code coverage we need to systematically understand what coverage we have today, this might well be aided by creating the appropriate metadata in 3 (above). Once we have a measurement we can objectively plan an increase of the coverage, and then measure our effectiveness in the future. Excellent point. Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture? My statement was not bold it was just a statement of fact. The current test-framework is not a suitable platform for fulfilling the requirements of tomorrow. We have to be building for the next 10 years which will include exascale, does anybody really feel that the current bash based, un-scalable and relatively unstructured test facility that we have today is that platform? I think not. The discussion here is a starting point specifically for identifying the current problems, with the hope of driving a an evolutionary way of getting there. Perhaps via OpenSFS we can drive this as a prioritized task of broad interest. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at whamcloud.com Fri Apr 13 13:29:07 2012 From: chris at whamcloud.com (Chris) Date: Fri, 13 Apr 2012 14:29:07 +0100 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> Message-ID: <4F882A23.7050301@whamcloud.com> Nathan, > > discussion: Xyratex started by suggesting that the test suite included > with every Lustre release needed reworking. This led to a group > conversation that focused on the following areas: 1) refactor unused > tests Test scripts have grown unwieldly and do not form a coherent > test package. As a result, there are some tests that are used > regularly and others that ineffective and skipped. We should remove > the unused tests. >> Removing tests that are genuinely ineffective is a good idea, but we >> must be careful because historically tests have been skipped for many >> reasons including the non-function of a useful test, or even the fact >> that Lustre could not be made to pass! I think this refactor process >> should be framed more as an examination of currently skipped steps to >> understand which should be fixed+re-enabled and which should be removed. >> >> It's also worth noting that the ratio of 'used regularly' to skipped >> is probably 50:1, skipped tests have no runtime cost and so the ROI >> is probably not significant. In fact the ROI is probably only going >> to be worthwhile if skipped tests can be made useful and re-enabled. > There's also a component here of removing redundant tests -- Robert > Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs > SLOW=YES as virtually identical, but YES takes hours to run. I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly. I would also suggest that focusing on increasing test coverage and quality will create greater return then removing redundant tests, although if someone wanted to spend time seeking out the redundant tests it would not do any harm. > >> >>> 2) create single test suite >>> Each release has its own test suite and assumes that the client >>> and server will use the same test version. It is becoming common >>> today to have different client/server versions. In this case, it >>> is unclear which tests from which release package should be used. >>> Instead, we should have a single test suite that accommodates >>> different combinations of client and server versions. >> What is the problem statement here that leads to the conclusion that >> a single test suite should be produced? > The test suite from 1.8.7 is not the same as from 2.2. Therefore, > when running the tests on a 1.8.7 client talking to a 2.2 server, > which versions' test suite should be run? An independent test RPM, > divorced from the Lustre versioning, would provide a definitive > answer. Yes, I know the tests change with code changes; they should > change in the "master" test repo, and a mechanism should be > implemented to skip known failures with older versions of non-interop > (e.g. 1.8.7-1.8.7) and interop cases. This mechanism should probably > involve pairs of server-client versions and simple lists of tests. > This also solves the problem of having to backport each test to the > old test suite as new tests are developed. > In your example you would use 1.8.7 test scripts on the client, and 2.2 test scripts on the server. When the client makes an RPC to the server script it is making a functional request, the receiving script then performs that function in a way applicable to the server version in question. Whilst keeping the scripts separate does require back porting it also removes the need for the already unwieldy scripts to contain code to make them version aware. I would say with some certainty that separate scripts with explicit back-porting achieves a more reliable result than attempting to extend the bash scripts to cover multiple versions. Also I don't think we could merge 47000 lines of bash, and then verify results. Verification would require that we see each test pass and fail. We would also be unable to remove and verify the 1.8 parts of the new common test-framework once 1.8 becomes an unsupported product, which will happen one day. Don't get me wrong, if the test-framework was written in a modern, structured and object orientated language a single validation and verification framework might be possible, but today we don't have that. >> >> It can't be interoperability because the suites do run together ( >> b1_8 with b2_x >> >> ) and where interoperability issues exist tickets are raised and the >> issues resolved. It's worth noting that the tests behind the link do >> not perform significantly worse than a straight master test cycle and >> that most of the issues are Lustre not test-interoperability. >> >> Perhaps a complete problem statement will allow us to understand what >> the ROI would be from a single test suite. >>> 3) extract individual tests from the scripts >>> Many of the tests are run as a suite. There needs to be some >>> mechanism to allow running components of the suite independently. >>> To do this, we will need to make clear the dependencies of the >>> subtests so that users know a test can be run only after another >>> component of the test has executed. Also mentioned was the need >>> for a common header on all test results to facilitate >>> post-processing. >> What is to be achieved here that ONLY/EXCEPT does not allow, using a >> mixture of ONLY and EXCEPT any combination of tests can be run. > That would be the ideal. Unfortunately, it simply is not the case > today. Plenty of tests depend utterly on the tests before them doing > some kind of setup or "first half" of a test. Definition of what > constitutes a "test" and a "sub-test" muddies the issue somewhat. I think there is confusion here about dependencies and extracting the tests. If we extracted the tests the dependencies would still exist, and if we understood and solved the dependencies of each test then ONLY/EXCEPT would allow us to do exactly what was needed. So we are in agreement that we need to understand and document using meta-data the dependencies, but I don't really understand what extracting the tests from the scripts really buys us. I don't think it does harm, if the tools are put in place to allow developers to still run 'suites', but it certainly doesn't solve the dependency issue either. > >> >> On the topic of meta data I think placing machine readable extensible >> data in a comments section above every test to store this data would >> make a lot of sense. We probably need nothing more than a set of >> key:value pairings. If people have seen good working solutions that >> should be considered it would be useful if they posted them. > Yes, I'm a big proponent of YAML. Beyond this keyword descriptions of > the test would allow automatic testing of specific areas, "acl" or > "fopen" or "quotas" etc. I'll try an propose something to this list. > Obviously this is not as bold as Chris' statement, which, by the way, > I am happy to entertain as well. Do others have anything to add to the > list above, or thoughts on proceeding with a completely new architecture? >> My statement was not bold it was just a statement of fact. The >> current test-framework is not a suitable platform for fulfilling the >> requirements of tomorrow. We have to be building for the next 10 >> years which will include exascale, does anybody really feel that the >> current bash based, un-scalable and relatively unstructured test >> facility that we have today is that platform? > I think not. The discussion here is a starting point specifically for > identifying the current problems, with the hope of driving a an > evolutionary way of getting there. Perhaps via OpenSFS we can drive > this as a prioritized task of broad interest. > We can all agree that we don't have the luxury of a big-bang change over to a new method, but evolution in this case needs to be a new species that will overtime become dominant and eventually replace the current bashous-thalensis! Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From rread at whamcloud.com Fri Apr 13 17:49:10 2012 From: rread at whamcloud.com (Robert Read) Date: Fri, 13 Apr 2012 10:49:10 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <4F882A23.7050301@whamcloud.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> Message-ID: <020C5B51-75A4-4E05-BF67-714E0BD21BA5@whamcloud.com> Hi, On Apr 13, 2012, at 6:29 , Chris wrote: > Nathan, > >> >> discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. >>> >>> Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass! I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed. >>> >>> It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled. >> There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run. > I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly. I don't have that data anymore, but it is true that we saw virtually no difference in static code coverage with SLOW=yes. However we did see significantly more bugs with SLOW=yes, and so we always did testing in slow mode. Also, to give credit where it's due, Li Wei did the actual coverage analysis. robert From Nathan_Rutman at xyratex.com Fri Apr 13 18:03:15 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 13 Apr 2012 11:03:15 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <020C5B51-75A4-4E05-BF67-714E0BD21BA5@whamcloud.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <020C5B51-75A4-4E05-BF67-714E0BD21BA5@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10963@XYUS-EX22.xyus.xyratex.com> On Apr 13, 2012, at 10:49 AM, Robert Read wrote: Hi, On Apr 13, 2012, at 6:29 , Chris wrote: Nathan, discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass! I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed. It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled. There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run. I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly. I don't have that data anymore, but it is true that we saw virtually no difference in static code coverage with SLOW=yes. However we did see significantly more bugs with SLOW=yes, and so we always did testing in slow mode. http://wiki.lustre.org/images/e/ee/QualityInitiativeTalk.pdf Also, to give credit where it's due, Li Wei did the actual coverage analysis. robert -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Fri Apr 13 23:26:49 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 13 Apr 2012 16:26:49 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: > On 2012-04-10, at 1:53 PM, Nathan Rutman wrote: >> I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. > > Nathan, > thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly. I appreciate your time! > > I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. I view this as different solutions to the problem, but I see no reason why they can't coexist. > The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. There is little consensus on what/who gets to use the APP tag; we were going to use it for our own purposes at the MDRAID layer. > > One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). >From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. >From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. > > There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. > > Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). > > The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. This is a good point; I think a reasonable solution would require a common sector size within an OST. > > Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. > > The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. > > Some other comments on the design: > 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the > PG_writeback/page_mkwrite() support in newer kernels. It should > no longer be possible for pages to change after calculating the > checksum but before the RPC is sent. See LU-884 for details. Excellent! > 8.5.2: why not compute the CRC in the correct endianness in the first > place? The kernel already has e.g. crc32_le() and crc32_be() > routines, so it may also have the needed routines for T10-CRC. > It also isn't clear whether CRC-then-swab is the same as doing > the CRC in the other endianness in the first place? Typically, > RPCs are in "receiver makes right" format, which reduces overhead > in the common case that both ends are the same. However, if > the underlying DIF API requires the CRC to always be big endian, > it makes sense to swab it on the client already. Our conclusion as well - big endian everywhere. > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > From carrier at cray.com Sat Apr 14 05:06:41 2012 From: carrier at cray.com (John Carrier) Date: Sat, 14 Apr 2012 05:06:41 +0000 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> Message-ID: Nathan, I'm confused by your comment that "the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk." I thought the appeal of T10 is that the intervening layers can verify the checksums as the data moves through them. The idea is to catch errors _before_ they land on the disk. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Nathan Rutman Sent: Friday, April 13, 2012 4:27 PM To: Andreas Dilger Cc: Andrew Perepechko; twg at lists.opensfs.org Subject: Re: [Twg] T10 End-to-End Data Integrity HLD On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: > On 2012-04-10, at 1:53 PM, Nathan Rutman wrote: >> I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. > > Nathan, > thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly. I appreciate your time! > > I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. I view this as different solutions to the problem, but I see no reason why they can't coexist. > The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. There is little consensus on what/who gets to use the APP tag; we were going to use it for our own purposes at the MDRAID layer. > > One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). >From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. >From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. > > There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. > > Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). > > The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. This is a good point; I think a reasonable solution would require a common sector size within an OST. > > Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. > > The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. > > Some other comments on the design: > 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the > PG_writeback/page_mkwrite() support in newer kernels. It should > no longer be possible for pages to change after calculating the > checksum but before the RPC is sent. See LU-884 for details. Excellent! > 8.5.2: why not compute the CRC in the correct endianness in the first > place? The kernel already has e.g. crc32_le() and crc32_be() > routines, so it may also have the needed routines for T10-CRC. > It also isn't clear whether CRC-then-swab is the same as doing > the CRC in the other endianness in the first place? Typically, > RPCs are in "receiver makes right" format, which reduces overhead > in the common case that both ends are the same. However, if > the underlying DIF API requires the CRC to always be big endian, > it makes sense to swab it on the client already. Our conclusion as well - big endian everywhere. > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From Nathan_Rutman at xyratex.com Mon Apr 16 16:56:10 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 16 Apr 2012 09:56:10 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10971@XYUS-EX22.xyus.xyratex.com> Verification on multiple levels can help isolate where faults occur, but that's more for debugging/fixing purposes than end-to-end proof of correctness. "Calculated as early as possible, verified as late as possible" catches anything in-between. Of course, the verification must happen before the original data buffers are released, so the data can be re-sent. >From Martin Peterson's original T10 Linux patches: The main idea of DIF/DIX is to allow integrity metadata to be generated as close to the original data as possible. On Apr 13, 2012, at 10:06 PM, John Carrier wrote: Nathan, I'm confused by your comment that "the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk." I thought the appeal of T10 is that the intervening layers can verify the checksums as the data moves through them. The idea is to catch errors _before_ they land on the disk. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Nathan Rutman Sent: Friday, April 13, 2012 4:27 PM To: Andreas Dilger Cc: Andrew Perepechko; twg at lists.opensfs.org Subject: Re: [Twg] T10 End-to-End Data Integrity HLD On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: On 2012-04-10, at 1:53 PM, Nathan Rutman wrote: I'm posting our T10 HLD here for public review. I'll be talking about this at LUG, but if you want a sneak peak or more detail, read on. Nathan, thanks for posting this. I've been meaning to reply (I'm swamped with other things) but thought it important to get back to you quickly. I appreciate your time! I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. I view this as different solutions to the problem, but I see no reason why they can't coexist. The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. There is little consensus on what/who gets to use the APP tag; we were going to use it for our own purposes at the MDRAID layer. One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. This is a good point; I think a reasonable solution would require a common sector size within an OST. Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. Some other comments on the design: 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the PG_writeback/page_mkwrite() support in newer kernels. It should no longer be possible for pages to change after calculating the checksum but before the RPC is sent. See LU-884 for details. Excellent! 8.5.2: why not compute the CRC in the correct endianness in the first place? The kernel already has e.g. crc32_le() and crc32_be() routines, so it may also have the needed routines for T10-CRC. It also isn't clear whether CRC-then-swab is the same as doing the CRC in the other endianness in the first place? Typically, RPCs are in "receiver makes right" format, which reduces overhead in the common case that both ends are the same. However, if the underlying DIF API requires the CRC to always be big endian, it makes sense to swab it on the client already. Our conclusion as well - big endian everywhere. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Tue Apr 17 07:06:18 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 17 Apr 2012 00:06:18 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> Message-ID: <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> On 2012-04-13, at 4:26 PM, Nathan Rutman wrote: > On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: >> I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. > > I view this as different solutions to the problem, but I see no reason why they can't coexist. The difficulty with having two independent, but closely related features that solve the same general problem is that this introduces additional code that needs to be maintained indefinitely for the future. I think that having separate checksum algorithms is fine, and possibly different transports for the checksums (if that is really needed, more below), but there should definitely only be a single framework for the checksum calculation and in-core storage that varies only by the selection of the algorithm. >> The only potential area of difficulty would be around the APP tags (if those are being used by the client/application at all) since they cannot be regenerated at the server. > > There is little consensus on what/who gets to use the APP tag; we were going to use it for our own purposes at the MDRAID layer. That definitely simplifies things, since it means that the APP tags do not need to be sent from the client at all, since they would just be discarded by the server and replaced by the MDRAID layer. >> One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). > > From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. That is potentially true, but if the checksums are not verified until you get down to the disk, if there ever is a real checksum error you won't have any idea where the error is being introduced. I thought the same thing as John - that the appeal of standardized T10 DIF is that it can be verified all along the IO path to isolate the failing components. > From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. I don't see this argument. Having the OST verify the checksum from the network does not invalidate the additional checking down at the controller and/or disk layer (if there actually are disks that validate DIF, I'm not sure). If the OST is regenerating the individual DIF sector checksums for verification by the hardware based on a per-RPC checksum (which has itself been verified to match the RPC checksum), then there is still end-to-end checksum coverage. It would be a strange software bug that generated new DIF checksums that were accepted by the hardware (for possibly corrupt data?) but also generated a valid hashed checksum to match the RPC. Since the RPC hash is a direct hash of the DIF checksums, the only way I can see this happening (for any reasonable RPC hash function, with a 32-bit or 256-bit RPC hash) is if the 16-bit DIF hash on the corrupt data matched the original value, in which case DIF would have missed the corruption as well... I can buy the argument that doing the DIF data checksum on the OST consumes CPU cycles, but I had always imagined that it would do this anyway for data/checksum verification as soon as possible instead of waiting until the IO makes it to the controller. That would ensure that the data the RPC put into the OST cache is valid. Waiting for the controller to be the first to do the verification precludes ever having writeback caching on the OST. A writeback cached RPC would be completed before the data was verified by the controller, and it is not possible (today, at least) for the OST to force the client to resend just that RPC/data for replay. Any checksum error caused by network errors would require the OST to reset and start recovery for all RPCs in order to get the client to resend the failed RPC. > Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. > >> There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. >> >> Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). >> >> The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. > > This is a good point; I think a reasonable solution would require a common sector size within an OST. I hope that it would always to be the case that the disks in a single OST have the same sector size, but my concern was more about different sector sizes between OSTs. In that case, because the data checksum would be dependent on the sector size on each OST, the client per-page checksums would need to be computed based on the OST on which each page was to be written, which adds complexity into the IO path. In some sense this is no worse than having different checksum algorithms on different OSTs, but that would largely be an administrator choice that would otherwise default to using the same checksum algorithm for all OSTs. In contrast, different sector sizes on different OSTs is nearly inevitable over the next several years if new storage is added to an existing system, as 4kB sector drives arrive on the market and users may have little choice but to buy them. In summary, I'm not wholly against the idea of shipping the extra T10 GEN+REF data with the RPCs as you propose, so long as there is a single framework for rest of the checksum code for both the pages and the RPC is shared with existing checksum algorithms. Cheers, Andreas >> Since the client would always resend the whole RPC on checksum failure anyway, having a single checksum that is validated at both ends is enough for the Lustre part of the transfer. If it compares correctly, then the GRD+REF tags must match the originals, and can be used to submit to the underlying DIF hardware (if available) and/or recomputed to match the actual sector size if it is different between OSTs. >> >> The one issue is the APP tags. They are arbitrary 16-bit values, but it seems for the purpose of this design that the client and MDRAID layer will be generating the APP tags internally (versions), so there is currently no need (or ability) to store userspace APP tags all the way to disk? That would also avoid the need to pass them over the wire, since there is no chance of a torn/lost write for the RPC part of the data path. >> >> Some other comments on the design: >> 6.3: the mmap issue was fixed for Lustre 2.2 by Jinshan using the >> PG_writeback/page_mkwrite() support in newer kernels. It should >> no longer be possible for pages to change after calculating the >> checksum but before the RPC is sent. See LU-884 for details. > Excellent! > >> 8.5.2: why not compute the CRC in the correct endianness in the first >> place? The kernel already has e.g. crc32_le() and crc32_be() >> routines, so it may also have the needed routines for T10-CRC. >> It also isn't clear whether CRC-then-swab is the same as doing >> the CRC in the other endianness in the first place? Typically, >> RPCs are in "receiver makes right" format, which reduces overhead >> in the common case that both ends are the same. However, if >> the underlying DIF API requires the CRC to always be big endian, >> it makes sense to swab it on the client already. > Our conclusion as well - big endian everywhere. > >> >> Cheers, Andreas >> -- >> Andreas Dilger Whamcloud, Inc. >> Principal Lustre Engineer http://www.whamcloud.com/ >> >> >> >> Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From alexey_lyashkov at xyratex.com Tue Apr 17 17:39:47 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Tue, 17 Apr 2012 21:39:47 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> Message-ID: On Apr 17, 2012, at 11:06, Andreas Dilger wrote: > >>> One of the attractive features of the Merkle tree design is that it doesn't impose any extra overhead on the bulk RPC transmission, since the checksum sent with the RPC is distilled down to a single number, regardless of whether it is a single checksum on the data as is done today, or a hash of individual sector/page/chunk checksums as is proposed in both of our designs. Similarly, this avoids the need to store long lists of GRD+REF+APP tags on large pages (which is really the smallest unit of coherency on the client), as would be needed with the straight T10-DIX implementation (128 * 8 bytes for a 64kB PAGE_SIZE with 512-byte sectors). >> >> From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. > > That is potentially true, but if the checksums are not verified until you get down to the disk, if there ever is a real checksum error you won't have any idea where the error is being introduced. I may wrong, but I think, disk controller will verify checksum before put to the disk. so we will have a write error if checksum is wrong. -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com From Nathan_Rutman at xyratex.com Tue Apr 17 18:02:04 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 17 Apr 2012 11:02:04 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> On Apr 17, 2012, at 12:06 AM, Andreas Dilger wrote: On 2012-04-13, at 4:26 PM, Nathan Rutman wrote: On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. I view this as different solutions to the problem, but I see no reason why they can't coexist. The difficulty with having two independent, but closely related features that solve the same general problem is that this introduces additional code that needs to be maintained indefinitely for the future. I think that having separate checksum algorithms is fine, and possibly different transports for the checksums (if that is really needed, more below), but there should definitely only be a single framework for the checksum calculation and in-core storage that varies only by the selection of the algorithm. As far as Lustre is concerned, I view the T10 method as more of a out-of-band data transfer mechanism. When you say single framework, you imply that the checksum is performed at the same point in the code path, but I think that is fundamentally different between Merkle tree/ZFS (in core) and T10 (in hardware). From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. That is potentially true, but if the checksums are not verified until you get down to the disk, if there ever is a real checksum error you won't have any idea where the error is being introduced. That's a question about debugging really. Just as today many sites run without checksums at all due to server CPU loading, it makes sense to me to make intermediate verifications optional. I thought the same thing as John - that the appeal of standardized T10 DIF is that it can be verified all along the IO path to isolate the failing components. T10-DIX is not about fault isolation; it's about keeping protection information with the data for as long as possible. >From the http://snia.org/sites/default/files/Data_Integrity_Architectural_Model_v0.26.pdf Section 5: In the T10 Protection Information model both Data Integrity Metadata and actual data move through any communication path as a single unit. One can consider that two logical paths exist, one for data and the other for Data Integrity Metadata. The data path is called the Data Channel and the Data Integrity Metadata path is called the Protection Information Channel. 5.1 Movement of Protection Information An important purpose of this specification is to facilitate extending the Protection Envelope from the host-storage interface to the application endpoint. Without extending the protection envelope, the Protection Information Channel terminates at the host storage interface and only the Data Channel continues to the application. Consequently, a key element of this architecture describes how the Protection Information Channel may be extended to the application endpoint. From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. I don't see this argument. Having the OST verify the checksum from the network does not invalidate the additional checking down at the controller and/or disk layer (if there actually are disks that validate DIF, I'm not sure). There are. If the OST is regenerating the individual DIF sector checksums for verification by the hardware based on a per-RPC checksum (which has itself been verified to match the RPC checksum), then there is still end-to-end checksum coverage. It would be a strange software bug that generated new DIF checksums that were accepted by the hardware (for possibly corrupt data?) but also generated a valid hashed checksum to match the RPC. Since the RPC hash is a direct hash of the DIF checksums, the only way I can see this happening (for any reasonable RPC hash function, with a 32-bit or 256-bit RPC hash) is if the 16-bit DIF hash on the corrupt data matched the original value, in which case DIF would have missed the corruption as well... I can give you a trivial example: bad code in the Merkle tree implementation always generates a checksum of 0xDEADBEEF. Client computes it, server computes it, hey they match, and individual sector recomputed CRC's can still be correct, but the data do not match. I can buy the argument that doing the DIF data checksum on the OST consumes CPU cycles, but I had always imagined that it would do this anyway for data/checksum verification as soon as possible instead of waiting until the IO makes it to the controller. That would ensure that the data the RPC put into the OST cache is valid. Waiting for the controller to be the first to do the verification precludes ever having writeback caching on the OST. A writeback cached RPC would be completed before the data was verified by the controller, and it is not possible (today, at least) for the OST to force the client to resend just that RPC/data for replay. Any checksum error caused by network errors would require the OST to reset and start recovery for all RPCs in order to get the client to resend the failed RPC. I agree - if you're going to store the data in a writeback cache with the assumption that all other buffered copies are unavailable, it becomes a requirement to verify the data integrity when storing it in the cache. Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. This is a good point; I think a reasonable solution would require a common sector size within an OST. I hope that it would always to be the case that the disks in a single OST have the same sector size, but my concern was more about different sector sizes between OSTs. In that case, because the data checksum would be dependent on the sector size on each OST, the client per-page checksums would need to be computed based on the OST on which each page was to be written, which adds complexity into the IO path. In some sense this is no worse than having different checksum algorithms on different OSTs, but that would largely be an administrator choice that would otherwise default to using the same checksum algorithm for all OSTs. In contrast, different sector sizes on different OSTs is nearly inevitable over the next several years if new storage is added to an existing system, as 4kB sector drives arrive on the market and users may have little choice but to buy them. In summary, I'm not wholly against the idea of shipping the extra T10 GEN+REF data with the RPCs as you propose, so long as there is a single framework for rest of the checksum code for both the pages and the RPC is shared with existing checksum algorithms. Cheers, Andreas -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzzz.tomas at gmail.com Tue Apr 17 18:11:36 2012 From: bzzz.tomas at gmail.com (Alex Tomas) Date: Tue, 17 Apr 2012 22:11:36 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> Message-ID: On Tue, Apr 17, 2012 at 10:02 PM, Nathan Rutman wrote: > As far as Lustre is concerned, I view the T10 method as more of a > out-of-band data transfer mechanism.  When you say single framework, you > imply that the checksum is performed at the same point in the code path, but > I think that is fundamentally different between Merkle tree/ZFS (in core) > and T10 (in hardware). I'd hope core Lustre just doesn't care much passing checksums down to specific OSD, which then attach them to arc buffers (like ZFS) or to BIOs (like ldiskfs) ? thanks, Alex From Nathan_Rutman at xyratex.com Tue Apr 17 18:18:44 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 17 Apr 2012 11:18:44 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1097E@XYUS-EX22.xyus.xyratex.com> On Apr 17, 2012, at 11:11 AM, Alex Tomas wrote: > On Tue, Apr 17, 2012 at 10:02 PM, Nathan Rutman > wrote: >> As far as Lustre is concerned, I view the T10 method as more of a >> out-of-band data transfer mechanism. When you say single framework, you >> imply that the checksum is performed at the same point in the code path, but >> I think that is fundamentally different between Merkle tree/ZFS (in core) >> and T10 (in hardware). > > I'd hope core Lustre just doesn't care much passing checksums down to > specific OSD, > which then attach them to arc buffers (like ZFS) or to BIOs (like ldiskfs) ? That is my view as well; Lustre is just passing the protection information through. From Nathan_Rutman at xyratex.com Tue Apr 17 18:32:42 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 17 Apr 2012 11:32:42 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10982@XYUS-EX22.xyus.xyratex.com> On Apr 17, 2012, at 11:02 AM, Nathan Rutman wrote: >> >> In summary, I'm not wholly against the idea of shipping the extra T10 GEN+REF data with the RPCs as you propose, so long as there is a single framework for rest of the checksum code for both the pages and the RPC is shared with existing checksum algorithms. In case it wasn't clear, I'm in complete agreement that there should be a single integrated framework as much as possible; I just don't want to be pushed into the Merkle tree plan for the over-the-wire bit necessarily. I think a flexible description of the PI in the bulk descriptor should be able to cover all cases. From bzzz.tomas at gmail.com Tue Apr 17 18:35:33 2012 From: bzzz.tomas at gmail.com (Alex Tomas) Date: Tue, 17 Apr 2012 22:35:33 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1097E@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F1097E@XYUS-EX22.xyus.xyratex.com> Message-ID: On Tue, Apr 17, 2012 at 10:18 PM, Nathan Rutman wrote: > That is my view as well; Lustre is just passing the protection information through. then the question what does it pass specifically :) though I would agree with "early" detection above, say at OST. thanks, Alex From Nathan_Rutman at xyratex.com Wed Apr 18 19:52:08 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 18 Apr 2012 12:52:08 -0700 Subject: [Twg] Lustre 2.1-2.2 code delta breakdown Message-ID: <73AED5C780AE05478241DB067651A92102F10998@XYUS-EX22.xyus.xyratex.com> For general interest, here is our automated breakdown of the code areas modified between Lustre 2.1.0 and 2.2.0. This might tie back to the code coverage / quality discussions we were having earlier, in helping to point out areas of higher churn as potential candidates for heavier testing. These are web pages -- sorry for the zip format; apparently I'm not to be trusted with a website. If OpenSFS has got a place for these, please post them. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Lustre_git_stat_2.1_2.2.zip Type: application/zip Size: 117598 bytes Desc: Lustre_git_stat_2.1_2.2.zip URL: From adilger at whamcloud.com Wed Apr 18 20:37:28 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 18 Apr 2012 13:37:28 -0700 Subject: [Twg] Lustre 2.1-2.2 code delta breakdown In-Reply-To: <73AED5C780AE05478241DB067651A92102F10998@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10998@XYUS-EX22.xyus.xyratex.com> Message-ID: <95199997-11FC-4568-8D44-D72B84332A65@whamcloud.com> On 2012-04-18, at 12:52 PM, Nathan Rutman wrote: > For general interest, here is our automated breakdown of the code areas modified between Lustre 2.1.0 and 2.2.0. This might tie back to the code coverage / quality discussions we were having earlier, in helping to point out areas of higher churn as potential candidates for heavier testing. > > These are web pages -- sorry for the zip format; apparently I'm not to be trusted with a website. If OpenSFS has got a place for these, please post them. Very interesting. Presumably this was generated automatically, based on the LU ticket number and the Jira issue type? It would be interesting to see this over time, e.g. to see if the areas that were changed in 2.1 the most result in more bug fixes in later releases... Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Nathan_Rutman at xyratex.com Wed Apr 18 21:05:43 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 18 Apr 2012 14:05:43 -0700 Subject: [Twg] Lustre 2.1-2.2 code delta breakdown In-Reply-To: <95199997-11FC-4568-8D44-D72B84332A65@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10998@XYUS-EX22.xyus.xyratex.com> <95199997-11FC-4568-8D44-D72B84332A65@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1099B@XYUS-EX22.xyus.xyratex.com> On Apr 18, 2012, at 1:37 PM, Andreas Dilger wrote: > On 2012-04-18, at 12:52 PM, Nathan Rutman wrote: >> For general interest, here is our automated breakdown of the code areas modified between Lustre 2.1.0 and 2.2.0. This might tie back to the code coverage / quality discussions we were having earlier, in helping to point out areas of higher churn as potential candidates for heavier testing. >> >> These are web pages -- sorry for the zip format; apparently I'm not to be trusted with a website. If OpenSFS has got a place for these, please post them. > > Very interesting. Presumably this was generated automatically, based on the LU ticket number and the Jira issue type? Correct. Counts lines of change, looks up Jira ticket number to help classify. > It would be interesting to see this over time, e.g. to see if the areas that were changed in 2.1 the most result in more bug fixes in later releases... Exactly. Or areas that see constant noodling might be candidates for broader rework. We can generate similar graphs between arbitrary dates - we should come up with a web interface for that. = From adilger at whamcloud.com Wed Apr 18 21:55:27 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 18 Apr 2012 14:55:27 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> Message-ID: <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> On 2012-04-17, at 11:02 AM, Nathan Rutman wrote: > On Apr 17, 2012, at 12:06 AM, Andreas Dilger wrote: >> On 2012-04-13, at 4:26 PM, Nathan Rutman wrote: >>> On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: >>>> I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. >>> >>> I view this as different solutions to the problem, but I see no reason why they can't coexist. >> >> The difficulty with having two independent, but closely related features that solve the same general problem is that this introduces additional code that needs to be maintained indefinitely for the future. I think that having separate checksum algorithms is fine, and possibly different transports for the checksums (if that is really needed, more below), but there should definitely only be a single framework for the checksum calculation and in-core storage that varies only by the selection of the algorithm. > > As far as Lustre is concerned, I view the T10 method as more of a out-of-band data transfer mechanism. When you say single framework, you imply that the checksum is performed at the same point in the code path, but I think that is fundamentally different between Merkle tree/ZFS (in core) and T10 (in hardware). Sure, but the Merkle tree ends up being just a different checksum algorithm supported by the OST, that doesn't need any specific knowledge of the backing filesystem (blocksize, sector size, etc), and it doesn't impose any significant overhead on the protocol. The only potential change outside of adding the algorithm would be to send a full 256-bit checksum instead of only a 32-bit one today. Exposing a low-level detail like the disk sector size to the Lustre client (or application, should it get that far) to compute the GRD tag is a terrible layering violation, IMHO. This flies in the face of code abstraction and proper layering/isolation of different components in the storage stack, and is really contrary to the principles of Lustre that try to isolate the clients from internal details of the storage. It doesn't make sense that a Lustre client doesn't need to know the block size or even the type of filesystem on the OST, yet it needs to know the sector size of the disks living some layers below each OST? Perhaps if there was a simple transformation for merging eight 512-byte GRDs into a single 4kB GRD, then at least the Lustre client could always compute and send the GRD tags for the smallest sector size for any OST it is connected to (512 bytes, if there are mixed sector sizes, or 4kB if all OSTs have 4kB AF sector drives), and then the OST could efficiently merge them if it has a larger sector size locally. My math is too rusty to work this out, and I suspect it would be hard to do for CRC16, but I think it is possible to do with a matrix multiply for the IP16 checksum also used by the Linux DIF/DIX layer? >>> From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. >> >> That is potentially true, but if the checksums are not verified until you get down to the disk, if there ever is a real checksum error you won't have any idea where the error is being introduced. > > That's a question about debugging really. Just as today many sites run without checksums at all due to server CPU loading, it makes sense to me to make intermediate verifications optional. Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. See https://ext4.wiki.kernel.org/articles/e/x/t/Ext4_Metadata_Checksums_4d24.html for a useful recent comparison of checksum algorithm performance. It would be interesting to see if there was some way to improve the kernel crc16-t10 performance through loop unrolling or table lookups, given that significant performance gains were seen for crc32c-kern to the updated crc32c-by8 implementation. >> I thought the same thing as John - that the appeal of standardized T10 DIF is that it can be verified all along the IO path to isolate the failing components. > > T10-DIX is not about fault isolation; it's about keeping protection information with the data for as long as possible. > From the http://snia.org/sites/default/files/Data_Integrity_Architectural_Model_v0.26.pdf Section 5: > In the T10 Protection Information model both Data Integrity Metadata and actual data move through any communication path as a single unit. > One can consider that two logical paths exist, one for data and the other for Data Integrity Metadata. The data path is called the Data Channel and the Data Integrity Metadata path is called the Protection Information Channel. > > 5.1 Movement of Protection Information > An important purpose of this specification is to facilitate extending the Protection Envelope from the host-storage interface to the application endpoint. Without extending the protection envelope, the Protection Information Channel terminates at the host storage interface and only the Data Channel continues to the application. Consequently, a key element of this architecture describes how the Protection Information Channel may be extended to the application endpoint. > >> >>> From a philosophical point of view, it's between trusting the software to do the right thing in all cases, versus "my own checksums verified by disk hardware" (and trusting only the disk). I think the appeal of T10 to many people is exactly that - no matter what happens in the intervening layers, in the end I know my checksums have made it to disk. >> >> I don't see this argument. Having the OST verify the checksum from the network does not invalidate the additional checking down at the controller and/or disk layer (if there actually are disks that validate DIF, I'm not sure). > > There are. > >> If the OST is regenerating the individual DIF sector checksums for verification by the hardware based on a per-RPC checksum (which has itself been verified to match the RPC checksum), then there is still end-to-end checksum coverage. >> >> It would be a strange software bug that generated new DIF checksums that were accepted by the hardware (for possibly corrupt data?) but also generated a valid hashed checksum to match the RPC. Since the RPC hash is a direct hash of the DIF checksums, the only way I can see this happening (for any reasonable RPC hash function, with a 32-bit or 256-bit RPC hash) is if the 16-bit DIF hash on the corrupt data matched the original value, in which case DIF would have missed the corruption as well... > > I can give you a trivial example: bad code in the Merkle tree implementation always generates a checksum of 0xDEADBEEF. Client computes it, server computes it, hey they match, and individual sector recomputed CRC's can still be correct, but the data do not match. Can you give me a non-trivial example? :-) This same argument (though admittedly a strange bug :-) could be used for the non-Merkle checksums, and we are no further ahead. Some might also argue that at petascale or exascale data sizes/rates, a 16-bit checksum is going to let so many data corruptions go undetected (1 in 65k random corruptions) that it doesn't really solve the problem either. To clarify, I'm not dead set on using Merkle tree in place of large RPC requests needed to send the T10 GRD and REF tags for every sector. I would be OK to land such a capability as long as it would only be a protocol overhead for systems using T10-DIF, and would not exist for anyone using existing simple checksums or a Merkle checksum. My suggestion was an attempt to potentially avoid adding 4kB or 16kB overhead to every bulk RPC sent to T10-enabled servers, or alternately sending inefficient 1020kB or 1008kB writes to a 1024kB RAID stripe. I would guess that increasing the RPC request size or sending odd-sized writes has the potential to introduce far more performance overhead than an extra checksum calculation, since it means read-modify-write cycles for the RAID layer, or sending two bulk RPCs for each 1MB of data, but only time will tell. >> I can buy the argument that doing the DIF data checksum on the OST consumes CPU cycles, but I had always imagined that it would do this anyway for data/checksum verification as soon as possible instead of waiting until the IO makes it to the controller. That would ensure that the data the RPC put into the OST cache is valid. >> >> Waiting for the controller to be the first to do the verification precludes ever having writeback caching on the OST. A writeback cached RPC would be completed before the data was verified by the controller, and it is not possible (today, at least) for the OST to force the client to resend just that RPC/data for replay. Any checksum error caused by network errors would require the OST to reset and start recovery for all RPCs in order to get the client to resend the failed RPC. > > I agree - if you're going to store the data in a writeback cache with the assumption that all other buffered copies are unavailable, it becomes a requirement to verify the data integrity when storing it in the cache. As an academic question/comment, since I doubt it is possible to change the standards-entrenched DIF checksum algorithm, I don't think that Merkle trees have to be incompatible with T10-DIF. If there could be a hierarchical checksum algorithm in use for the GRD tag instead of straight CRC16, it would avoid real-world issues that arise from having to compute the CRC differently for 8x 512-byte sectors vs. a single 4kbyte sector. For example, if the GRD tag for a 4kB sector was the CRC16 of eight 512-byte CRC16s, then it would be trivial to compute and verify the GRD tag at an interface boundary that changed from 4kB/16kB/64kB sector_size or PAGE_SIZE to 512B sectors (or vice versa) without doubling the checksum overhead. Cheers, Andreas >>> Mainly for that latter reason we choose to send the CRCs instead of a function of them. But as I said, our goal is to introduce enough flexibility that multiple data integrity schemes can be used. >>> >>>> There is enough flexibility in the current Lustre RPC checksum wire format that both the T10 GRD+REF and the ZFS Fletcher4 checksum could be chosen as leaf checksums, and a tree hash can be used for both to get a single RPC hash, giving 2 or 3 new Lustre wire checksum formats. Since the server side needs to recompute the checksums anyway, it can regenerated GRD tag from the data, get the REF (offset) values from the RPC niobufs, and then rehash it to compare to the RPC checksum. >>>> >>>> Using a Merkle tree for the RPC checksum also avoids the potential problems as described in section 8.5, since the rather large amount of GRD+REF+APP tags do not actually need to be sent with the RPC, only the hash of them (currently a 32-bit value, but it could be expanded to be a 256-bit value if needed/desired). >>>> >>>> The Merkle tree also has the added benefit that the client does not need to know the sector sizes of the OSTs and handle them separately. Otherwise, there is either a restriction that the OSTs all have to have drives with the same sector sizes (to avoid the need for the client to compute GRD tags differently for each page in a striped file), or the llite layer will need to poke into the OSC for every page to figure out the sector size. >>> >>> This is a good point; I think a reasonable solution would require a common sector size within an OST. >> >> I hope that it would always to be the case that the disks in a single OST have the same sector size, but my concern was more about different sector sizes between OSTs. In that case, because the data checksum would be dependent on the sector size on each OST, the client per-page checksums would need to be computed based on the OST on which each page was to be written, which adds complexity into the IO path. >> >> In some sense this is no worse than having different checksum algorithms on different OSTs, but that would largely be an administrator choice that would otherwise default to using the same checksum algorithm for all OSTs. In contrast, different sector sizes on different OSTs is nearly inevitable over the next several years if new storage is added to an existing system, as 4kB sector drives arrive on the market and users may have little choice but to buy them. >> >> >> In summary, I'm not wholly against the idea of shipping the extra T10 GEN+REF data with the RPCs as you propose, so long as there is a single framework for rest of the checksum code for both the pages and the RPC is shared with existing checksum algorithms. >> >> Cheers, Andreas > Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Nathan_Rutman at xyratex.com Wed Apr 18 22:47:40 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 18 Apr 2012 15:47:40 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F1099F@XYUS-EX22.xyus.xyratex.com> Perhaps we can talk about this further at LUG, but I did want to answer a couple of things. On Apr 18, 2012, at 2:55 PM, Andreas Dilger wrote: On 2012-04-17, at 11:02 AM, Nathan Rutman wrote: On Apr 17, 2012, at 12:06 AM, Andreas Dilger wrote: On 2012-04-13, at 4:26 PM, Nathan Rutman wrote: On Apr 11, 2012, at 11:42 PM, Andreas Dilger wrote: I think it should be possible to come to a common solution between the T10 design you presented, and the Merkle tree design that was done for HPCS a few years ago. I view this as different solutions to the problem, but I see no reason why they can't coexist. The difficulty with having two independent, but closely related features that solve the same general problem is that this introduces additional code that needs to be maintained indefinitely for the future. I think that having separate checksum algorithms is fine, and possibly different transports for the checksums (if that is really needed, more below), but there should definitely only be a single framework for the checksum calculation and in-core storage that varies only by the selection of the algorithm. As far as Lustre is concerned, I view the T10 method as more of a out-of-band data transfer mechanism. When you say single framework, you imply that the checksum is performed at the same point in the code path, but I think that is fundamentally different between Merkle tree/ZFS (in core) and T10 (in hardware). Sure, but the Merkle tree ends up being just a different checksum algorithm supported by the OST, that doesn't need any specific knowledge of the backing filesystem (blocksize, sector size, etc), and it doesn't impose any significant overhead on the protocol. The only potential change outside of adding the algorithm would be to send a full 256-bit checksum instead of only a 32-bit one today. Exposing a low-level detail like the disk sector size to the Lustre client (or application, should it get that far) to compute the GRD tag is a terrible layering violation, IMHO. This flies in the face of code abstraction and proper layering/isolation of different components in the storage stack, and is really contrary to the principles of Lustre that try to isolate the clients from internal details of the storage. It doesn't make sense that a Lustre client doesn't need to know the block size or even the type of filesystem on the OST, yet it needs to know the sector size of the disks living some layers below each OST? Perhaps if there was a simple transformation for merging eight 512-byte GRDs into a single 4kB GRD, then at least the Lustre client could always compute and send the GRD tags for the smallest sector size for any OST it is connected to (512 bytes, if there are mixed sector sizes, or 4kB if all OSTs have 4kB AF sector drives), and then the OST could efficiently merge them if it has a larger sector size locally. You prompted me to go on a search here -- turns out the 4k sectors are bumped up to 4160 bytes for T10 in Marten Peterson's original Linux patches, so 8 bytes of PI per 512 bytes, fixed, no matter if either sector size is used in the backend. So complete layering horribleness is avoided, and only minor violation allowed to sneak in. But at least Lustre doesn't have to care about it. It would be a strange software bug that generated new DIF checksums that were accepted by the hardware (for possibly corrupt data?) but also generated a valid hashed checksum to match the RPC. Since the RPC hash is a direct hash of the DIF checksums, the only way I can see this happening (for any reasonable RPC hash function, with a 32-bit or 256-bit RPC hash) is if the 16-bit DIF hash on the corrupt data matched the original value, in which case DIF would have missed the corruption as well... I can give you a trivial example: bad code in the Merkle tree implementation always generates a checksum of 0xDEADBEEF. Client computes it, server computes it, hey they match, and individual sector recomputed CRC's can still be correct, but the data do not match. Can you give me a non-trivial example? :-) Sure: bad Merkle tree implementation provides poor coverage of the available hash space, leading to potential missed detection of lower-level T10 errors. This same argument (though admittedly a strange bug :-) could be used for the non-Merkle checksums, and we are no further ahead. To me, this is the point of transporting the unmolested checksum all the way to disk -- presumably the hardware calcs are simple and correct, and the user app is responsible for the other side, and nobody else's software has to be trusted at all. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Wed Apr 18 23:00:48 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 18 Apr 2012 16:00:48 -0700 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <73AED5C780AE05478241DB067651A92102F1099F@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1099F@XYUS-EX22.xyus.xyratex.com> Message-ID: On 2012-04-18, at 15:47, "Nathan Rutman" wrote: > Perhaps we can talk about this further at LUG, but I did want to answer a couple of things. > > On Apr 18, 2012, at 2:55 PM, Andreas Dilger wrote: >> Sure, but the Merkle tree ends up being just a different checksum algorithm supported by the OST, that doesn't need any specific knowledge of the backing filesystem (blocksize, sector size, etc), and it doesn't impose any significant overhead on the protocol. The only potential change outside of adding the algorithm would be to send a full 256-bit checksum instead of only a 32-bit one today. >> >> Exposing a low-level detail like the disk sector size to the Lustre client (or application, should it get that far) to compute the GRD tag is a terrible layering violation, IMHO. This flies in the face of code abstraction and proper layering/isolation of different components in the storage stack, and is really contrary to the principles of Lustre that try to isolate the clients from internal details of the storage. It doesn't make sense that a Lustre client doesn't need to know the block size or even the type of filesystem on the OST, yet it needs to know the sector size of the disks living some layers below each OST? >> >> Perhaps if there was a simple transformation for merging eight 512-byte GRDs into a single 4kB GRD, then at least the Lustre client could always compute and send the GRD tags for the smallest sector size for any OST it is connected to (512 bytes, if there are mixed sector sizes, or 4kB if all OSTs have 4kB AF sector drives), and then the OST could efficiently merge them if it has a larger sector size locally. > > You prompted me to go on a search here -- turns out the 4k sectors are bumped up to 4160 bytes for T10 in Marten Peterson's original Linux patches, so 8 bytes of PI per 512 bytes, fixed, no matter if either sector size is used in the backend. > > So complete layering horribleness is avoided, and only minor violation allowed to sneak in. But at least Lustre doesn't have to care about it. Excellent news. I have absolutely no problem with this, since a fixed 512-byte CRC size is just a parameter of the checksum algorithm. It has the less serious drawback that it is a very small unit of currency for Lustre, so it will add the maximum overhead for every RPC without giving a strong checksum. It has the benefit that T10 is a standard that users (and hardware) understand. If we are going to the length of sending the GRD and REF tags for every sector, we may as well go whole-hog and allow for the APP tags to be sent as well (or at least conditionally pass them and set a flag on the RPC if the RPC growth seems prohibitive) so that if userspace ever gets its act together we don't have to change the protocol again in the future. I don't recall anymore if this was covered in your original design. Cheers, Andreas From carrier at cray.com Thu Apr 19 17:05:45 2012 From: carrier at cray.com (John Carrier) Date: Thu, 19 Apr 2012 17:05:45 +0000 Subject: [Twg] link to requirements doc Message-ID: https://docs.google.com/document/d/1gXs0VPaCNEU3l9OuXYmBCyxKEm3dkIV60smCVAD5Un8/edit -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Apr 19 17:16:46 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 19 Apr 2012 11:16:46 -0600 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> Message-ID: On 2012-04-18, at 3:55 PM, Andreas Dilger wrote: > On 2012-04-17, at 11:02 AM, Nathan Rutman wrote: >> That's a question about debugging really. Just as today many sites run without checksums at all due to server CPU loading, it makes sense to me to make intermediate verifications optional. > > Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. > > See the table in https://ext4.wiki.kernel.org/articles/e/x/t/Ext4_Metadata_Checksums_4d24.html > > for a useful recent comparison of checksum algorithm performance. It would be interesting to see if there was some way to improve the kernel crc16-t10 performance through loop unrolling or table lookups, given that significant performance gains were seen for crc32c-kern to the updated crc32c-by8 implementation. Just reposting as requested by discussion on the call today. The summary is that crc32c is fast in both hardware (4.4GB/s) and software (1.6GB/s), while crc16-t10dif is very slow in software (300MB/s). There is a generic CRC instruction in newer Intel CPUs that might be usable to improve the crc16-t10dif performance. Nathan mentioned that someone at Xyratex can take a look at whether this could be implemented efficiently in hardware (via an asm wrapper in the short term, via cryptoapi in the long term). Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From alexey_lyashkov at xyratex.com Thu Apr 19 17:57:30 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Thu, 19 Apr 2012 21:57:30 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> Message-ID: <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: > > >>>> From a technical point of view, the tradeoff here is between transmitting a small amount of data (4k per 1MB) and recalculating that data on the servers. Depending on where/how often we re-verify the checksums, we could remove the recalculation from the server entirely. >>> >>> That is potentially true, but if the checksums are not verified until you get down to the disk, if there ever is a real checksum error you won't have any idea where the error is being introduced. >> >> That's a question about debugging really. Just as today many sites run without checksums at all due to server CPU loading, it makes sense to me to make intermediate verifications optional. > > Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. > Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. yes, main problem is OST side, not a client side checksumming. Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com From carrier at cray.com Thu Apr 19 19:34:03 2012 From: carrier at cray.com (John Carrier) Date: Thu, 19 Apr 2012 19:34:03 +0000 Subject: [Twg] new requirement Message-ID: Evan, You added the following comment to our google requirements doc: "It would be very useful for me as an admin to be able to easily localize who is using my IO. Even pointing to a specific set of nodes that are doing a lot of IO 'right now' would be very helpful." I think this is a new requirement for us. John Hammond's lltop (https://github.com/jhammond/lltop) provides a job-by-job accounting of file system load. Have you tried it? If it doesn't do what you want, could you help describe your requirement? Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From Evan.Felix at pnnl.gov Thu Apr 19 20:46:53 2012 From: Evan.Felix at pnnl.gov (Felix, Evan J) Date: Thu, 19 Apr 2012 13:46:53 -0700 Subject: [Twg] new requirement In-Reply-To: Message-ID: John, I just looked at his read me, it very well may do what I need, with some changes to get it to work with servers on separate networks… It would really be better if we had a way to query directly from the server for some of the statistics over lnet, since some of the larger configs with routers and such wont have direct ssh access. I will have to see if I can get it working. Evan From: John Carrier > To: "Felix, Evan J" > Cc: "twg at lists.opensfs.org" >, "'discuss at lists. org'" > Subject: new requirement Evan, You added the following comment to our google requirements doc: “It would be very useful for me as an admin to be able to easily localize who is using my IO. Even pointing to a specific set of nodes that are doing a lot of IO 'right now' would be very helpful.” I think this is a new requirement for us. John Hammond’s lltop (https://github.com/jhammond/lltop) provides a job-by-job accounting of file system load. Have you tried it? If it doesn’t do what you want, could you help describe your requirement? Thanks, --jc From spitzcor at cray.com Thu Apr 19 20:53:48 2012 From: spitzcor at cray.com (Cory Spitz) Date: Thu, 19 Apr 2012 15:53:48 -0500 Subject: [Twg] [Discuss] new requirement In-Reply-To: References: Message-ID: <4F907B5C.2000302@cray.com> If you just want the nodes (clients) and not the per-application data then the per-export server side stats can give you some info such as brw_stats. -Cory On 04/19/2012 03:46 PM, Felix, Evan J wrote: > John, I just looked at his read me, it very well may do what I need, with some changes to get it to work with servers on separate networks… It would really be better if we had a way to query directly from the server for some of the statistics over lnet, since some of the larger configs with routers and such wont have direct ssh access. I will have to see if I can get it working. > > Evan > > From: John Carrier > > To: "Felix, Evan J" > > Cc: "twg at lists.opensfs.org" >, "'discuss at lists. org'" > > Subject: new requirement > > Evan, > > You added the following comment to our google requirements doc: > > “It would be very useful for me as an admin to be able to easily localize who is using my IO. Even pointing to a specific set of nodes that are doing a lot of IO 'right now' would be very helpful.” > > I think this is a new requirement for us. John Hammond’s lltop (https://github.com/jhammond/lltop) provides a job-by-job accounting of file system load. Have you tried it? If it doesn’t do what you want, could you help describe your requirement? > Thanks, > > --jc > > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org From adilger at whamcloud.com Fri Apr 20 04:41:41 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 19 Apr 2012 22:41:41 -0600 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> Message-ID: <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: > On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >> >> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >> > Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. > yes, main problem is OST side, not a client side checksumming. > > Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. > Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. Cheers, Andreas From carrier at cray.com Fri Apr 20 07:05:18 2012 From: carrier at cray.com (John Carrier) Date: Fri, 20 Apr 2012 07:05:18 +0000 Subject: [Twg] requirements document update Message-ID: During the TWG concall this morning, we walked through the comments several reviewers had made to our 2012 requirements document. Several of us spent the rest of the day editing the text and resolving the remaining comments (thanks, Alex, Andreas, Cory, Josh and Nathan!) It's now your opportunity to review the document (http://goo.gl/63u9Q) to help us select the requirements we will recommend to the OpenSFS Board for development in 2013. As we did last year, we need to create a short list of 4-5 items we think are most important to the Lustre community. The OpenSFS Board must agree with our recommendation before we can pursue proposals to meet these requirements. Now that we have finished gathering requirements, the first step is for community members to send their top 10 requirements from the list below in priority order to discuss at lists.opensfs.org. We will discuss how to resolve our lists and make the recommendation to the board at the TWG face to face meeting (Weds 4/25 after LUG in Austin). Please send questions or comments to the discuss at lists.opensfs.org reflector. --jc The following table summarizes the 31 gathered requirements described in our document (http://goo.gl/63u9Q). Category definitions are only meant to help organize requirements for discussion. There is no significance to the order in which the requirements are listed. Category Requirement Performance Single client performance Single shared file performance Quality of service Locality and scalability Availability Scalable fault management Avoid RPC timeouts Manageability and Administrative Better support for newer kernels Improved configuration robustness Administrative shutdown Better userspace tools Snapshots Arbitrary OST assignment Storage Management HSM and storage management infrastructure OST migration/rebalancing Asynchronous file replication (mirroring) Complex file layouts Dynamic layout for subset of a file Storage pool quotas Unified storage target LNET IPv6 Dynamic LNET configuration Improved LNET robustness LNET channel bonding Application Interface Improved storage semantics/interfaces Better user tool API POSIX extensions for file sets POSIX extensions for scalable opens Other Varying page-sizes Mixed endian support Improved security infrastructure Improved Lustre Tests -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Sat Apr 21 05:35:37 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 09:35:37 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: <6E323B3D-9F08-465C-93B8-5BD14553EED1@xyratex.com> On Apr 20, 2012, at 08:41, Andreas Dilger wrote: > On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>> >>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>> >> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >> yes, main problem is OST side, not a client side checksumming. >> >> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. > > I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. > per Xyratex internal discussion, we can use same way as crc32 improvement to provide crc16 speed up, so it`s should be have speed near of 2Gb/s. From alexey_lyashkov at xyratex.com Sat Apr 21 05:39:03 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 09:39:03 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: On Apr 20, 2012, at 08:41, Andreas Dilger wrote: > On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>> >>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>> >> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >> yes, main problem is OST side, not a client side checksumming. >> >> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. > > I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. > i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. so i think we don't need any separation for both these cases. -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com From carrier at cray.com Sat Apr 21 05:50:37 2012 From: carrier at cray.com (John Carrier) Date: Sat, 21 Apr 2012 05:50:37 +0000 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: Alexey, If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov Sent: Friday, April 20, 2012 10:39 PM To: Andreas Dilger Cc: AndrewPerepechko; twg at lists.opensfs.org Subject: Re: [Twg] T10 End-to-End Data Integrity HLD On Apr 20, 2012, at 08:41, Andreas Dilger wrote: > On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>> >>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>> >> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >> yes, main problem is OST side, not a client side checksumming. >> >> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. > > I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. > i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. so i think we don't need any separation for both these cases. -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From alexey_lyashkov at xyratex.com Sat Apr 21 05:53:55 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 09:53:55 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: John, don't understand question. Per Andrew HLD - client generate a CRC and attach information to the data transfer (bulk transfer) - CRC verified via HCA on client side. so i think it's same as you suggest ? On Apr 21, 2012, at 09:50, John Carrier wrote: > Alexey, > > If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. > > --jc > > -----Original Message----- > From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov > Sent: Friday, April 20, 2012 10:39 PM > To: Andreas Dilger > Cc: AndrewPerepechko; twg at lists.opensfs.org > Subject: Re: [Twg] T10 End-to-End Data Integrity HLD > > > On Apr 20, 2012, at 08:41, Andreas Dilger wrote: > >> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >>> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>>> >>>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>>> >>> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >>> yes, main problem is OST side, not a client side checksumming. >>> >>> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >>> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. >> >> I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. >> > i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. > from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. > so i think we don't need any separation for both these cases. > > > -------------------------------------------- > Alexey Lyashkov > alexey_lyashkov at xyratex.com > > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From alexey_lyashkov at xyratex.com Sat Apr 21 05:56:08 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 09:56:08 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: <987886E5-0487-4915-B55C-978596668157@xyratex.com> grrr. "HCA on server side" On Apr 21, 2012, at 09:53, Alexey Lyashkov wrote: > John, > > don't understand question. Per Andrew HLD - client generate a CRC and attach information to the data transfer (bulk transfer) - CRC verified via HCA on client side. > so i think it's same as you suggest ? > > On Apr 21, 2012, at 09:50, John Carrier wrote: > >> Alexey, >> >> If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. >> >> --jc >> >> -----Original Message----- >> From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov >> Sent: Friday, April 20, 2012 10:39 PM >> To: Andreas Dilger >> Cc: AndrewPerepechko; twg at lists.opensfs.org >> Subject: Re: [Twg] T10 End-to-End Data Integrity HLD >> >> >> On Apr 20, 2012, at 08:41, Andreas Dilger wrote: >> >>> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >>>> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>>>> >>>>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>>>> >>>> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >>>> yes, main problem is OST side, not a client side checksumming. >>>> >>>> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >>>> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. >>> >>> I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. >>> >> i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. >> from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. >> so i think we don't need any separation for both these cases. >> >> >> -------------------------------------------- >> Alexey Lyashkov >> alexey_lyashkov at xyratex.com >> >> >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From carrier at cray.com Sat Apr 21 06:06:28 2012 From: carrier at cray.com (John Carrier) Date: Sat, 21 Apr 2012 06:06:28 +0000 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: No, I'm saying that if the PI generation in SW is too expensive, then we can use a different checksum between the client and server. I recognize that this is not compatible with Nathan's HLD and that it will mean the server has to validate the checksum before the HCA touches the data. The point is that there is no requirement for the client to do the PI validation. I agree that doing the PI validation/generation on the client is a clean end-to-end solution. But if the PI calculation so significantly reduces throughput, then there is no point using it. -----Original Message----- From: Alexey Lyashkov [mailto:alexey_lyashkov at xyratex.com] Sent: Friday, April 20, 2012 10:54 PM To: John Carrier Cc: Andreas Dilger; AndrewPerepechko; twg at lists.opensfs.org; discuss at lists.opensfs.org Subject: Re: [Twg] T10 End-to-End Data Integrity HLD John, don't understand question. Per Andrew HLD - client generate a CRC and attach information to the data transfer (bulk transfer) - CRC verified via HCA on client side. so i think it's same as you suggest ? On Apr 21, 2012, at 09:50, John Carrier wrote: > Alexey, > > If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. > > --jc > > -----Original Message----- > From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov > Sent: Friday, April 20, 2012 10:39 PM > To: Andreas Dilger > Cc: AndrewPerepechko; twg at lists.opensfs.org > Subject: Re: [Twg] T10 End-to-End Data Integrity HLD > > > On Apr 20, 2012, at 08:41, Andreas Dilger wrote: > >> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >>> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>>> >>>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>>> >>> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >>> yes, main problem is OST side, not a client side checksumming. >>> >>> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >>> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. >> >> I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. >> > i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. > from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. > so i think we don't need any separation for both these cases. > > > -------------------------------------------- > Alexey Lyashkov > alexey_lyashkov at xyratex.com > > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ From alexey_lyashkov at xyratex.com Sat Apr 21 06:12:44 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 10:12:44 +0400 Subject: [Twg] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> John, you are wrong. It's Andrew HLD. possible it isn't clear - but per internal discussion, server isn't validate checksum before HCA. Using a HCA to checksum validation and provide end to end integrity checks - main idea of using T10 in lustre. per my previous comment server validation isn't need - because client will resend IO request in both cases - checksum error or real IO error. On Apr 21, 2012, at 10:06, John Carrier wrote: > No, I'm saying that if the PI generation in SW is too expensive, then we can use a different checksum between the client and server. I recognize that this is not compatible with Nathan's HLD and that it will mean the server has to validate the checksum before the HCA touches the data. The point is that there is no requirement for the client to do the PI validation. I agree that doing the PI validation/generation on the client is a clean end-to-end solution. But if the PI calculation so significantly reduces throughput, then there is no point using it. > > > > -----Original Message----- > From: Alexey Lyashkov [mailto:alexey_lyashkov at xyratex.com] > Sent: Friday, April 20, 2012 10:54 PM > To: John Carrier > Cc: Andreas Dilger; AndrewPerepechko; twg at lists.opensfs.org; discuss at lists.opensfs.org > Subject: Re: [Twg] T10 End-to-End Data Integrity HLD > > John, > > don't understand question. Per Andrew HLD - client generate a CRC and attach information to the data transfer (bulk transfer) - CRC verified via HCA on client side. > so i think it's same as you suggest ? > > On Apr 21, 2012, at 09:50, John Carrier wrote: > >> Alexey, >> >> If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. >> >> --jc >> >> -----Original Message----- >> From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov >> Sent: Friday, April 20, 2012 10:39 PM >> To: Andreas Dilger >> Cc: AndrewPerepechko; twg at lists.opensfs.org >> Subject: Re: [Twg] T10 End-to-End Data Integrity HLD >> >> >> On Apr 20, 2012, at 08:41, Andreas Dilger wrote: >> >>> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >>>> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>>>> >>>>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>>>> >>>> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >>>> yes, main problem is OST side, not a client side checksumming. >>>> >>>> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >>>> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. >>> >>> I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. >>> >> i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. >> from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. >> so i think we don't need any separation for both these cases. >> >> >> -------------------------------------------- >> Alexey Lyashkov >> alexey_lyashkov at xyratex.com >> >> >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > > From johann at whamcloud.com Sat Apr 21 08:22:03 2012 From: johann at whamcloud.com (Johann Lombardi) Date: Sat, 21 Apr 2012 10:22:03 +0200 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> Message-ID: Hi Alexey, On Saturday, April 21, 2012, Alexey Lyashkov wrote: > > you are wrong. It's Andrew HLD. possible it isn't clear - but per internal > discussion, server isn't validate checksum before HCA. Using a HCA to > checksum validation and provide end to end integrity checks - main idea of > using T10 in lustre. > per my previous comment server validation isn't need - because client will > resend IO request in both cases - checksum error or real IO error. This means that you can't use asynchronous journal commit in conjunction with T10, right? Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Sun Apr 22 03:36:27 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 21 Apr 2012 22:36:27 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> Message-ID: <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> John, Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. so we may don't want a commit as HCA verify a checksum and it's correct. PS. Andrew - please add that clarification to the document. On Apr 21, 2012, at 03:22, Johann Lombardi wrote: > Hi Alexey, > > On Saturday, April 21, 2012, Alexey Lyashkov wrote: > you are wrong. It's Andrew HLD. possible it isn't clear - but per internal discussion, server isn't validate checksum before HCA. Using a HCA to checksum validation and provide end to end integrity checks - main idea of using T10 in lustre. > per my previous comment server validation isn't need - because client will resend IO request in both cases - checksum error or real IO error. > > This means that you can't use asynchronous journal commit in conjunction with T10, right? > > Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Sun Apr 22 19:41:02 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Sun, 22 Apr 2012 14:41:02 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> Message-ID: On Apr 21, 2012, at 1:06 AM, "John Carrier" wrote: > No, I'm saying that if the PI generation in SW is too expensive, then we can use a different checksum between the client and server. A few points here: A. Intel's pclmulqdq instruction can be used to speed crc16-t10; Intel has sample code on their website. This can speed t10 calc on clients. B. Part of the benefit of T10 is that the server need not compute anything - it's done in hardware in the HBA and drives. C. Although in single-client apps the client side crc speed matters, for i/o bound applications it does not - the clients in toto can always keep the OSS pipes full, and since we're removing the OSS crc bottleneck the total system throughput will be improved. > I recognize that this is not compatible with Nathan's HLD and that it will mean the server has to validate the checksum before the HCA touches the data. The point is that there is no requirement for the client to do the PI validation. I agree that doing the PI validation/generation on the client is a clean end-to-end solution. But if the PI calculation so significantly reduces throughput, then there is no point using it. > > > > -----Original Message----- > From: Alexey Lyashkov [mailto:alexey_lyashkov at xyratex.com] > Sent: Friday, April 20, 2012 10:54 PM > To: John Carrier > Cc: Andreas Dilger; AndrewPerepechko; twg at lists.opensfs.org; discuss at lists.opensfs.org > Subject: Re: [Twg] T10 End-to-End Data Integrity HLD > > John, > > don't understand question. Per Andrew HLD - client generate a CRC and attach information to the data transfer (bulk transfer) - CRC verified via HCA on client side. > so i think it's same as you suggest ? > > On Apr 21, 2012, at 09:50, John Carrier wrote: > >> Alexey, >> >> If you are using the HCA to offload PI calculation on the server, why not use a CRC between the client and server that is optimal for software calculations. The DIX architecture from SNIA does not require the PI calculation to extend to the app layer and describes translation between segments of the path. >> >> --jc >> >> -----Original Message----- >> From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Alexey Lyashkov >> Sent: Friday, April 20, 2012 10:39 PM >> To: Andreas Dilger >> Cc: AndrewPerepechko; twg at lists.opensfs.org >> Subject: Re: [Twg] T10 End-to-End Data Integrity HLD >> >> >> On Apr 20, 2012, at 08:41, Andreas Dilger wrote: >> >>> On 2012-04-19, at 11:57, Alexey Lyashkov wrote: >>>> On Apr 19, 2012, at 01:55, Andreas Dilger wrote: >>>>> >>>>> Based on the data in LU-1201, I think with Nehalem CPUs and multi-threaded ptlrpcd the checksum overhead will become a non-issue, and people will happily leave it enabled all the time. The unfortunate thing is that the CRC16 used by T10-DIF has such poor software performance that I agree there is no option but to leave the verification to the hardware. >>>>> >>>> Based on Xyratex test results - crc32c HW implementation isn't best choose, and multi thread ptlrpcd isn't help to improve a OST side loading. >>>> yes, main problem is OST side, not a client side checksumming. >>>> >>>> Typical speed is less 1Gb/s for crc32c hw, and it's less speed then adler may provide. >>>> Current top speed list - adler, crc32 hw based on (not crc32c), crc32c hw, both crc32c table implementation. >>> >>> I think we both agree in any case that the current crc16-t10dif is not nearly as fast as any of these 32-bit checksums. This may prove to be a major discouragement to using T10-PI even if it is available. >>> >> i forget one note, in case T10 DIF OSS CPU isn't included into CRC verification - it's done by raid/sas adapter transparency so we will dramatically reduce load on OSS node with that type checksumming enabled. >> from OSC view - not have differences in handling EIO error - if it's real disk IO error or it's error during network transmitting, in both cases we need resend a data to OSS from a client. >> so i think we don't need any separation for both these cases. >> >> >> -------------------------------------------- >> Alexey Lyashkov >> alexey_lyashkov at xyratex.com >> >> >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > > > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org From andrew_perepechko at xyratex.com Tue Apr 24 10:08:51 2012 From: andrew_perepechko at xyratex.com (Andrew Perepechko) Date: Tue, 24 Apr 2012 13:08:51 +0300 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: Hello! Alex, the HLD already expects all data writes to be synchronous on the OSS (filter_do_bio() does wait in cfs_wait_event() for I/O completion on all buffers so that we can get errors synchronously; even if a drive signals I/O completion prior to physical writing data, I think, it makes the whole of verification in advance). However, since there is a common misconception that async journal commit has anything to do with async data commit, I'll try to stress it even more. Nathan, I was thinking that 4096-byte sectors are really 4104-bytes sectors with T10-capable disks (as in http://www.mjmwired.net/kernel/Documentation/block/data-integrity.txt). Are there really any 4160-byte sector disks? Andreas, I've got the impression that the mkwrite() patch can only help with mmap races if the client cache protection is disabled. The idea of the patch seems to make mmaped pages write-protected while a write request (including checksum calculation) is serviced, other threads trying to write to the same page will cause a page fault and stick in Lustre waiting for I/O completion. However, if any protection information is attached to a page, we need to update it with every page write access. This is possible through the common VFS write path, but hardly through a PROT_WRITE mapping. P.S. Sorry for merging multiple replies in a single email, this is only to prevent spamming. Thank you, Andrew ----- Original Message ----- From: Alexey Lyashkov To: Johann Lombardi Cc: John Carrier ; twg at lists.opensfs.org ; discuss at lists.opensfs.org ; AndrewPerepechko Sent: Sunday, April 22, 2012 6:36 AM Subject: Re: [Discuss] [Twg] T10 End-to-End Data Integrity HLD John, Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. so we may don't want a commit as HCA verify a checksum and it's correct. PS. Andrew - please add that clarification to the document. On Apr 21, 2012, at 03:22, Johann Lombardi wrote: Hi Alexey, On Saturday, April 21, 2012, Alexey Lyashkov wrote: you are wrong. It's Andrew HLD. possible it isn't clear - but per internal discussion, server isn't validate checksum before HCA. Using a HCA to checksum validation and provide end to end integrity checks - main idea of using T10 in lustre. per my previous comment server validation isn't need - because client will resend IO request in both cases - checksum error or real IO error. This means that you can't use asynchronous journal commit in conjunction with T10, right? Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From johann at whamcloud.com Tue Apr 24 19:03:00 2012 From: johann at whamcloud.com (Johann Lombardi) Date: Tue, 24 Apr 2012 21:03:00 +0200 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov < alexey_lyashkov at xyratex.com> wrote: > Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO > request - so it's before a call bio_done() function and before send a reply > to client. > > so we may don't want a commit as HCA verify a checksum and it's correct. > I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? Cheers, Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Tue Apr 24 19:44:40 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Tue, 24 Apr 2012 14:44:40 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: <516906EF-1E9A-4C53-889D-834CA2747EE7@xyratex.com> Johann, what you mean as metadata blocks? extent's on OST or MDT disk io traffic ? On Apr 24, 2012, at 14:03, Johann Lombardi wrote: > On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov wrote: > Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. > > so we may don't want a commit as HCA verify a checksum and it's correct. > > I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? > > Cheers, > Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From johann at whamcloud.com Tue Apr 24 19:53:13 2012 From: johann at whamcloud.com (Johann Lombardi) Date: Tue, 24 Apr 2012 21:53:13 +0200 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: <516906EF-1E9A-4C53-889D-834CA2747EE7@xyratex.com> References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> <516906EF-1E9A-4C53-889D-834CA2747EE7@xyratex.com> Message-ID: On Tue, Apr 24, 2012 at 9:44 PM, Alexey Lyashkov < alexey_lyashkov at xyratex.com> wrote: > what you mean as metadata blocks? extent's on OST or MDT disk io traffic ? > Yes, anything that goes to the journal (inode, block bitmap, ...) which isn't user data. Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Tue Apr 24 19:58:26 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Tue, 24 Apr 2012 14:58:26 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: Message-ID: <24AFE0E1-0DEF-4F25-9A79-C5B0C4DF7CE9@xyratex.com> T10 generally isn't about metadata changes, but we may generate checksums for that IO on service nodes locally. On Apr 24, 2012, at 14:53, Johann Lombardi wrote: > On Tue, Apr 24, 2012 at 9:44 PM, Alexey Lyashkov wrote: > what you mean as metadata blocks? extent's on OST or MDT disk io traffic ? > > Yes, anything that goes to the journal (inode, block bitmap, ...) which isn't user data. > > Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew_perepechko at xyratex.com Tue Apr 24 20:50:11 2012 From: andrew_perepechko at xyratex.com (Andrew Perepechko) Date: Tue, 24 Apr 2012 23:50:11 +0300 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com><97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com><73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com><02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com><73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com><61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com><31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com><8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com><8EA735BD-A346-4580-818A-336838F268AD@xyratex.com><32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: <79DE9986234349C2984692B6A3C78275@homee5f91f0004> Hi Johann! The original design does not cover metadata integrity. Guard tag value 0xffff can be used to avoid guard tag integrity checks on specific (metadata) sectors. If metadata is to be included in integrity checks, then I/O replay should be implemented for recoverable errors. Since T10 tag mismatches don't seem to be the only possible source of recoverable errors, this looks like an existing defect in current BRW RPC handling/async_journal implementation (?). Andrew ----- Original Message ----- From: Johann Lombardi To: Alexey Lyashkov Cc: John Carrier ; twg at lists.opensfs.org ; discuss at lists.opensfs.org ; AndrewPerepechko Sent: Tuesday, April 24, 2012 10:03 PM Subject: Re: [Discuss] [Twg] T10 End-to-End Data Integrity HLD On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov wrote: Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. so we may don't want a commit as HCA verify a checksum and it's correct. I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? Cheers, Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew_perepechko at xyratex.com Tue Apr 24 20:50:20 2012 From: andrew_perepechko at xyratex.com (Andrew Perepechko) Date: Tue, 24 Apr 2012 23:50:20 +0300 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com><97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com><73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com><02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com><73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com><61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com><31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com><8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com><8EA735BD-A346-4580-818A-336838F268AD@xyratex.com><32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: <2D170ACB95104F4A9B2D9C599907CF10@homee5f91f0004> Hi Johann! The original design does not cover metadata integrity. Guard tag value 0xffff can be used to avoid guard tag integrity checks on specific (metadata) sectors. If metadata is to be included in integrity checks, then I/O replay should be implemented for recoverable errors. Since T10 tag mismatches don't seem to be the only possible source of recoverable errors, this looks like an existing defect in current BRW RPC handling/async_journal implementation (?). Andrew ----- Original Message ----- From: Johann Lombardi To: Alexey Lyashkov Cc: John Carrier ; twg at lists.opensfs.org ; discuss at lists.opensfs.org ; AndrewPerepechko Sent: Tuesday, April 24, 2012 10:03 PM Subject: Re: [Discuss] [Twg] T10 End-to-End Data Integrity HLD On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov wrote: Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. so we may don't want a commit as HCA verify a checksum and it's correct. I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? Cheers, Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From andrew_perepechko at xyratex.com Tue Apr 24 20:50:32 2012 From: andrew_perepechko at xyratex.com (Andrew Perepechko) Date: Tue, 24 Apr 2012 23:50:32 +0300 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com><97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com><73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com><02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com><73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com><61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com><31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com><8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com><8EA735BD-A346-4580-818A-336838F268AD@xyratex.com><32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: <212F76EC4C32460F99DD170F46223510@homee5f91f0004> Hi Johann! The original design does not cover metadata integrity. Guard tag value 0xffff can be used to avoid guard tag integrity checks on specific (metadata) sectors. If metadata is to be included in integrity checks, then I/O replay should be implemented for recoverable errors. Since T10 tag mismatches don't seem to be the only possible source of recoverable errors, this looks like an existing defect in current BRW RPC handling/async_journal implementation (?). Andrew ----- Original Message ----- From: Johann Lombardi To: Alexey Lyashkov Cc: John Carrier ; twg at lists.opensfs.org ; discuss at lists.opensfs.org ; AndrewPerepechko Sent: Tuesday, April 24, 2012 10:03 PM Subject: Re: [Discuss] [Twg] T10 End-to-End Data Integrity HLD On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov wrote: Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. so we may don't want a commit as HCA verify a checksum and it's correct. I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? Cheers, Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Tue Apr 24 21:19:31 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Tue, 24 Apr 2012 16:19:31 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: Message-ID: <347CA580-C12E-4A1F-9FFD-AAD94E66F2DC@xyratex.com> some results of online discussion. Johann worry about journal checksumming (to avoid overhead in that area) or other metadata checks on client store uncommitted data in own queue for a replay (if need), but in case journal checksum errors on write client will able to resend failed request via special protocol extension (don't exist now). same situation if we need a T10 support for these data we will be need trigger a failover or change a wire protocol to resend a failed request. failover is easily. On Apr 24, 2012, at 14:53, Johann Lombardi wrote: > On Tue, Apr 24, 2012 at 9:44 PM, Alexey Lyashkov wrote: > what you mean as metadata blocks? extent's on OST or MDT disk io traffic ? > > Yes, anything that goes to the journal (inode, block bitmap, ...) which isn't user data. > > Johann -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Wed Apr 25 08:07:33 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 25 Apr 2012 03:07:33 -0500 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F10941@XYUS-EX22.xyus.xyratex.com> <97E3079B-B2D3-46E3-8955-B1FCCC1DB49C@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1096D@XYUS-EX22.xyus.xyratex.com> <02965A12-03E7-4B25-B91B-7A0377CB4717@whamcloud.com> <73AED5C780AE05478241DB067651A92102F1097C@XYUS-EX22.xyus.xyratex.com> <61E09DC3-805F-4143-B841-0EDF33178F19@whamcloud.com> <31F047F0-66F0-4348-9C6A-7B7A0C6D3587@xyratex.com> <8856852A-6102-4CFF-B481-22BB67C560C4@whamcloud.com> <8EA735BD-A346-4580-818A-336838F268AD@xyratex.com> <32586EEA-55ED-42B9-98F4-1768EE3D4651@xyratex.com> Message-ID: <95318858-DD53-4CAA-B040-F40CC0DE8F51@whamcloud.com> There is actually work well underway to add metadata checksums for ext4 into the upstream kernel. This is using both inline and out-of-band locations to store the block checksum for the different metadata block types. The checksums on disk are crc32c. Cheers, Andreas On 2012-04-24, at 14:03, Johann Lombardi wrote: > On Sun, Apr 22, 2012 at 5:36 AM, Alexey Lyashkov wrote: > Why not? HCA provide a T10 checksumming when obdfilter (osd) submit a IO request - so it's before a call bio_done() function and before send a reply to client. > > so we may don't want a commit as HCA verify a checksum and it's correct. > > I was actually concerned by checksum errors when committing metadata blocks to disk. That said, i assume that the low level driver is supposed to resubmit the I/O several times (ideally via multipath) on checksum errors and ultimately triggers a failover (to replay the I/O), right? > > Cheers, > Johann > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Wed Apr 25 16:54:49 2012 From: carrier at cray.com (John Carrier) Date: Wed, 25 Apr 2012 16:54:49 +0000 Subject: [Twg] TWG F2F meeting today Message-ID: OpenSFS will have a F2F meeting today after LUG. We will be in room Texas 4 at the Austin Hyatt. The schedule is 1pm-2pm General Meeting 2pm-3pm Technical Working Group Meeting 3pm-4pm Community Development Working Group Meeting Our proposed agenda is * Review the Requirements document (http://goo.gl/63u9Q) * Moving the client upstream * Long-term use of the test cluster There will be no TWG call tomorrow (Thurs 4/26) --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Wed Apr 25 17:06:41 2012 From: carrier at cray.com (John Carrier) Date: Wed, 25 Apr 2012 17:06:41 +0000 Subject: [Twg] FW: Fermi requirements list Re: requirements document update Message-ID: Below please find Fermi's top 10 requirements for 2012. If your organization wants to have a say in the development roadmap OpenSFS will pursue for next year, then this is a reminder that you need to send your list to us too. -thanks, --jc From: Alex Kulyavtsev [mailto:aik at fnal.gov] Sent: Tuesday, April 24, 2012 2:13 PM To: John Carrier Cc: Alex Kulyavtsev; Gene Oleynik Subject: Fermi requirements list Re: [Twg] requirements document update On Apr 20, 2012, at 2:05 AM, John Carrier wrote: the first step is for community members to send their top 10 requirements from the list below in priority order Hi John, below please find ordered list of requirements from Fermilab. Alex. 1 HSM and storage management infrastructure 2 OST migration/rebalancing 3 Quality of service 4 Scalable fault management 5 Avoid RPC timeouts 6 Better support for newer kernels 7 Improved configuration robustness 8 Administrative shutdown 9 Improved security infrastructure 10 Locality and scalability -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexey_lyashkov at xyratex.com Wed Apr 25 19:40:12 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Wed, 25 Apr 2012 14:40:12 -0500 Subject: [Twg] LNet channel bounding Message-ID: Hi All, i write a simple document about creating a channel bounding in lnet some time ago and want to discus that is before we will start and make design and code for that project. LNet bounding conceptual design LNet is network module designed to exchange messages between lustre nodes. Channel bounding is need to extend network bandwidth between nodes, it’s allow to send many messages parallel, each in own network channel. Exist some channel bounding modes · System bounding – some networks channels merged in single logical channel on system side and lustre see that as one channel with single address space. · LND based bounding – currently supported on socklnd (tcp network module). In that case nodes use extended HELO protocol to send all addresses from node to remote peer and peer a look to these addresses to send request in round robin order. Both these modes don’t support a channels bounding between different network types such as IB and Ethernet networks. To avoid that problem we need an implementing bounding inside LNet module. Exist some possibilities to implementing it. 1. Reuse LNet routing code. In that case each destination nid mapped some gateway via list of routing rules. Problems in that area · We need an implementing destination address mapping inside LNet code. · We need change lnet_ni_t to hide interfaces joined in bounding group. · We need a change half of code in routing send codepath to kill packet attributes change. · Changes in incoming codepath – need extract bounding messages from routed. · Troubles with peer health detection (it need to be know – peer is router or not). · Don’t possible to resend message in case interface makes down in message transmit time. · I think it task will be need ~1000-1500 LOC of changes and need hard testing on both modes (node and router). 2. Implementing “Bounding” LND It’s way much clear, because needs a all new code placed in new module. We need implementing a simple LND (lustre network driver) with channel bounding algorithm (similar to socklnd has). Bounding LND will responsible to: · Mapping from virtual address to real on one of bounding interfaces. · Finding best interface to send · Handle peer health detection. Some background. Send path: When LNet want to send a message to peer, it start from finding NI (lustre network interface aka LND instance) – to sending it and call .lnd_send method of founded NI to transfer message over wire. LND has responsible to address resolution and free to change source and destination address. Connection to remote peer handled inside LND when LNet want a send message to remote peer and don’t visible to upper (LNet) layer. Receive path: When network driver found a incoming request on wire it read a header and call a lnet_parse() with self pointer to handle incoming message. Lnet_parse() call a .lnd_recv method to get data over wire into buffer. Single check exist on that codepath – is destination nid is assigned to local NI or not. Peer health detection – LND responsible to get health info for peer submitted from LNet and LNet scan a peer list to find unhealthy peers in timed manner. These preconditions use to create a design of bounding driver. Bounding LND structures · Own list of networks from each driver assigned to bounding as similar as routing code does. · Fake peer list, which a mapped to real peer, that is need to make peer health detection happy (that is similar socklnd has). · Implementation details. We need a parse line similar “networks” to bounding create a new NI’s. But these NI’s need to marked with special flag to prevent finding via lnet_net2ni_locked() (or we can use same interface for non bounding transfer ???). ->lnd_send method of that “bounding” LND has responsible found best NI via lnet_compare_routes() function. Source and destination NID of message has changed to route based similar as route code does (I think we can use simple rule as $network_start + offset). .lnd_send method of founded NI called after change source and destination address to transfer message over wire. lnet_notify should be changed to use new method in lnd_t structure which responsible to send notify event to upper layer instead of direct call lnet_notify from LND’s. Bounding LND need to own handler to handle notify events from lower NI. Handler responsible to resend failed message via different links and will send notify event to lnet only if peer don’t accessible over all links (that situation adds delay if node is down, but bounding will try send message over all links). That handler also responsible to resend messages Non-bounding LND’s need pointer to lnet_notify directly. Peer health detection – LNet uses two ways to check peers alive. One is periodic asking via .lnd_query method, second is notify event from lnet_notify. lnet_notify need be implemented in way pointed before. .lnd_query interface can supported via own list of virtual (fake) peers. Estimation size. I think it need to have ~1000-1200 LOCs, with lots reuse code from lnet routing, like are create, destroy, find best route and parsing network line. From andrew_perepechko at xyratex.com Wed Apr 25 21:00:02 2012 From: andrew_perepechko at xyratex.com (Andrew Perepechko) Date: Thu, 26 Apr 2012 01:00:02 +0400 Subject: [Twg] [Discuss] T10 End-to-End Data Integrity HLD In-Reply-To: <347CA580-C12E-4A1F-9FFD-AAD94E66F2DC@xyratex.com> References: <347CA580-C12E-4A1F-9FFD-AAD94E66F2DC@xyratex.com> Message-ID: <4F9865D2.6040906@xyratex.com> Hello Alex! This is my understanding of the end-to-end integrity framework: When a device is configured to work in DIX mode, Linux calculates tags when processing an I/O request (see bio_integrity_prep()) unless integrity data is already attached to the bio. We seem to have the following options about metadata vs DIX here: 1) do not change anything, let Linux do all of DIX work, this would detect errors, but Lustre is not prepared to handle them; this would also take additional CPU time to generate tags at Linux block layer; we need to check if there are any races similar to mmap-write races (buffer is changed after guard tags calculated). 2) use special tags 0xffff for metadata, errors will go undetected, little additional CPU time, mostly like without DIX support; 3) process DIX errors everywhere, restart metadata operations, refilling buffers from potentially safe sources. This would require some ost code rewrite. Even if we can simply restart most metadata operations, journal code needs to be changed to handle this; also, complex ldiskfs operations may fail after having updated some blocks on disk - there does not seem to be any code that attempts to handle that... Thank you, Andrew On 04/25/2012 01:19 AM, Alexey Lyashkov wrote: > some results of online discussion. > > Johann worry about journal checksumming (to avoid overhead in that area) > or other metadata checks on > client store uncommitted data in own queue for a replay (if need), but > in case journal checksum errors on write client will able to resend > failed request via special protocol extension (don't exist now). > same situation if we need a T10 support for these data we will be need > trigger a failover or change a wire protocol to resend a failed request. > failover is easily. > > > On Apr 24, 2012, at 14:53, Johann Lombardi wrote: > >> On Tue, Apr 24, 2012 at 9:44 PM, Alexey Lyashkov >> > wrote: >> >> what you mean as metadata blocks? extent's on OST or MDT disk io >> traffic ? >> >> >> Yes, anything that goes to the journal (inode, block bitmap, ...) >> which isn't user data. >> >> Johann > From doug at whamcloud.com Mon Apr 30 18:36:40 2012 From: doug at whamcloud.com (Doug Oucharek) Date: Mon, 30 Apr 2012 11:36:40 -0700 Subject: [Twg] LNet channel bounding Message-ID: Hi Alexey, Thank you for posting your Channel Bonding proposal. I'm Doug Oucharek and work on LNet at Whamcloud. I have a few questions/comments: 1- I'm assuming you will be calling the LND "Bonding" rather than "Bounding". 2- I'm assuming that the bonding LND would pass the message on to one of the other LND's to be actually sent (i.e. passes to the socklnd when sending to a TCP channel). This sort of breaks the architecture of a protocol layer which usually passes messages "down" the stack rather than laterally to another element in the same layer. I assume this is being proposed as it will make the coding much easier. Would it be much more difficult to put bonding support in LNet itself rather than having a separate LND? 3-There has been some work in LND's, particularly the qswlnd, to support multi-rail. This could be a good template on how to get the bonding LND to manage multiple networks in the bond. 4- Can you provided more details on how the virtual NID's get converted to the actual NID's for the different networks in the bond? Is all of this configured from module options or is there a configuration server involved (sort of like DNS, but for NID translation)? 5- What load balancing algorithms are being supported? If a weighted mechanism will be used, how will that work? Doug From carrier at cray.com Mon Apr 30 18:53:57 2012 From: carrier at cray.com (John Carrier) Date: Mon, 30 Apr 2012 18:53:57 +0000 Subject: [Twg] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... Message-ID: Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? --jc Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 Nathan Rutman Metadata server performance Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Mon Apr 30 18:59:42 2012 From: carrier at cray.com (John Carrier) Date: Mon, 30 Apr 2012 18:59:42 +0000 Subject: [Twg] LNet channel bounding In-Reply-To: References: Message-ID: I would like to remind everyone that Sun prepared an HLD for LNET channel bonding three years ago: http://wiki.lustre.org/images/e/ee/Channel_Bonding_06_15_09.pdf Lack of progress on the proposal was due to lack of funding at Sun (and their decision not to accept NRE), rather than any issue with the design itself. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Doug Oucharek Sent: Monday, April 30, 2012 11:37 AM To: twg at lists.opensfs.org Subject: [Twg] LNet channel bounding Hi Alexey, Thank you for posting your Channel Bonding proposal. I'm Doug Oucharek and work on LNet at Whamcloud. I have a few questions/comments: 1- I'm assuming you will be calling the LND "Bonding" rather than "Bounding". 2- I'm assuming that the bonding LND would pass the message on to one of the other LND's to be actually sent (i.e. passes to the socklnd when sending to a TCP channel). This sort of breaks the architecture of a protocol layer which usually passes messages "down" the stack rather than laterally to another element in the same layer. I assume this is being proposed as it will make the coding much easier. Would it be much more difficult to put bonding support in LNet itself rather than having a separate LND? 3-There has been some work in LND's, particularly the qswlnd, to support multi-rail. This could be a good template on how to get the bonding LND to manage multiple networks in the bond. 4- Can you provided more details on how the virtual NID's get converted to the actual NID's for the different networks in the bond? Is all of this configured from module options or is there a configuration server involved (sort of like DNS, but for NID translation)? 5- What load balancing algorithms are being supported? If a weighted mechanism will be used, how will that work? Doug _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From Nathan_Rutman at xyratex.com Mon Apr 30 19:23:16 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 30 Apr 2012 12:23:16 -0700 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? On Apr 30, 2012, at 11:53 AM, John Carrier wrote: Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? --jc Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 Nathan Rutman Metadata server performance Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From bzzz.tomas at gmail.com Mon Apr 30 19:47:23 2012 From: bzzz.tomas at gmail.com (Alex Tomas) Date: Mon, 30 Apr 2012 23:47:23 +0400 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> Message-ID: hmm? pdirops is not supposed to improve directory listing, AFAIK. thanks, Alex On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: > pdirops was delivered, but the improvement results were somewhat > disappointing. I've seen no metrics for various directory listing > improvements, although they may exist. > Should we set a target rate? E.g. "ls -al" for a dir with 100K > single-stripe files? > > > On Apr 30, 2012, at 11:53 AM, John Carrier wrote: > > Nathan adds a good question: metadata performance was the focus of our > last RFP. Are there any performance targets not covered by our current > development contract? > ** ** > --jc**** > ** ** > Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 > **** > *Nathan Rutman***** > Metadata server performance**** > Have we achieved the single-server MDS performance goals? Do we need to > move some more solid requirements to 2012?**** > ** ** > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oralhs at ornl.gov Mon Apr 30 20:03:23 2012 From: oralhs at ornl.gov (Oral, H. Sarp) Date: Mon, 30 Apr 2012 16:03:23 -0400 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: Message-ID: I think it was the statahead (I don't remember the bug number) to improve the "ls –la" performance… Sarp -- Sarp Oral, PhD National Center for Computational Sciences Oak Ridge National Laboratory oralhs at ornl.gov 865-574-2173 From: Alex Tomas > Date: Mon, 30 Apr 2012 15:47:23 -0400 To: Nathan Rutman > Cc: "twg at lists.opensfs.org" >, "discuss at lists.opensfs.org" > Subject: Re: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... hmm? pdirops is not supposed to improve directory listing, AFAIK. thanks, Alex On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman > wrote: pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? On Apr 30, 2012, at 11:53 AM, John Carrier wrote: Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? --jc Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 Nathan Rutman Metadata server performance Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From Nathan_Rutman at xyratex.com Mon Apr 30 20:41:35 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 30 Apr 2012 13:41:35 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <4F882A23.7050301@whamcloud.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> Chris and I talked a little at LUG about the Lustre testing suite, and came up with a some thoughts about how to get out of the current mess: 1. New language / library Bash has several problems: slow, non-portable, obscure syntax constructions, difficulty doing math, and most importantly it's not function-oriented. A replacement language that called clearly-defined functions with clear parameters allows for the building of a function library. Ideally current bash scripts could call the new library functions as we slowly replace the old framework. 2. Start new tests with new framework. New tests would be required to use the new framework, thus building up the library. 3. Remote operations via lctl messages. The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. Roman and I additionally had some more thoughts: 4. Clarify local / remote operations Often tests are written assuming a particular configuration (eg. llmount.sh) or version, and all tests execute local script data remotely. These must be rewritten as encapsulated library functions which may execute remote procedure calls. A library on the remote node would be responsible for executing the procedure. 5. Clarify test independence Currently many tests need all tests in a particular range to run, in sequence, in order to correctly set up and clean up a particular environment. The pre-requisites and post-requisites for each test need to be broken out from the tests themselves. 6. Add meta-information to tests Tags bearing descriptions and other characteristics of the tests should be included with each test to allow for automated test selection. E.g. label:quota could be used to run all quotas tests. 7. Remove version dependency Having the Lustre tests tied to the Lustre source makes it difficult to run interoperability tests consistently (e.g. use old or new version of tests?). Instead, we can separate out the tests into an independent set, and have interoperability matrixes for tests that should or should not be run. This also eliminates the need to backport tests between Lustre versions. 8. Splitting packages Lustre, lustre test suite, LNET, and ldiskfs should all be separated into different packages. Dependencies between package versions should be addressed by normal RPM methods. This is probably useful for a variety of different projects: zfs, the lustre-client-in-upstream-kernel, etc. On Apr 13, 2012, at 6:29 AM, Chris wrote: Nathan, discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass! I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed. It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled. There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run. I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly. I would also suggest that focusing on increasing test coverage and quality will create greater return then removing redundant tests, although if someone wanted to spend time seeking out the redundant tests it would not do any harm. 2) create single test suite Each release has its own test suite and assumes that the client and server will use the same test version. It is becoming common today to have different client/server versions. In this case, it is unclear which tests from which release package should be used. Instead, we should have a single test suite that accommodates different combinations of client and server versions. What is the problem statement here that leads to the conclusion that a single test suite should be produced? The test suite from 1.8.7 is not the same as from 2.2. Therefore, when running the tests on a 1.8.7 client talking to a 2.2 server, which versions' test suite should be run? An independent test RPM, divorced from the Lustre versioning, would provide a definitive answer. Yes, I know the tests change with code changes; they should change in the "master" test repo, and a mechanism should be implemented to skip known failures with older versions of non-interop (e.g. 1.8.7-1.8.7) and interop cases. This mechanism should probably involve pairs of server-client versions and simple lists of tests. This also solves the problem of having to backport each test to the old test suite as new tests are developed. In your example you would use 1.8.7 test scripts on the client, and 2.2 test scripts on the server. When the client makes an RPC to the server script it is making a functional request, the receiving script then performs that function in a way applicable to the server version in question. Whilst keeping the scripts separate does require back porting it also removes the need for the already unwieldy scripts to contain code to make them version aware. I would say with some certainty that separate scripts with explicit back-porting achieves a more reliable result than attempting to extend the bash scripts to cover multiple versions. Also I don't think we could merge 47000 lines of bash, and then verify results. Verification would require that we see each test pass and fail. We would also be unable to remove and verify the 1.8 parts of the new common test-framework once 1.8 becomes an unsupported product, which will happen one day. Don't get me wrong, if the test-framework was written in a modern, structured and object orientated language a single validation and verification framework might be possible, but today we don't have that. It can't be interoperability because the suites do run together ( b1_8 with b2_x ) and where interoperability issues exist tickets are raised and the issues resolved. It's worth noting that the tests behind the link do not perform significantly worse than a straight master test cycle and that most of the issues are Lustre not test-interoperability. Perhaps a complete problem statement will allow us to understand what the ROI would be from a single test suite. 3) extract individual tests from the scripts Many of the tests are run as a suite. There needs to be some mechanism to allow running components of the suite independently. To do this, we will need to make clear the dependencies of the subtests so that users know a test can be run only after another component of the test has executed. Also mentioned was the need for a common header on all test results to facilitate post-processing. What is to be achieved here that ONLY/EXCEPT does not allow, using a mixture of ONLY and EXCEPT any combination of tests can be run. That would be the ideal. Unfortunately, it simply is not the case today. Plenty of tests depend utterly on the tests before them doing some kind of setup or "first half" of a test. Definition of what constitutes a "test" and a "sub-test" muddies the issue somewhat. I think there is confusion here about dependencies and extracting the tests. If we extracted the tests the dependencies would still exist, and if we understood and solved the dependencies of each test then ONLY/EXCEPT would allow us to do exactly what was needed. So we are in agreement that we need to understand and document using meta-data the dependencies, but I don't really understand what extracting the tests from the scripts really buys us. I don't think it does harm, if the tools are put in place to allow developers to still run 'suites', but it certainly doesn't solve the dependency issue either. On the topic of meta data I think placing machine readable extensible data in a comments section above every test to store this data would make a lot of sense. We probably need nothing more than a set of key:value pairings. If people have seen good working solutions that should be considered it would be useful if they posted them. Yes, I'm a big proponent of YAML. Beyond this keyword descriptions of the test would allow automatic testing of specific areas, "acl" or "fopen" or "quotas" etc. I'll try an propose something to this list. Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture? My statement was not bold it was just a statement of fact. The current test-framework is not a suitable platform for fulfilling the requirements of tomorrow. We have to be building for the next 10 years which will include exascale, does anybody really feel that the current bash based, un-scalable and relatively unstructured test facility that we have today is that platform? I think not. The discussion here is a starting point specifically for identifying the current problems, with the hope of driving a an evolutionary way of getting there. Perhaps via OpenSFS we can drive this as a prioritized task of broad interest. We can all agree that we don't have the luxury of a big-bang change over to a new method, but evolution in this case needs to be a new species that will overtime become dominant and eventually replace the current bashous-thalensis! Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Mon Apr 30 20:48:14 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 30 Apr 2012 13:48:14 -0700 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> Right, these are different things. But both were intended to be metadata improvements. The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: hmm? pdirops is not supposed to improve directory listing, AFAIK. thanks, Alex On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? On Apr 30, 2012, at 11:53 AM, John Carrier wrote: Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? --jc Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 Nathan Rutman Metadata server performance Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Mon Apr 30 20:58:45 2012 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Mon, 30 Apr 2012 13:58:45 -0700 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> Message-ID: <4F9EFD05.8090607@llnl.gov> On 04/30/2012 01:41 PM, Nathan Rutman wrote: > 3. Remote operations via lctl messages. > The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. It is not immediately clear how this fixes anything. Such a facility built into lnet needs to have exactly the same configuration issues as ssh, munge, or whatever. If it DOESN'T have the same issues, then it sounds like it would be a very serious security risk. Such a feature needs to at least be off by default and only enabled by a human deciding that such a remote-access feature is acceptable. Chris From carrier at cray.com Mon Apr 30 21:00:53 2012 From: carrier at cray.com (John Carrier) Date: Mon, 30 Apr 2012 21:00:53 +0000 Subject: [Twg] [Lustre-devel] your opinion about testing In-Reply-To: <4F9EFD05.8090607@llnl.gov> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> <4F9EFD05.8090607@llnl.gov> Message-ID: Please reply further to this thread so that both twg and discuss lists are included. -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Christopher J. Morrone Sent: Monday, April 30, 2012 1:59 PM To: twg at lists.opensfs.org Subject: Re: [Twg] [Lustre-devel] your opinion about testing On 04/30/2012 01:41 PM, Nathan Rutman wrote: > 3. Remote operations via lctl messages. > The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. It is not immediately clear how this fixes anything. Such a facility built into lnet needs to have exactly the same configuration issues as ssh, munge, or whatever. If it DOESN'T have the same issues, then it sounds like it would be a very serious security risk. Such a feature needs to at least be off by default and only enabled by a human deciding that such a remote-access feature is acceptable. Chris _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From Nathan_Rutman at xyratex.com Mon Apr 30 22:37:59 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Mon, 30 Apr 2012 15:37:59 -0700 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> <4F9EFD05.8090607@llnl.gov> Message-ID: <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> > -----Original Message----- > From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Christopher J. Morrone > Sent: Monday, April 30, 2012 1:59 PM > To: twg at lists.opensfs.org > Subject: Re: [Twg] [Lustre-devel] your opinion about testing > > On 04/30/2012 01:41 PM, Nathan Rutman wrote: > >> 3. Remote operations via lctl messages. >> The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. > > It is not immediately clear how this fixes anything. Such a facility > built into lnet needs to have exactly the same configuration issues as > ssh, munge, or whatever. If it DOESN'T have the same issues, then it > sounds like it would be a very serious security risk. What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET. No router problems, no ssh'isms; if your client can talk to your servers, you're good to go. Obviously it doesn't resolve any security issues, and may introduce new ones. > > Such a feature needs to at least be off by default and only enabled by a > human deciding that such a remote-access feature is acceptable. Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here. The goal is to fix the testing, and I think this is one useful component of that. From adilger at whamcloud.com Mon Apr 30 23:01:07 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Mon, 30 Apr 2012 17:01:07 -0600 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> Message-ID: On 2012-04-30, at 2:48 PM, Nathan Rutman wrote: > Right, these are different things. But both were intended to be metadata improvements. > The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. I see this as a case of Amdahl's law - the MDS-only performance improvements from pdirops were reasonably good, but this is only part of the system being tested and improving performance of just the MDS will show diminishing returns if the OST performance isn't similarly improved. I've long suspected that the MDS-OSS precreate operations could be improved through some in-depth analysis and profiling, and Ben @ Terascala showed some of this at LUG 2011 as well. Some of the MDS-side performance improvements (pdirops) will be available in the OSS stack once it moves over to using the OFD module (i.e. obdfilter on top of OSD), and some of the current SMP scaling work being done for LNet, libcfs, ptlrpc will also apply to the OSS. > I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). I posted some of these results in a recent presentation: http://storageconference.com/2012/Presentations/T01.Dilger.pdf The graphs on slide 13 were generated based on data from Fan Yong's testing, and I'd think this was delivered to OpenSFS, but I don't know if there was a separate presentation for these results elsewhere. > The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? I think it would be reasonable to have follow-on project for OSS SMP tuning (including OST precreate, among other things) once the current SMP work for LNET/ptlrpc is finished, along with a similar investment on the client side to ensure that it is not hitting SMP scaling bottlenecks as the number of cores on client nodes goes through the roof. Cheers, Andreas > On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: > >> hmm? pdirops is not supposed to improve directory listing, AFAIK. >> >> thanks, Alex >> >> On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: >> pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. >> Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? >> >> >> On Apr 30, 2012, at 11:53 AM, John Carrier wrote: >> >>> Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? >>> >>> --jc >>> >>> Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 >>> Nathan Rutman >>> Metadata server performance >>> Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? >>> >>> _______________________________________________ >>> discuss mailing list >>> discuss at lists.opensfs.org >>> http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> >> > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/