From liang at whamcloud.com Tue May 1 08:15:44 2012 From: liang at whamcloud.com (Liang Zhen) Date: Tue, 1 May 2012 16:15:44 +0800 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> Message-ID: I would say performance improvement is a long term work, we can't expect one patch to resolve all performance issues. If there's a chance to make a longer term comparison, for example, compare 2.2 with 2.0, I think there should be nice improvement, we can't say which one contribute the most because one patch/feature is supposed to resolve one issue and we can see overall improvement with many of them. Regards Liang On May 1, 2012, at 4:48 AM, Nathan Rutman wrote: > Right, these are different things. But both were intended to be metadata improvements. > The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. > I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). > > The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? > > > On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: > >> hmm? pdirops is not supposed to improve directory listing, AFAIK. >> >> thanks, Alex >> >> On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: >> pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. >> Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? >> >> >> On Apr 30, 2012, at 11:53 AM, John Carrier wrote: >> >>> Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? >>> >>> --jc >>> >>> Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 >>> Nathan Rutman >>> Metadata server performance >>> Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? >>> >>> _______________________________________________ >>> discuss mailing list >>> discuss at lists.opensfs.org >>> http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> >> > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce_korb at xyratex.com Tue May 1 17:07:36 2012 From: bruce_korb at xyratex.com (Bruce Korb) Date: Tue, 01 May 2012 10:07:36 -0700 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> Message-ID: Hi, On 4/30/12 3:37 PM, Nathan Rutman wrote: >> >> It is not immediately clear how this fixes anything. Such a facility >> built into lnet needs to have exactly the same configuration issues as >> ssh, munge, or whatever. If it DOESN'T have the same issues, then it >> sounds like it would be a very serious security risk. >What it fixes is the ease of testing, by not requiring a separate >communication path outside of LNET. No router problems, no ssh'isms; if >your client can talk to your servers, you're good to go. >Obviously it doesn't resolve any security issues, and may introduce new >ones. It would introduce new ones. By the time you are done addressing them, you've likely spent the effort required to set up ssh daemons. For folks that find it difficult to configure, perhaps make a separate "lustre-testing" package that does all the configury necessary to set up ssh for a specific testing user. I think developing that would be far easier than rolling your own new mechanism. > >> >> Such a feature needs to at least be off by default and only enabled by >>a >> human deciding that such a remote-access feature is acceptable. >Agreed. Hopefully it might be easier to qualify a particular upcall >program as "safe", Installing the package would be a good human indicator. You'd want such a package to be easy to turn off, too, so it is only operational when testing is known to be active. > rather than allowing arbitrary code execution as ssh does, ssh can be constrained. Constrain it to a particular testing user. Over time, that user could have a testing harness shell that constrains the command set however you like. Having done this, I can say I found it fairly trivial. From Nathan_Rutman at xyratex.com Tue May 1 20:21:35 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 1 May 2012 13:21:35 -0700 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A92102F109FA@XYUS-EX22.xyus.xyratex.com> I really don't want to rathole here on the security implications of sending messages through LNET. I'll repost: this is a set of concrete proposals for improving the test situation with Lustre. My goal with this list is to spark other ideas, and see if we can get some momentum toward addressing the problems with the current testing framework. > Chris and I talked a little at LUG about the Lustre testing suite, and came up with a some thoughts about how to get out of the current mess: > > 1. New language / library > Bash has several problems: slow, non-portable, obscure syntax constructions, difficulty doing math, and most importantly it's not function-oriented. A replacement language that called clearly-defined functions with clear parameters allows for the building of a function library. Ideally current bash scripts could call the new library functions as we slowly replace the old framework. > > 2. Start new tests with new framework. > New tests would be required to use the new framework, thus building up the library. > > 3. Remote operations via lctl messages. > The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. > > Roman and I additionally had some more thoughts: > > 4. Clarify local / remote operations > Often tests are written assuming a particular configuration (eg. llmount.sh) or version, and all tests execute local script data remotely. These must be rewritten as encapsulated library functions which may execute remote procedure calls. A library on the remote node would be responsible for executing the procedure. > > 5. Clarify test independence > Currently many tests need all tests in a particular range to run, in sequence, in order to correctly set up and clean up a particular environment. The pre-requisites and post-requisites for each test need to be broken out from the tests themselves. > > 6. Add meta-information to tests > Tags bearing descriptions and other characteristics of the tests should be included with each test to allow for automated test selection. E.g. label:quota could be used to run all quotas tests. > > 7. Remove version dependency > Having the Lustre tests tied to the Lustre source makes it difficult to run interoperability tests consistently (e.g. use old or new version of tests?). Instead, we can separate out the tests into an independent set, and have interoperability matrixes for tests that should or should not be run. This also eliminates the need to backport tests between Lustre versions. > > 8. Splitting packages > Lustre, lustre test suite, LNET, and ldiskfs should all be separated into different packages. Dependencies between package versions should be addressed by normal RPM methods. This is probably useful for a variety of different projects: zfs, the lustre-client-in-upstream-kernel, etc. On May 1, 2012, at 10:07 AM, Bruce Korb wrote: > Hi, > > On 4/30/12 3:37 PM, Nathan Rutman wrote: >>> >>> It is not immediately clear how this fixes anything. Such a facility >>> built into lnet needs to have exactly the same configuration issues as >>> ssh, munge, or whatever. If it DOESN'T have the same issues, then it >>> sounds like it would be a very serious security risk. >> What it fixes is the ease of testing, by not requiring a separate >> communication path outside of LNET. No router problems, no ssh'isms; if >> your client can talk to your servers, you're good to go. >> Obviously it doesn't resolve any security issues, and may introduce new >> ones. > > It would introduce new ones. By the time you are done addressing them, > you've > likely spent the effort required to set up ssh daemons. For folks that > find > it difficult to configure, perhaps make a separate "lustre-testing" package > that does all the configury necessary to set up ssh for a specific testing > user. > I think developing that would be far easier than rolling your own new > mechanism. > >> >>> >>> Such a feature needs to at least be off by default and only enabled by >>> a >>> human deciding that such a remote-access feature is acceptable. >> Agreed. Hopefully it might be easier to qualify a particular upcall >> program as "safe", > > Installing the package would be a good human indicator. > You'd want such a package to be easy to turn off, too, > so it is only operational when testing is known to be active. > >> rather than allowing arbitrary code execution as ssh does, > > ssh can be constrained. Constrain it to a particular testing user. > Over time, that user could have a testing harness shell that constrains > the command set however you like. Having done this, I can say I found > it fairly trivial. > > From Nathan_Rutman at xyratex.com Tue May 1 20:50:46 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 1 May 2012 13:50:46 -0700 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F109FB@XYUS-EX22.xyus.xyratex.com> Thanks Andreas for that link. These look like great improvements certainly - but on an absolute scale: is one minute for 100k 0-stripe files sufficient? 4.5 min for 100k 4-stripe files? If not, is there a particular number that we think will be acceptable? Anyone have some GPFS numbers? For the graph below, are these 0-stripe files? Liang's original pdirops report didn't show nearly these gains. Are there further changes is 2.3? On Apr 30, 2012, at 4:01 PM, Andreas Dilger wrote: On 2012-04-30, at 2:48 PM, Nathan Rutman wrote: Right, these are different things. But both were intended to be metadata improvements. The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. I see this as a case of Amdahl's law - the MDS-only performance improvements from pdirops were reasonably good, but this is only part of the system being tested and improving performance of just the MDS will show diminishing returns if the OST performance isn't similarly improved. I've long suspected that the MDS-OSS precreate operations could be improved through some in-depth analysis and profiling, and Ben @ Terascala showed some of this at LUG 2011 as well. Some of the MDS-side performance improvements (pdirops) will be available in the OSS stack once it moves over to using the OFD module (i.e. obdfilter on top of OSD), and some of the current SMP scaling work being done for LNet, libcfs, ptlrpc will also apply to the OSS. I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). I posted some of these results in a recent presentation: http://storageconference.com/2012/Presentations/T01.Dilger.pdf The graphs on slide 13 were generated based on data from Fan Yong's testing, and I'd think this was delivered to OpenSFS, but I don't know if there was a separate presentation for these results elsewhere. The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? I think it would be reasonable to have follow-on project for OSS SMP tuning (including OST precreate, among other things) once the current SMP work for LNET/ptlrpc is finished, along with a similar investment on the client side to ensure that it is not hitting SMP scaling bottlenecks as the number of cores on client nodes goes through the roof. Cheers, Andreas On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: hmm? pdirops is not supposed to improve directory listing, AFAIK. thanks, Alex On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? On Apr 30, 2012, at 11:53 AM, John Carrier wrote: Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? --jc Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 Nathan Rutman Metadata server performance Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-2.tiff Type: image/tiff Size: 1076054 bytes Desc: PastedGraphic-2.tiff URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: PastedGraphic-1.tiff Type: image/tiff Size: 1009486 bytes Desc: PastedGraphic-1.tiff URL: From adilger at whamcloud.com Tue May 1 21:17:52 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 1 May 2012 15:17:52 -0600 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <73AED5C780AE05478241DB067651A92102F109FB@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109FB@XYUS-EX22.xyus.xyratex.com> Message-ID: <312B7FC2-14F9-4861-A1F0-26C522F1F288@whamcloud.com> On 2012-05-01, at 2:50 PM, Nathan Rutman wrote: > Thanks Andreas for that link. > These look like great improvements certainly - but on an absolute scale: > is one minute for 100k 0-stripe files sufficient? > 4.5 min for 100k 4-stripe files? > If not, is there a particular number that we think will be acceptable? Anyone have some GPFS numbers? These are definitely good questions. I noticed from a number of other papers and presentations that show e.g. GPFS or Ceph scaling file create performance as MDS nodes are added that the performance can definitely scale, but if you look at the absolute numbers they are 1/2 or 1/8th of the performance from a single Lustre MDS. So, I don't have performance numbers for "ls -l", also recall that this is a single-threaded process on the client, to which the in-kernel statahead is adding automatic parallelism. The same mechanism (multi-threaded ptlrpcd) is also improving the single-client checksum performance. I'm not against having an ongoing task to investigate performance bottlenecks, understand the root problems, and possibly resolve or document for later rework. That said, I think we should "spread the love" to the OSS and client as well, so that we can fix the obvious problems there before spending more time on optimizing the MDS further. > > > For the graph below, are these 0-stripe files? Liang's original pdirops report didn't show nearly these gains. Are there further changes is 2.3? Yes, and the "2.3" performance numbers are based on early version of the SMP scaling and affinity changes (LNET locking, ptlrpc service thread locking, NUMA affinity), which is why it is labelled 2.3 and not 2.2. > > > On Apr 30, 2012, at 4:01 PM, Andreas Dilger wrote: > >> On 2012-04-30, at 2:48 PM, Nathan Rutman wrote: >>> Right, these are different things. But both were intended to be metadata improvements. >>> The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. >> >> I see this as a case of Amdahl's law - the MDS-only performance improvements from pdirops were reasonably good, but this is only part of the system being tested and improving performance of just the MDS will show diminishing returns if the OST performance isn't similarly improved. >> >> I've long suspected that the MDS-OSS precreate operations could be improved through some in-depth analysis and profiling, and Ben @ Terascala showed some of this at LUG 2011 as well. Some of the MDS-side performance improvements (pdirops) will be available in the OSS stack once it moves over to using the OFD module (i.e. obdfilter on top of OSD), and some of the current SMP scaling work being done for LNet, libcfs, ptlrpc will also apply to the OSS. >> >>> I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). >> >> I posted some of these results in a recent presentation: >> http://storageconference.com/2012/Presentations/T01.Dilger.pdf >> >> The graphs on slide 13 were generated based on data from Fan Yong's testing, and I'd think this was delivered to OpenSFS, but I don't know if there was a separate presentation for these results elsewhere. >> >>> The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? >> >> I think it would be reasonable to have follow-on project for OSS SMP tuning (including OST precreate, among other things) once the current SMP work for LNET/ptlrpc is finished, along with a similar investment on the client side to ensure that it is not hitting SMP scaling bottlenecks as the number of cores on client nodes goes through the roof. >> >> Cheers, Andreas >> >>> On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: >>> >>>> hmm? pdirops is not supposed to improve directory listing, AFAIK. >>>> >>>> thanks, Alex >>>> >>>> On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: >>>> pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. >>>> Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? >>>> >>>> >>>> On Apr 30, 2012, at 11:53 AM, John Carrier wrote: >>>> >>>>> Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? >>>>> >>>>> --jc >>>>> >>>>> Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 >>>>> Nathan Rutman >>>>> Metadata server performance >>>>> Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list >>>>> discuss at lists.opensfs.org >>>>> http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org >>>> >>>> >>>> _______________________________________________ >>>> twg mailing list >>>> twg at lists.opensfs.org >>>> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >>>> >>>> >>> >>> _______________________________________________ >>> twg mailing list >>> twg at lists.opensfs.org >>> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> >> >> Cheers, Andreas >> -- >> Andreas Dilger Whamcloud, Inc. >> Principal Lustre Engineer http://www.whamcloud.com/ >> >> >> >> > Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Nathan_Rutman at xyratex.com Tue May 1 22:25:10 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 1 May 2012 15:25:10 -0700 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <312B7FC2-14F9-4861-A1F0-26C522F1F288@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109FB@XYUS-EX22.xyus.xyratex.com> <312B7FC2-14F9-4861-A1F0-26C522F1F288@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F109FC@XYUS-EX22.xyus.xyratex.com> On May 1, 2012, at 2:17 PM, Andreas Dilger wrote: > On 2012-05-01, at 2:50 PM, Nathan Rutman wrote: >> Thanks Andreas for that link. >> These look like great improvements certainly - but on an absolute scale: >> is one minute for 100k 0-stripe files sufficient? >> 4.5 min for 100k 4-stripe files? >> If not, is there a particular number that we think will be acceptable? Anyone have some GPFS numbers? > > These are definitely good questions. I noticed from a number of other papers and presentations that show e.g. GPFS or Ceph scaling file create performance as MDS nodes are added that the performance can definitely scale, but if you look at the absolute numbers they are 1/2 or 1/8th of the performance from a single Lustre MDS. > > So, I don't have performance numbers for "ls -l", also recall that this is a single-threaded process on the client, to which the in-kernel statahead is adding automatic parallelism. The same mechanism (multi-threaded ptlrpcd) is also improving the single-client checksum performance. > > I'm not against having an ongoing task to investigate performance bottlenecks, understand the root problems, and possibly resolve or document for later rework. That said, I think we should "spread the love" to the OSS and client as well, so that we can fix the obvious problems there before spending more time on optimizing the MDS further. That's fine by me. I actually think that client-osc interaction for MD ops (e.g. file size, unlink) could stand some focus. > >> >> >> For the graph below, are these 0-stripe files? Liang's original pdirops report didn't show nearly these gains. Are there further changes is 2.3? > > Yes, and the "2.3" performance numbers are based on early version of the SMP scaling and affinity changes (LNET locking, ptlrpc service thread locking, NUMA affinity), which is why it is labelled 2.3 and not 2.2. > >> >> >> On Apr 30, 2012, at 4:01 PM, Andreas Dilger wrote: >> >>> On 2012-04-30, at 2:48 PM, Nathan Rutman wrote: >>>> Right, these are different things. But both were intended to be metadata improvements. >>>> The pdirops results have been posted, but for actual files (e.g. with OST objects) the create/unlink rates didn't show much improvement. >>> >>> I see this as a case of Amdahl's law - the MDS-only performance improvements from pdirops were reasonably good, but this is only part of the system being tested and improving performance of just the MDS will show diminishing returns if the OST performance isn't similarly improved. >>> >>> I've long suspected that the MDS-OSS precreate operations could be improved through some in-depth analysis and profiling, and Ben @ Terascala showed some of this at LUG 2011 as well. Some of the MDS-side performance improvements (pdirops) will be available in the OSS stack once it moves over to using the OFD module (i.e. obdfilter on top of OSD), and some of the current SMP scaling work being done for LNet, libcfs, ptlrpc will also apply to the OSS. >>> >>>> I have not seen any measurements of 'ls -la' for any delivered features (although that doesn't mean they don't exist). >>> >>> I posted some of these results in a recent presentation: >>> http://storageconference.com/2012/Presentations/T01.Dilger.pdf >>> >>> The graphs on slide 13 were generated based on data from Fan Yong's testing, and I'd think this was delivered to OpenSFS, but I don't know if there was a separate presentation for these results elsewhere. >>> >>>> The question now is, do we as OpenSFS members want to continue to prioritize single-server MDS performance improvements? >>> >>> I think it would be reasonable to have follow-on project for OSS SMP tuning (including OST precreate, among other things) once the current SMP work for LNET/ptlrpc is finished, along with a similar investment on the client side to ensure that it is not hitting SMP scaling bottlenecks as the number of cores on client nodes goes through the roof. >>> >>> Cheers, Andreas >>> >>>> On Apr 30, 2012, at 12:47 PM, Alex Tomas wrote: >>>> >>>>> hmm? pdirops is not supposed to improve directory listing, AFAIK. >>>>> >>>>> thanks, Alex >>>>> >>>>> On Mon, Apr 30, 2012 at 11:23 PM, Nathan Rutman wrote: >>>>> pdirops was delivered, but the improvement results were somewhat disappointing. I've seen no metrics for various directory listing improvements, although they may exist. >>>>> Should we set a target rate? E.g. "ls -al" for a dir with 100K single-stripe files? >>>>> >>>>> >>>>> On Apr 30, 2012, at 11:53 AM, John Carrier wrote: >>>>> >>>>>> Nathan adds a good question: metadata performance was the focus of our last RFP. Are there any performance targets not covered by our current development contract? >>>>>> >>>>>> --jc >>>>>> >>>>>> Nathan Rutman added a comment to OpenSFS TWG Requirements 2012 >>>>>> Nathan Rutman >>>>>> Metadata server performance >>>>>> Have we achieved the single-server MDS performance goals? Do we need to move some more solid requirements to 2012? >>>>>> >>>>>> _______________________________________________ >>>>>> discuss mailing list >>>>>> discuss at lists.opensfs.org >>>>>> http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org >>>>> >>>>> >>>>> _______________________________________________ >>>>> twg mailing list >>>>> twg at lists.opensfs.org >>>>> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> twg mailing list >>>> twg at lists.opensfs.org >>>> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >>> >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger Whamcloud, Inc. >>> Principal Lustre Engineer http://www.whamcloud.com/ >>> >>> >>> >>> >> > > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > From chris at whamcloud.com Wed May 2 09:36:40 2012 From: chris at whamcloud.com (Chris) Date: Wed, 02 May 2012 10:36:40 +0100 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: References: Message-ID: <4FA10028.3040703@whamcloud.com> > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 1 May 2012 13:21:35 -0700 > From: "Nathan Rutman" > To: "Bruce Korb" > Cc: discuss at lists.opensfs.org, twg at lists.opensfs.org > Subject: Re: [Twg] [Discuss] [Lustre-devel] your opinion about > testing > Message-ID: > <73AED5C780AE05478241DB067651A92102F109FA at XYUS-EX22.xyus.xyratex.com> > Content-Type: text/plain; charset="iso-8859-1" > > I really don't want to rathole here on the security implications of sending messages through LNET. > > I'll repost: this is a set of concrete proposals for improving the test situation with Lustre. > My goal with this list is to spark other ideas, and see if we can get some momentum toward addressing the problems with the current testing framework. > Nathan, Thanks for starting this conversation. Can I suggest that we set-up a Wiki page somewhere that everybody interested can access to allow this much needed proposal to be fleshed out and matured before it is turned into a concrete document. If you want to host the public page then that's fine by me, otherwise we can find space on the Whamcloud Wiki. The purpose of the process should I think be to (in priority and logical order); 1. Identify the areas where improvement is necessary, including a clear description of the current issues which need to be resolved 2. Identify the features which must be carried forward with a clear description of the current feature 3. Identify additional improvements that extend test forward to support exascale and beyond (cue Buzz Lightyear impression) 4. Identify some candidate proposals to requirements (requirements = required improvements + required existing features + additional improvements) 5. The 20 other things I've not thought of so please fill in as required! 6. Work out a strategy to implement/build a system that fulfils the proposals 7. Find some money to build the grand design we came up with in 6 Does that make sense? Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roman_Grigoryev at xyratex.com Wed May 2 15:44:31 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Wed, 02 May 2012 19:44:31 +0400 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> <4F9EFD05.8090607@llnl.gov> <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> Message-ID: <4FA1565F.7040505@xyratex.com> Hi Nathan, I think it is very interesting possibility but, but from QA/QE logic there is problem : we are planing to use for test control messages channel which is under test too. So, we have to have separated lnet test for prove that lnet is working ok before starting full testing session. Also we must keep possibility to execute tests (maybe not all) without this feature, in classic way, f.e. via ssh. Thanks, Roman On 05/01/2012 02:37 AM, Nathan Rutman wrote: >> >>> 3. Remote operations via lctl messages. >>> The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. >> >> It is not immediately clear how this fixes anything. Such a facility >> built into lnet needs to have exactly the same configuration issues as >> ssh, munge, or whatever. If it DOESN'T have the same issues, then it >> sounds like it would be a very serious security risk. > What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET. No router problems, no ssh'isms; if your client can talk to your servers, you're good to go. > Obviously it doesn't resolve any security issues, and may introduce new ones. > >> >> Such a feature needs to at least be off by default and only enabled by a >> human deciding that such a remote-access feature is acceptable. > Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here. > The goal is to fix the testing, and I think this is one useful component of that. > _______________________________________________ From Nathan_Rutman at xyratex.com Wed May 2 16:19:08 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 2 May 2012 09:19:08 -0700 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: <4FA1565F.7040505@xyratex.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> <4F9EFD05.8090607@llnl.gov> <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> <4FA1565F.7040505@xyratex.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10A00@XYUS-EX22.xyus.xyratex.com> On May 2, 2012, at 8:44 AM, Roman Grigoryev wrote: > Hi Nathan, > > I think it is very interesting possibility but, but from QA/QE logic > there is problem : we are planing to use for test control messages > channel which is under test too. So, we have to have separated lnet test > for prove that lnet is working ok before starting full testing session. To some degree. Most (all?) of the test-framework current tests are actually Lustre tests, and not LNET tests, so I don't think this will have much of an impact. If there's some minor additional LNET load due to the control message overhead, I would expect LNET to handle this reasonably -- LNET has historically been quite robust. This actually ties in to the "8. Splitting packages" point; if LNET and Lustre are separate packages, I expect Lustre testing could depend on proper LNET setup and operation. The bigger issue to me is that this does require a network and/or Lustre FS to already be set up -- configured, routed, running at least LNET. This is a fairly big change from the current situation, where we assume we can start/stop/reconfigure things willy-nilly. However, I also think that it's probably a good thing to separate the configuration / setup of the cluster from the test execution in most cases. > Also we must keep possibility to execute tests (maybe not all) without > this feature, in classic way, f.e. via ssh. I think that's probably true; in any case it needs to be encapsulated in a library layer. I think in general terms the simplification of running the tests in complex environments will be worth the possibly reduced scope -- but others please chime in! This is my opinion of writing/using the tests as a developer, and may not match other experiences. > > Thanks, > Roman > > > On 05/01/2012 02:37 AM, Nathan Rutman wrote: >>> >>>> 3. Remote operations via lctl messages. >>>> The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. >>> >>> It is not immediately clear how this fixes anything. Such a facility >>> built into lnet needs to have exactly the same configuration issues as >>> ssh, munge, or whatever. If it DOESN'T have the same issues, then it >>> sounds like it would be a very serious security risk. >> What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET. No router problems, no ssh'isms; if your client can talk to your servers, you're good to go. >> Obviously it doesn't resolve any security issues, and may introduce new ones. >> >>> >>> Such a feature needs to at least be off by default and only enabled by a >>> human deciding that such a remote-access feature is acceptable. >> Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here. >> The goal is to fix the testing, and I think this is one useful component of that. >> _______________________________________________ From adilger at whamcloud.com Wed May 2 16:30:43 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 2 May 2012 10:30:43 -0600 Subject: [Twg] [Discuss] [Lustre-devel] your opinion about testing In-Reply-To: <73AED5C780AE05478241DB067651A92102F10A00@XYUS-EX22.xyus.xyratex.com> References: <4F86D9A1.8090105@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com> <4F882A23.7050301@whamcloud.com> <73AED5C780AE05478241DB067651A92102F109F3@XYUS-EX22.xyus.xyratex.com> <4F9EFD05.8090607@llnl.gov> <73AED5C780AE05478241DB067651A92102F109F5@XYUS-EX22.xyus.xyratex.com> <4FA1565F.7040505@xyratex.com> <73AED5C780AE05478241DB067651A92102F10A00@XYUS-EX22.xyus.xyratex.com> Message-ID: <94A51991-2687-41D0-84A7-A3DF4E09C17F@whamcloud.com> On 2012-05-02, at 10:19 AM, Nathan Rutman wrote: > On May 2, 2012, at 8:44 AM, Roman Grigoryev wrote: >> I think it is very interesting possibility but, but from QA/QE logic >> there is problem : we are planing to use for test control messages >> channel which is under test too. So, we have to have separated lnet test >> for prove that lnet is working ok before starting full testing session. > > To some degree. Most (all?) of the test-framework current tests are actually Lustre tests, and not LNET tests, so I don't think this will have much of an impact. If there's some minor additional LNET load due to the control message overhead, I would expect LNET to handle this reasonably -- LNET has historically been quite robust. This actually ties in to the "8. Splitting packages" point; if LNET and Lustre are separate packages, I expect Lustre testing could depend on proper LNET setup and operation. > > The bigger issue to me is that this does require a network and/or Lustre FS to already be set up -- configured, routed, running at least LNET. LNET and LST can be run independently of Lustre. > This is a fairly big change from the current situation, where we assume we can start/stop/reconfigure things willy-nilly. However, I also think that it's probably a good thing to separate the configuration / setup of the cluster from the test execution in most cases. Right, and this is done, or ongoing in the existing tests (e.g. check_and_setup_lustre()). >> Also we must keep possibility to execute tests (maybe not all) without >> this feature, in classic way, f.e. via ssh. > > I think that's probably true; in any case it needs to be encapsulated in a library layer. Sure, like do_facet() or do_nodes(). Any test, or even helper function, calling SSH/RSH/pdsh directly should be considered broken. > I think in general terms the simplification of running the tests in complex environments will be worth the possibly reduced scope -- but others please chime in! This is my opinion of writing/using the tests as a developer, and may not match other experiences. > >> >> Thanks, >> Roman >> >> >> On 05/01/2012 02:37 AM, Nathan Rutman wrote: >>>> >>>>> 3. Remote operations via lctl messages. >>>>> The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required. >>>> >>>> It is not immediately clear how this fixes anything. Such a facility >>>> built into lnet needs to have exactly the same configuration issues as >>>> ssh, munge, or whatever. If it DOESN'T have the same issues, then it >>>> sounds like it would be a very serious security risk. >>> What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET. No router problems, no ssh'isms; if your client can talk to your servers, you're good to go. >>> Obviously it doesn't resolve any security issues, and may introduce new ones. >>> >>>> >>>> Such a feature needs to at least be off by default and only enabled by a >>>> human deciding that such a remote-access feature is acceptable. >>> Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here. >>> The goal is to fix the testing, and I think this is one useful component of that. >>> _______________________________________________ > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From carrier at cray.com Wed May 2 23:50:48 2012 From: carrier at cray.com (John Carrier) Date: Wed, 2 May 2012 23:50:48 +0000 Subject: [Twg] "TWG" and "discuss" reflectors. Message-ID: Dave and I would like to restructure how the TWG uses our two email reflectors, "TWG" and "discuss". The existence and use of these two reflectors are confusing to people exploring OpenSFS. TWG discussions are open and are posted on both reflectors. "discuss" is open to all subscribers from the Lustre community. Though its archive is open to the public, the "TWG" reflector only allows subscribers from OpenSFS member organizations. The "TWG" reflector exists for those few topics where input only from OpenSFS members is appropriate. One such topic is how we allocate funds for future development. All members of the Lustre community have been encouraged to propose features and participate in the requirements conversation, but only organizations that have contributed money to support OpenSFS directly will have a vote on what we fund. Our goal is to make access to the TWG easier and our use of the two reflectors less confusing. As a result, we propose that we change how we use the two reflectors. "discuss" should be the reflector for all Lustre community technical discussions and "TWG" should be the reflector for members-only business. "TWG" will still be world readable, but only members can contribute content. This change will mean that we will move all non-OpenSFS subscribers to the "discuss" list. We would like to discuss this briefly during tomorrow's TWG concall (9:30a PT). Your feedback will be greatly appreciated. Thanks, John & Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu May 3 05:39:02 2012 From: carrier at cray.com (John Carrier) Date: Thu, 3 May 2012 05:39:02 +0000 Subject: [Twg] meeting reminder for Thu 5/3/12 Message-ID: The OpenSFS Technical Working Group will be meeting Thursday (5/3/2012) at 12:30ET/9:30PT. Conference info: Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 There have been a couple good discussions on the reflectors. In particular, there may be a new metadata performance requirement to add to our list. Please post agenda topics to the reflectors. Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu May 3 08:35:16 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 3 May 2012 02:35:16 -0600 Subject: [Twg] [Discuss] FW: OpenSFS TWG Requi... - Have we achieved the single-server MD... In-Reply-To: <312B7FC2-14F9-4861-A1F0-26C522F1F288@whamcloud.com> References: <73AED5C780AE05478241DB067651A92102F109EE@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109F4@XYUS-EX22.xyus.xyratex.com> <73AED5C780AE05478241DB067651A92102F109FB@XYUS-EX22.xyus.xyratex.com> <312B7FC2-14F9-4861-A1F0-26C522F1F288@whamcloud.com> Message-ID: <631215A7-E0B9-4EA2-83DD-ACD8CC7F0732@whamcloud.com> An HTML attachment was scrubbed... URL: From jjw at iu.edu Thu May 3 17:17:36 2012 From: jjw at iu.edu (Joshua Walgenbach) Date: Thu, 03 May 2012 13:17:36 -0400 Subject: [Twg] requirements Message-ID: <4FA2BDB0.4010502@iu.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 List is sorted by priority, where the top is a higher priority. - -Josh -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+iva8ACgkQcqyJPuRTYp+2XgCfVg3/Gh+ngaVt+zBBrL430pIy fDMAn2Cje6tjKgDn1//OlFsyQceZ0D7F =jy2b -----END PGP SIGNATURE----- -------------- next part -------------- single client IO performance better support for newer kernel support avoid rpc timeouts improved lnet robustness unified storage target HSM and storage pool quotas administrative shutdown ost migration/rebuilding improved lustre tests From carrier at cray.com Thu May 3 18:14:09 2012 From: carrier at cray.com (John Carrier) Date: Thu, 3 May 2012 18:14:09 +0000 Subject: [Twg] requirements doc update Message-ID: During our TWG call today, we agreed that we should continue to push for metadata performance improvements and decided to add two new requirements for consideration as 2013 OpenSFS development projects: * File create performance * Directory traversal and attribute retrieval Please review the updated requirements document (http://goo.gl/63u9Q) when selecting your top10 requirements for the TWG to submit to the OpenSFS Board. Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Tue May 8 16:46:05 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 8 May 2012 10:46:05 -0600 Subject: [Twg] OpenSFS 2012 TWG Requirements References: <01d101cd2b4d$5e5a8840$1b0f98c0$@com> Message-ID: After internal discussion, the Whamcloud feature list in order of decreasing preference is: 1) OST migration/rebalancing 2) Dynamic LNET configuration 3) Asynchronous file mirroring 4) File create performance (+ other OSS performance tuning) 5) Unified storage target 6) Storage pool quotas 7) Complex file layouts 8) Single client IO performance 9) Administrative Shutdown 10) Dynamic file layout Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From jacques-charles.lafoucriere at cea.fr Wed May 9 11:07:37 2012 From: jacques-charles.lafoucriere at cea.fr (Jacques-Charles Lafoucriere) Date: Wed, 09 May 2012 13:07:37 +0200 Subject: [Twg] TWG Requirements Message-ID: <4FAA4FF9.8080907@cea.fr> Hello all CEA requirements priority list is: 1) Scalable fault management 2) Directory traversal and attribute retrieval 3) Avoid RPC timeouts 4) OST migration/rebalancing 5) Improved configuration robustness 6) Single client IO performance 7) POSIX extensions for file sets 8) Improved security infrastructure 9) Complex file layouts 10) Improved Lustre Tests Cheers, JC -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastien.buisson at bull.net Thu May 10 06:25:16 2012 From: sebastien.buisson at bull.net (=?ISO-8859-1?Q?S=E9bastien_Buisson?=) Date: Thu, 10 May 2012 08:25:16 +0200 Subject: [Twg] TWG Requirements Message-ID: <4FAB5F4C.5030702@bull.net> Hi there, Bull prioritized requirements list is: 1. HSM and storage management infrastructure 2. Avoid RPC timeouts 3. Dynamic LNET Configuration 4. Quality of service 5. POSIX extensions for file sets 6. Dynamic layout for subset of a file 7. LNET channel Bonding 8. Snapshots 9. Locality and scalability 10. Scalable fault management Cheers, Sebastien. From Roger.Spellman at terascala.com Thu May 10 16:08:40 2012 From: Roger.Spellman at terascala.com (Roger Spellman) Date: Thu, 10 May 2012 12:08:40 -0400 Subject: [Twg] TWG Requirements Message-ID: <2C7DE72B9BD00F44BAECA5B0CBB8739501CC32D0@hermes.terascala.com> Terascala's list is as follows. 1. Scalable fault management 2. HSM and storage management infrastructure 3. Directory traversal and attribute retrieval 4. Better support for newer kernels 5. Unified storage target 6. Avoid RPC timeouts 7. Asynchronous file replication (mirroring) 8. OST migration/rebalancing 9. Single shared file performance 10. BEN Quality of service Roger Spellman Staff Engineer Terascala, Inc. 508-588-1501 www.terascala.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhenwood at whamcloud.com Fri May 11 20:23:20 2012 From: rhenwood at whamcloud.com (Richard Henwood) Date: Fri, 11 May 2012 15:23:20 -0500 Subject: [Twg] MDS performance: SMP Node Affinity Docs for OpenSFS NRE Message-ID: Hi All, OpenSFS and Whamcloud are engaged in delivering high-priority enhancements to Lustre for the community. The Solution Architecture and High Level Design for MDS Performance: SMP Node Affinity have been completed and are available for comments and feedback: http://wiki.whamcloud.com/display/PUB/MDS+SMP+Node+Affinity+Solution+Architecture http://wiki.whamcloud.com/display/PUB/MDS+SMP+Node+Affinity+High+Level+Design With best regards, Richard -- Richard.Henwood at whamcloud.com Whamcloud Inc. tel: +1 512 410 9612 From Alexander_Lezhoev at xyratex.com Mon May 14 20:59:16 2012 From: Alexander_Lezhoev at xyratex.com (Alexander Lezhoev) Date: Tue, 15 May 2012 00:59:16 +0400 Subject: [Twg] Separated test execution Message-ID: <4FB17224.1050105@xyratex.com> Hi there, Let me raise the question about Lustre tests separated execution. We've discussed this problem already, but I'd like to clear up some details. Usually we run all tests sequentially, but in the automation tool we are using we need to run tests separately, with ONLY parameter. This allows us to have full control over the test execution: terminate hung tests by timeout or restore environment in case of the file system damage. At the moment some of tests are designed to be run sequentially. By our estimation, there are about 30 tests need for the improvement. If we settle this question, we can significantly improve test-framework automation potential. Please share your opinions about this question and help to make a decision about it. The questions are 1. Do we want to have an ability to run each test independently? 2. What is more acceptable - unite sequential tests into complex ones or supplement exists test with additional code steps? Some technical details: Typical problem is sanityn test_1 test_1a() { touch $DIR1/f1 [ -f $DIR2/f1 ] || error } run_test 1a "check create on 2 mtpt's ==========================" test_1b() { chmod 777 $DIR2/f1 $CHECKSTAT -t file -p 0777 $DIR1/f1 || error chmod a-x $DIR2/f1 } run_test 1b "check attribute updates on 2 mtpt's ===============" test_1c() { $CHECKSTAT -t file -p 0666 $DIR1/f1 || error } run_test 1c "check after remount attribute updates on 2 mtpt's =" test_1d() { rm $DIR2/f1 $CHECKSTAT -a $DIR1/f1 || error } run_test 1d "unlink on one mountpoint removes file on other ====" They cannot be run separately, because the next index uses the code of previous one. This means all tests should be run in groups of letter indexes, or they should be refactored to run independently. Some of tests have been already refactored to run "letters" separately, but we have to make a rule which we should follow and use for further refactoring. There are three decisions we can take about this situation * Join the code of all test steps into single test with corresponding number. So we will have one test_1 instead of test_1a .. test_1d in the described case. * Move the code of steps to corresponding functions which will be called from each step. In other words the next indexed test will duplicate some functionality of previous one. * Do nothing and decide that "letters" mustn't be executed independently, but only in "number" group. The first variant could be implemented as follows. test_1() { touch $DIR1/f1 [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed" chmod 777 $DIR2/f1 $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates on 2 mtpt's failed" chmod a-x $DIR2/f1 $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount attribute updates on 2 mtpt's failed" rm $DIR2/f1 $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file on other failed" } run_test 1 "check attributes updates on 2 mtpt's" This approach has disadvantage that such kind of refactoring will lead to reduction of test numbering and it will hard to work with regression history of the refactored tests. The second case of refactoring can look like this: test_1a() { test_1_create } run_test 1a "check create on 2 mtpt's ==========================" test_1b() { test_1_create test_1_check_attr } run_test 1b "check attribute updates on 2 mtpt's ===============" test_1c() { test_1_create test_1_check_attr test_1_check_attr2 } run_test 1c "check after remount attribute updates on 2 mtpt's =" test_1d() { test_1_create test_1_check_attr test_1_check_attr2 test_1_unlink } run_test 1d "unlink on one mountpoint removes file on other ====" I've omitted functions code - their content is obvious. This disadvantage of this approach --- summary increase of tests run-time (the next test duplicates code of previous one). But the necessity of all these tests is doubtful here, because the last one includes first three tests. Very similar situation is for recovery-small 1, 2, 3 and 4, 5. test_1() { drop_request "mcreate $DIR/f1" || return 1 drop_reint_reply "mcreate $DIR/f2" || return 2 } run_test 1 "mcreate: drop req, drop rep" test_2() { drop_request "tchmod 111 $DIR/f2" || return 1 drop_reint_reply "tchmod 666 $DIR/f2" || return 2 } run_test 2 "chmod: drop req, drop rep" test_3() { drop_request "statone $DIR/f2" || return 1 drop_reply "statone $DIR/f2" || return 2 } run_test 3 "stat: drop req, drop rep" These three tests are actually steps of a single test scenario, because they work with the results of previous ones. We can separate these tests: test_1() { test_1_mcreate } run_test 1 "mcreate: drop req, drop rep" test_2() { test_1_mcreate test_2_chmod } run_test 2 "chmod: drop req, drop rep" test_3() { test_1_mcreate test_3_stat } run_test 3 "stat: drop req, drop rep" or join them into one and remove test_2 and test_3. test_1() { drop_request "mcreate $DIR/f1" || return 1 drop_reint_reply "mcreate $DIR/f2" || return 2 drop_request "tchmod 111 $DIR/f2" || return 3 drop_reint_reply "tchmod 666 $DIR/f2" || return 4 drop_request "statone $DIR/f2" || return 5 drop_reply "statone $DIR/f2" || return 6 } run_test 1 "mcreate, chmod,stat: drop req, drop,req" Another big example are sanity tests 200 and 201. Here is the part of the resulting code after refactoring, so we can separately run each letter index: test_200a() { test_200_create_pool test_201_remove_pool } run_test 200a "Create new pool ==========================================" test_200b() { test_200_create_pool test_200_add_targets test_201_remove_all_targets test_201_remove_pool } . . . test_201b() { test_200_create_pool test_200_add_targets test_200_dir_set_pool test_200_check_dir_pool test_200_check_file_alloc test_200_create_files test_200_create_relative_path_files test_201_remove_all_targets test_201_remove_pool } run_test 201b "Remove all targets from a pool ==========================" test_201c() { test_200_create_pool test_201_remove_pool } run_test 201c "Remove a pool ============================================" We have to include cleanup steps here to make possible to run letter indexes independently. With that cleanup steps, 200a and 201c became absolutely equal and need to be reduced. Same situation is for 200h and 201b. Sorry for so long email and thanks to Kyr (Kyrylo_Shatskyy at xyratex.com) for it's preparing. -- Alexander Lezhoev. Morpheus test team. Xyratex. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at whamcloud.com Tue May 15 16:02:14 2012 From: chris at whamcloud.com (Chris) Date: Tue, 15 May 2012 17:02:14 +0100 Subject: [Twg] twg Digest, Vol 19, Issue 11 In-Reply-To: References: Message-ID: <4FB27E06.3090807@whamcloud.com> > Hi there, > > Let me raise the question about Lustre tests separated execution. We've > discussed this problem already, but I'd like to clear up some details. > > Usually we run all tests sequentially, but in the automation tool we are > using we need to run tests separately, with ONLY parameter. This allows > us to have full control over the test execution: terminate hung tests by > timeout or restore environment in case of the file system damage. At the > moment some of tests are designed to be run sequentially. > > By our estimation, there are about 30 tests need for the improvement. > If we settle this question, we can significantly improve test-framework > automation potential. > Please share your opinions about this question and help to make a > decision about it. > The questions are > > 1. Do we want to have an ability to run each test independently? At first I thought this was a good target but actually after some thought I think quite the opposite is true. Having each discrete step a separate test is very useful, for example. test 1: create a file test 2: rename the file test 3: chmod the file test 4: test file can't be deleted test 5: copy the file to file2 test 6: compare file to file2 test 7: chmod the file and file 2 test 8: test file and file 2 can be deleted These tests are linked and it is useful to know which failed, but you can't start anywhere other than test1 and proceed to test 8. Having specific setup would be artificial, and having a requirement that they could not be link would cause people to group discrete tests into bigger tests. > 2. What is more acceptable - unite sequential tests into complex ones > or supplement exists test with additional code steps? > I release don't think we should make tests being atomic a requirement for the above, and because it will lead to test becoming more complex, or the rather strange situation where test 2 (above) calls test 1 silently if it was not already run, (test 3 calls test 2....) which I think is what you are suggesting in your example. I think that both the solutions in your mail add complexity and remove readability from the test environment and when a test fails you do not know what actually failed in a systematic way. I don't really think that merging tests together or having each test silently do all the functionally of the previous tests is adding value. A piece of information that might help us come up with the right solution would be the driving force behind being able to run individual tests? Why do you want to be able to do it. Chris From bruce_korb at xyratex.com Tue May 15 16:16:11 2012 From: bruce_korb at xyratex.com (Bruce Korb) Date: Tue, 15 May 2012 09:16:11 -0700 Subject: [Twg] stand alone testing In-Reply-To: <4FB27E06.3090807@whamcloud.com> Message-ID: On 5/15/12 9:02 AM, Chris wrote: >> 1. Do we want to have an ability to run each test independently? Yes. >At first I thought this was a good target but actually after some >thought I think quite the opposite is true. > >Having each discrete step a separate test is very useful, for example. > >test 1: create a file >test 2: rename the file >test 3: chmod the file >test 4: test file can't be deleted >test 5: copy the file to file2 >test 6: compare file to file2 >test 7: chmod the file and file 2 >test 8: test file and file 2 can be deleted > >These tests are linked Alternatively, these tests are one. A test to verify "chmod" on a renamed file would be in several phases: phase 1) create a file || die cannot create phase 2) rename it || die cannot rename phase 3) chmod it || die cannot chmod cleanup) remove file || report cleanup failure This does not prevent any of these phases from being in a library for use by any test, but it would mean that a given test cannot presuppose that some other test had been run before it. In order to reduce redundant testing, as an example you would not want to regularly run the "create a file" stand alone test, given that few other tests would go far if that feature were broken. But there is too much confusion if every separately identifiable test does not stand on its own. From nic at cray.com Tue May 15 16:15:13 2012 From: nic at cray.com (Nicholas Henke) Date: Tue, 15 May 2012 11:15:13 -0500 Subject: [Twg] Separated test execution In-Reply-To: <4FB17224.1050105@xyratex.com> References: <4FB17224.1050105@xyratex.com> Message-ID: <4FB28111.8070602@cray.com> On 05/14/2012 03:59 PM, Alexander Lezhoev wrote: > Hi there, > ..snipped... > > They cannot be run separately, because the next index uses the code of > previous one. This means all tests should be run in groups of letter > indexes, or they should be refactored to run independently. Is there really a hard 'style' that has the letter indices meaning they depend on previous tests ? It seem to vary by test, including where a test 6d is just the 'Dth' style of running test6. > > Some of tests have been already refactored to run "letters" > separately, but we have to make a rule which we should follow and use > for further refactoring. > There are three decisions we can take about this situation > > * Join the code of all test steps into single test with > corresponding number. So we will have one test_1 instead of > test_1a .. test_1d in the described case. > It would certainly be much cleaner to have a single test contain all of the steps it needs. It is then repeatable and includes all of the setup and cleanup required to complete the test. > > > The first variant could be implemented as follows. > > test_1() { > > touch $DIR1/f1 > > [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed" > > chmod 777 $DIR2/f1 > > $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates > on 2 mtpt's failed" > > chmod a-x $DIR2/f1 > > $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount > attribute updates on 2 mtpt's failed" > > rm $DIR2/f1 > > $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file > on other failed" > > } > run_test 1 "check attributes updates on 2 mtpt's" > > This approach has disadvantage that such kind of refactoring will lead > to reduction of test numbering and it will hard to work with > regression history of the refactored tests. > The second case of refactoring can look like this: > I'm not sure this is a huge negative for me. The test is very clear and it gives us precise errors on each step -- something we should make sure is necessary. As we rework these tests, we'll be changing them enough that we'll effectively wipe some of the regression history. Perhaps it is enough to change the regression history to just look at test numbering, ignoring letters ? ...snipped... > We can separate these tests: > > test_1() { > test_1_mcreate > } > run_test 1 "mcreate: drop req, drop rep" > > test_2() { > test_1_mcreate > test_2_chmod > } > run_test 2 "chmod: drop req, drop rep" > > test_3() { > test_1_mcreate > test_3_stat > } > run_test 3 "stat: drop req, drop rep" I think this makes it much more complex and harder to code, not to mention adds overhead. Each instance of 'run_test' invokes a host of setup & cleanup code that can add significantly to the test runtime. > > or join them into one and remove test_2 and test_3. > > test_1() { > drop_request "mcreate $DIR/f1" || return 1 > drop_reint_reply "mcreate $DIR/f2" || return 2 > drop_request "tchmod 111 $DIR/f2" || return 3 > drop_reint_reply "tchmod 666 $DIR/f2" || return 4 > drop_request "statone $DIR/f2" || return 5 > drop_reply "statone $DIR/f2" || return 6 > } > run_test 1 "mcreate, chmod,stat: drop req, drop,req" > The return codes are not typically enough to distinguish errors from a test log. The style above with 'error "MSG with values $x $y"' seems more rich and helpful. Cheers, Nic -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at whamcloud.com Tue May 15 17:02:17 2012 From: chris at whamcloud.com (Chris) Date: Tue, 15 May 2012 18:02:17 +0100 Subject: [Twg] stand alone testing In-Reply-To: References: Message-ID: <4FB28C19.30706@whamcloud.com> On 15/05/2012 17:16, Bruce Korb wrote: > On 5/15/12 9:02 AM, Chris wrote: >>> 1. Do we want to have an ability to run each test independently? > Yes. > >> At first I thought this was a good target but actually after some >> thought I think quite the opposite is true. >> >> Having each discrete step a separate test is very useful, for example. >> >> test 1: create a file >> test 2: rename the file >> test 3: chmod the file >> test 4: test file can't be deleted >> test 5: copy the file to file2 >> test 6: compare file to file2 >> test 7: chmod the file and file 2 >> test 8: test file and file 2 can be deleted >> >> These tests are linked > Alternatively, these tests are one. A test to verify "chmod" > on a renamed file would be in several phases: > phase 1) create a file || die cannot create > phase 2) rename it || die cannot rename > phase 3) chmod it || die cannot chmod > cleanup) remove file || report cleanup failure The problem with this is that you need a machine digestable method of reporting back the failure. This test failing can be because of 4 things, so 4 failures might mean 4 different things. With the model above you would know exactly what had failed, and can run earlier tests separately. > This does not prevent any of these phases from being in a library > for use by any test, but it would mean that a given test cannot > presuppose that some other test had been run before it. One problem with this approach and looking at the history of the current bash code is that it will lead to duplication via copy and paste because people won't write libraries, bash does not lend itself to this and so solutions will be messy. > > In order to reduce redundant testing, as an example you would not > want to regularly run the "create a file" stand alone test, given > that few other tests would go far if that feature were broken. > But there is too much confusion if every separately identifiable > test does not stand on its own. Does this not mean that instead of requiring to know about dependency we now require an overlap list so that we don't duplicate the work. We could expand framework to never call a test twice (this would be pretty easy) then we could have. test_1 { create a file } run_test 1 test_2 { run_test 1 (only does something if test_1 has not already been called) rename the file } run_test 2 test_3 { run_test 2 (only does something if test_2 has not already been called) chmod the file } run_test 3 This way no duplication of code, no duplication of execution and very easy to read and Chris From adilger at whamcloud.com Tue May 15 17:41:43 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 15 May 2012 11:41:43 -0600 Subject: [Twg] Separated test execution In-Reply-To: <4FB17224.1050105@xyratex.com> References: <4FB17224.1050105@xyratex.com> Message-ID: <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> On 2012-05-14, at 2:59 PM, Alexander Lezhoev wrote: > By our estimation, there are about 30 tests need for the improvement. If we settle this question, we can significantly improve test-framework automation potential. > Please share your opinions about this question and help to make a decision about it. > The questions are > • Do we want to have an ability to run each test independently? > • What is more acceptable - unite sequential tests into complex ones or supplement exists test with additional code steps? Thanks for looking at this. I agree that it makes sense to be able to run subtests independently, and this has been the accepted policy for some time. While there are some subtests that may not be in this state today, that there are only about 30 such subtests indicates that there isn't a huge problem that needs a complex solution to resolve. I think the method to fix each subtest depends on the particular subtest under discussion. If the subtest itself is already complex, and just has a common setup step, then they should be kept separate. In the case shown below, these subtests were created back when Lustre could sometimes fail to run simple operations like cross-node create/unlink, but it probably makes sense to combine these subtests into a single subtest as in your example #1. Since there is already typically an increasing amount of overhead to run each subtest, having a single "mkdir" for a subtest probably does not make sense anymore. > Typical problem is sanityn test_1 > > test_1a() { > touch $DIR1/f1 > [ -f $DIR2/f1 ] || error > } > run_test 1a "check create on 2 mtpt's ==========================" > > test_1b() { > chmod 777 $DIR2/f1 > $CHECKSTAT -t file -p 0777 $DIR1/f1 || error > chmod a-x $DIR2/f1 > } > run_test 1b "check attribute updates on 2 mtpt's ===============" > > test_1c() { > $CHECKSTAT -t file -p 0666 $DIR1/f1 || error > } > run_test 1c "check after remount attribute updates on 2 mtpt's =" > > test_1d() { > rm $DIR2/f1 > $CHECKSTAT -a $DIR1/f1 || error > } > run_test 1d "unlink on one mountpoint removes file on other ====" > > > They cannot be run separately, because the next index uses the code of previous one. This means all tests should be run in groups of letter indexes, or they should be refactored to run independently. > > Some of tests have been already refactored to run “letters” separately, but we have to make a rule which we should follow and use for further refactoring. > There are three decisions we can take about this situation > • Join the code of all test steps into single test with corresponding number. So we will have one test_1 instead of test_1a .. test_1d in the described case. > • Move the code of steps to corresponding functions which will be called from each step. In other words the next indexed test will duplicate some functionality of previous one. > • Do nothing and decide that “letters” mustn’t be executed independently, but only in “number” group. > > The first variant could be implemented as follows. > > test_1() { > touch $DIR1/f1 > [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed" > chmod 777 $DIR2/f1 > $CHECKSTAT -t file -p 0777 $DIR1/f1 || > error "check attribute updates on 2 mtpt's failed" > chmod a-x $DIR2/f1 > $CHECKSTAT -t file -p 0666 $DIR1/f1 || > error "after remount attributes on 2 mtpt's failed" > rm $DIR2/f1 > $CHECKSTAT -a $DIR1/f1 || > error "unlink on one mtpt removes file on other failed" > } > run_test 1 "check attributes updates on 2 mtpt's" Note I've reformatted the above example to follow the proper coding style (80-column lines). Also note that in this case, it appears there originally was supposed to be an unmount/remount between "chmod a-x" and the following "$CHECKSTAT -p 0666", but that was lost over time. > This approach has disadvantage that such kind of refactoring will lead to reduction of test numbering and it will hard to work with regression history of the refactored tests. I doubt there are (m)any cases where these subtests have failed, so I don't think there is any serious loss in testing history. Also, we shouldn't be prevented from improving some of the testing code just because it will be a break from a few of the old subtest results. While testing continuity is important (e.g. I wouldn't advocate renumbering subtests just to give them "nicer" numbers), if this can improve some of the subtests that are causing problems there is a clear benefit to do so. > I’ve omitted functions code - their content is obvious. > This disadvantage of this approach — summary increase of tests run-time (the next test duplicates code of previous one). But the necessity of all these tests is doubtful here, because the last one includes first three tests. Right. While it is true that virtually all later subtests will already do a "mkdir" internally, there is still a benefit to knowing early that the "mkdir" itself is working properly before spending time trying to debug a complex subtest only to find some simple operation caused the failure. > Very similar situation is for recovery-small 1, 2, 3 and 4, 5. > > test_1() { > drop_request "mcreate $DIR/f1" || return 1 > drop_reint_reply "mcreate $DIR/f2" || return 2 > } > run_test 1 "mcreate: drop req, drop rep" > > test_2() { > drop_request "tchmod 111 $DIR/f2" || return 1 > drop_reint_reply "tchmod 666 $DIR/f2" || return 2 > } > run_test 2 "chmod: drop req, drop rep" > > test_3() { > drop_request "statone $DIR/f2" || return 1 > drop_reply "statone $DIR/f2" || return 2 > } > run_test 3 "stat: drop req, drop rep" > > These three tests are actually steps of a single test scenario, because they work with the results of previous ones. > > or join them into one and remove test_2 and test_3. > > test_1() { > drop_request "mcreate $DIR/f1" || return 1 > drop_reint_reply "mcreate $DIR/f2" || return 2 > drop_request "tchmod 111 $DIR/f2" || return 3 > drop_reint_reply "tchmod 666 $DIR/f2" || return 4 > drop_request "statone $DIR/f2" || return 5 > drop_reply "statone $DIR/f2" || return 6 > } > run_test 1 "mcreate, chmod,stat: drop req, drop,req" As someone else commented, I really dislike the use of error codes to signal failures. The "1", "2", ... return codes have no real meaning, and seeing a result like "recovery small test_1 failed with 3" is useless compared to "recovery_small test_1 failed when resending chmod request". If you are doing any significant subtest restructuring, please use "error" instead of "return". > Another big example are sanity tests 200 and 201. Here is the part of the resulting code after refactoring, so we can separately run each letter index: > > test_200a() { > test_200_create_pool > test_201_remove_pool > } > run_test 200a "Create new pool ==========================================" If restructuring subtests, please also drop the trailing "===" markers. These are handled by the test-framework now, and just make the output ugly. > test_200b() { > test_200_create_pool > test_200_add_targets > test_201_remove_all_targets > test_201_remove_pool > } > > . . . > > test_201b() { > test_200_create_pool > test_200_add_targets > test_200_dir_set_pool > test_200_check_dir_pool > test_200_check_file_alloc > test_200_create_files > test_200_create_relative_path_files > test_201_remove_all_targets > test_201_remove_pool > } > run_test 201b "Remove all targets from a pool ==========================" > > test_201c() { > test_200_create_pool > test_201_remove_pool > } > run_test 201c "Remove a pool ============================================" > > > We have to include cleanup steps here to make possible to run letter indexes independently. With that cleanup steps, 200a and 201c became absolutely equal and need to be reduced. Same situation is for 200h and 201b. The problem with doing a full setup and cleanup for each subtest is that this can make the test runtime longer than necessary. In previous cases where there was a dependency between subtests, it is better to have the "setup" step do a "check or setup", and if the previous subtest has already run, there is no additional work done for the setup. Similarly, if every subtest does its own cleanup, but the next test sets everything back up again, it is just a waste of testing time. I also don't think each subtest should do its own cleanup, unless the subtest itself could cause problems with other subtests because it significantly fills the OST(s) or the MDT(s). The cleanup for "normal" tests should be deferred until after the end of all subtests (e.g. the "rm -rf $DIR/[df][0-9]*" for sanity.sh and sanityn.sh). The pool tests themselves already have a "cleanup_pools" test that is run after all of the pools tests are completed. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Roman_Grigoryev at xyratex.com Tue May 15 18:31:54 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Tue, 15 May 2012 22:31:54 +0400 Subject: [Twg] stand alone testing In-Reply-To: <4FB28C19.30706@whamcloud.com> References: <4FB28C19.30706@whamcloud.com> Message-ID: <4FB2A11A.9040809@xyratex.com> On 05/15/2012 09:02 PM, Chris wrote: > On 15/05/2012 17:16, Bruce Korb wrote: >> On 5/15/12 9:02 AM, Chris wrote: >>>> 1. Do we want to have an ability to run each test independently? >> Yes. I think, it should very important for developers and for tests failure evaluators, because they often run limited set of tests. >> >>> At first I thought this was a good target but actually after some >>> thought I think quite the opposite is true. >>> >>> Having each discrete step a separate test is very useful, for example. >>> >>> test 1: create a file >>> test 2: rename the file >>> test 3: chmod the file >>> test 4: test file can't be deleted >>> test 5: copy the file to file2 >>> test 6: compare file to file2 >>> test 7: chmod the file and file 2 >>> test 8: test file and file 2 can be deleted >>> >>> These tests are linked >> Alternatively, these tests are one. A test to verify "chmod" >> on a renamed file would be in several phases: >> phase 1) create a file || die cannot create >> phase 2) rename it || die cannot rename >> phase 3) chmod it || die cannot chmod >> cleanup) remove file || report cleanup failure > The problem with this is that you need a machine digestable method of > reporting back the failure. This test failing can be because of 4 > things, so 4 failures might mean 4 different things. With the model > above you would know exactly what had failed, and can run earlier tests > separately. Just debug messages should be enough, i think. For example, if I asked framework to execute testsC (which failed) and framework would start executing testA, it is not straight behavior. If we use your logic, evaluator must understand that steps are dependent and see messages from all previous steps(because error may be happens on previous step and only indicated on current step). In general, you are suggest to use one big tests instead of set of tests, which could be only evaluated as one entity. Every separated part stop be a test and became other entity, .f.e subtest, test step, test case. > >> This does not prevent any of these phases from being in a library >> for use by any test, but it would mean that a given test cannot >> presuppose that some other test had been run before it. > One problem with this approach and looking at the history of the current > bash code is that it will lead to duplication via copy and paste because > people won't write libraries, bash does not lend itself to this and so > solutions will be messy. Duplicating in tests is not so big problem, i think. As we discussed previously, this is disciple. Thanks, Roman > >> >> In order to reduce redundant testing, as an example you would not >> want to regularly run the "create a file" stand alone test, given >> that few other tests would go far if that feature were broken. >> But there is too much confusion if every separately identifiable >> test does not stand on its own. > Does this not mean that instead of requiring to know about dependency we > now require an overlap list so that we don't duplicate the work. > > We could expand framework to never call a test twice (this would be > pretty easy) then we could have. > > test_1 { > create a file > } > run_test 1 > > test_2 { > run_test 1 (only does something if test_1 has not already been called) > rename the file > } > run_test 2 > > > test_3 { > run_test 2 (only does something if test_2 has not already been called) > chmod the file > } > run_test 3 > > This way no duplication of code, no duplication of execution and very > easy to read and > > > Chris > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From bruce_korb at xyratex.com Tue May 15 18:39:46 2012 From: bruce_korb at xyratex.com (Bruce Korb) Date: Tue, 15 May 2012 11:39:46 -0700 Subject: [Twg] stand alone testing In-Reply-To: Message-ID: Hi Chris, On 5/15/12 10:02 AM, Chris wrote: >> This does not prevent any of these phases from being in a library >> for use by any test, but it would mean that a given test cannot >> presuppose that some other test had been run before it. >One problem with this approach and looking at the history of the current >bash code is that it will lead to duplication via copy and paste because >people won't write libraries, bash does not lend itself to this and so >solutions will be messy. Bash *does* lend itself to libraries, but it is not a well understood technique. Shell scripts, by and large, do tend to be messy, but that is really much more a matter of education and technique than it is of inherent capabilities. Due to scripting misuse, using another tool less prone to messy source seems like a reasonable approach. But the issue is not that of inadequacies of bash. From alexey_lyashkov at xyratex.com Tue May 15 18:53:34 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Tue, 15 May 2012 22:53:34 +0400 Subject: [Twg] Separated test execution In-Reply-To: <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> References: <4FB17224.1050105@xyratex.com> <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> Message-ID: <18DE7FDC-E34B-4538-8530-34A1345E73C6@xyratex.com> On May 15, 2012, at 21:41, Andreas Dilger wrote: > > >> Very similar situation is for recovery-small 1, 2, 3 and 4, 5. >> >> test_1() { >> drop_request "mcreate $DIR/f1" || return 1 >> drop_reint_reply "mcreate $DIR/f2" || return 2 >> } >> run_test 1 "mcreate: drop req, drop rep" >> >> test_2() { >> drop_request "tchmod 111 $DIR/f2" || return 1 >> drop_reint_reply "tchmod 666 $DIR/f2" || return 2 >> } >> run_test 2 "chmod: drop req, drop rep" >> >> test_3() { >> drop_request "statone $DIR/f2" || return 1 >> drop_reply "statone $DIR/f2" || return 2 >> } >> run_test 3 "stat: drop req, drop rep" >> >> These three tests are actually steps of a single test scenario, because they work with the results of previous ones. >> >> or join them into one and remove test_2 and test_3. >> >> test_1() { >> drop_request "mcreate $DIR/f1" || return 1 >> drop_reint_reply "mcreate $DIR/f2" || return 2 >> drop_request "tchmod 111 $DIR/f2" || return 3 >> drop_reint_reply "tchmod 666 $DIR/f2" || return 4 >> drop_request "statone $DIR/f2" || return 5 >> drop_reply "statone $DIR/f2" || return 6 >> } >> run_test 1 "mcreate, chmod,stat: drop req, drop,req" > > As someone else commented, I really dislike the use of error codes to signal failures. The "1", "2", ... return codes have no real meaning, and seeing a result like "recovery small test_1 failed with 3" is useless compared to "recovery_small test_1 failed when resending chmod request". > > If you are doing any significant subtest restructuring, please use "error" instead of "return". Andreas, using a string is hard to parse and find a place to fail a test, but if we use a return codes we a easy map a error to place where it hit. I like to combine a two approaches. error should be set a return code and print a message. message usefull to a user, but error code user full for automatic execution. -------------------------------------------- Alexey Lyashkov alexey_lyashkov at xyratex.com From Roman_Grigoryev at xyratex.com Tue May 15 19:00:46 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Tue, 15 May 2012 23:00:46 +0400 Subject: [Twg] Separated test execution In-Reply-To: <4FB28111.8070602@cray.com> References: <4FB17224.1050105@xyratex.com> <4FB28111.8070602@cray.com> Message-ID: <4FB2A7DE.9090100@xyratex.com> On 05/15/2012 08:15 PM, Nicholas Henke wrote: > On 05/14/2012 03:59 PM, Alexander Lezhoev wrote: >> Hi there, >> > > ..snipped... >> >> They cannot be run separately, because the next index uses the code of >> previous one. This means all tests should be run in groups of letter >> indexes, or they should be refactored to run independently. > > Is there really a hard 'style' that has the letter indices meaning they > depend on previous tests ? It seem to vary by test, including where a > test 6d is just the 'Dth' style of running test6. No any definition haven't placed in documentation about it till now. Framework don't do different processing. Every tests creator do it by his own understanding. >> >> Some of tests have been already refactored to run “letters” >> separately, but we have to make a rule which we should follow and use >> for further refactoring. >> There are three decisions we can take about this situation >> >> * Join the code of all test steps into single test with >> corresponding number. So we will have one test_1 instead of >> test_1a .. test_1d in the described case. >> > > It would certainly be much cleaner to have a single test contain all of > the steps it needs. It is then repeatable and includes all of the setup > and cleanup required to complete the test. Absolutely agree. I think, we should not afraid existence of some percent of copy-paste code in tests it they became cleaner. >> >> >> The first variant could be implemented as follows. >> >> test_1() { >> >> touch $DIR1/f1 >> >> [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed" >> >> chmod 777 $DIR2/f1 >> >> $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates >> on 2 mtpt's failed" >> >> chmod a-x $DIR2/f1 >> >> $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount >> attribute updates on 2 mtpt's failed" >> >> rm $DIR2/f1 >> >> $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file >> on other failed" >> >> } >> run_test 1 "check attributes updates on 2 mtpt's" >> >> This approach has disadvantage that such kind of refactoring will lead >> to reduction of test numbering and it will hard to work with >> regression history of the refactored tests. >> The second case of refactoring can look like this: >> > I'm not sure this is a huge negative for me. The test is very clear and > it gives us precise errors on each step -- something we should make sure > is necessary. > > As we rework these tests, we'll be changing them enough that we'll > effectively wipe some of the regression history. Perhaps it is enough to > change the regression history to just look at test numbering, ignoring > letters ? Other possible approach: start to use test id. Now they could be same as test name and can migrate with test core, include case of merging tests and merged test could have field which have original tests ids. Thanks, Roman > > > ...snipped... >> We can separate these tests: >> >> test_1() { >> test_1_mcreate >> } >> run_test 1 "mcreate: drop req, drop rep" >> >> test_2() { >> test_1_mcreate >> test_2_chmod >> } >> run_test 2 "chmod: drop req, drop rep" >> >> test_3() { >> test_1_mcreate >> test_3_stat >> } >> run_test 3 "stat: drop req, drop rep" > > I think this makes it much more complex and harder to code, not to > mention adds overhead. Each instance of 'run_test' invokes a host of > setup & cleanup code that can add significantly to the test runtime. > >> >> or join them into one and remove test_2 and test_3. >> >> test_1() { >> drop_request "mcreate $DIR/f1" || return 1 >> drop_reint_reply "mcreate $DIR/f2" || return 2 >> drop_request "tchmod 111 $DIR/f2" || return 3 >> drop_reint_reply "tchmod 666 $DIR/f2" || return 4 >> drop_request "statone $DIR/f2" || return 5 >> drop_reply "statone $DIR/f2" || return 6 >> } >> run_test 1 "mcreate, chmod,stat: drop req, drop,req" >> > > The return codes are not typically enough to distinguish errors from a > test log. The style above with 'error "MSG with values $x $y"' seems > more rich and helpful. > > Cheers, > Nic > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From Nathan_Rutman at xyratex.com Tue May 15 23:30:40 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 15 May 2012 16:30:40 -0700 Subject: [Twg] Separated test execution In-Reply-To: <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> References: <4FB17224.1050105@xyratex.com> <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10A63@XYUS-EX22.xyus.xyratex.com> On May 15, 2012, at 10:41 AM, Andreas Dilger wrote: We have to include cleanup steps here to make possible to run letter indexes independently. With that cleanup steps, 200a and 201c became absolutely equal and need to be reduced. Same situation is for 200h and 201b. The problem with doing a full setup and cleanup for each subtest is that this can make the test runtime longer than necessary. In previous cases where there was a dependency between subtests, it is better to have the "setup" step do a "check or setup", and if the previous subtest has already run, there is no additional work done for the setup. Similarly, if every subtest does its own cleanup, but the next test sets everything back up again, it is just a waste of testing time. Another big problem is that these two states are not equal. After remounting Lustre does not have the same state (memory structures, cache, variable settings) that it has after running a test. If we only ever run tests after remounting, how do we know Lustre can successfully run more than one operation in a row? I also don't think each subtest should do its own cleanup, unless the subtest itself could cause problems with other subtests because it significantly fills the OST(s) or the MDT(s). The cleanup for "normal" tests should be deferred until after the end of all subtests (e.g. the "rm -rf $DIR/[df][0-9]*" for sanity.sh and sanityn.sh). The pool tests themselves already have a "cleanup_pools" test that is run after all of the pools tests are completed. I think tests should clean up after themselves, if they leave any persistent fingerprints -- changes in debug level, parameter settings, fills space > eg 10%, etc. That way, tests can be run individually, in a random order, repeatedly, etc. (BTW, I don't think there's a way to run a test more than once in a row with current test-framework.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roman_Grigoryev at xyratex.com Wed May 16 03:59:26 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Wed, 16 May 2012 07:59:26 +0400 Subject: [Twg] Separated test execution In-Reply-To: <73AED5C780AE05478241DB067651A92102F10A63@XYUS-EX22.xyus.xyratex.com> References: <4FB17224.1050105@xyratex.com> <0D05A3CA-A0CD-4BF8-8068-957A0AAC1853@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10A63@XYUS-EX22.xyus.xyratex.com> Message-ID: <4FB3261E.4090206@xyratex.com> On 05/16/2012 03:30 AM, Nathan Rutman wrote: > > On May 15, 2012, at 10:41 AM, Andreas Dilger wrote: > >>> >>> >>> We have to include cleanup steps here to make possible to run letter >>> indexes independently. With that cleanup steps, 200a and 201c became >>> absolutely equal and need to be reduced. Same situation is for 200h >>> and 201b. >> >> The problem with doing a full setup and cleanup for each subtest is >> that this can make the test runtime longer than necessary. In previous >> cases where there was a dependency between subtests, it is better to >> have the "setup" step do a "check or setup", and if the previous >> subtest has already run, there is no additional work done for the >> setup. Similarly, if every subtest does its own cleanup, but the next >> test sets everything back up again, it is just a waste of testing time. > > Another big problem is that these two states are not equal. After > remounting Lustre does not have the same state (memory structures, > cache, variable settings) that it has after running a test. If we only > ever run tests after remounting, how do we know Lustre can successfully > run more than one operation in a row? > >> >> I also don't think each subtest should do its own cleanup, unless the >> subtest itself could cause problems with other subtests because it >> significantly fills the OST(s) or the MDT(s). The cleanup for >> "normal" tests should be deferred until after the end of all subtests >> (e.g. the "rm -rf $DIR/[df][0-9]*" for sanity.sh and sanityn.sh). The >> pool tests themselves already have a "cleanup_pools" test that is run >> after all of the pools tests are completed. > > I think tests should clean up after themselves, if they leave any > persistent fingerprints -- changes in debug level, parameter settings, > fills space > eg 10%, etc. That way, tests can be run individually, in > a random order, repeatedly, etc. (BTW, I don't think there's a way to > run a test more than once in a row with current test-framework.) This feature is very simple to implement via using ONLY parameter and external script with knowledge of test names. Thanks, Roman From chris at whamcloud.com Wed May 16 12:18:21 2012 From: chris at whamcloud.com (Chris) Date: Wed, 16 May 2012 13:18:21 +0100 Subject: [Twg] stand alone testing In-Reply-To: <4FB2A11A.9040809@xyratex.com> References: <4FB28C19.30706@whamcloud.com> <4FB2A11A.9040809@xyratex.com> Message-ID: <4FB39B0D.7040704@whamcloud.com> On 15/05/2012 19:31, Roman Grigoryev wrote: > On 05/15/2012 09:02 PM, Chris wrote: >> On 15/05/2012 17:16, Bruce Korb wrote: >>> On 5/15/12 9:02 AM, Chris wrote: >>>>> 1. Do we want to have an ability to run each test independently? >>> Yes. > I think, it should very important for developers and for tests failure > evaluators, because they often run limited set of tests. I absolutely agree that a developer needs to be able to specify just the test that he or she wants, but what is the disadvantage if the test framework / system executes the required prerequisites before hand. The whole test cycle takes no longer (the prerequisite steps have to occur one way or the other), the results provide better granularity in the event of a prerequisite failing and the tests scripts are almost certainly simpler and easy to read. I would also suggest that looking at the history of the scripts people won't use libraries to ensure that setup / clear-up code only executes once what will happen is that setup/clearup will get executed for every test and so execution of groups of tests will take longer. My example below illustrates how we could simple provide pre-requisite test support using what is effectively simple inheritance, very simply and very readably and in a way that makes the desired outcome the likely outcome. This approach means that when a change is required it only needs to be made once and works well within the standards that we've actually used to right test scripts for the last 10 years not the standards we wish we had used. What is the disadvantage of this simple explicit (light inheritance) example below, expanding the framework to never call a test twice would be a no-brainer. test_1a { create a file } run_test 1a test_1b { run_test 1a (only does something if test_1a has not already been called) rename the file } run_test 1b test_1c { run_test 1b (only does something if test_1b has not already been called) chmod the file } run_test 1c There is no duplication of code and duplicating code would be harder work than not duplicating it, no duplication of execution and it is very easy to read. We could get the framework to do it automatically, i.e. if test_1c was called without 1b having been called it could call 1b... but that would make the code less clear to the reader. > >>> This does not prevent any of these phases from being in a library >>> for use by any test, but it would mean that a given test cannot >>> presuppose that some other test had been run before it. >> One problem with this approach and looking at the history of the current >> bash code is that it will lead to duplication via copy and paste because >> people won't write libraries, bash does not lend itself to this and so >> solutions will be messy. > Duplicating in tests is not so big problem, i think. As we discussed > previously, this is disciple. I am not sure if you are saying 'it is not a problem if tests are duplicated' in which case I will absolutely say the duplication of code is always a problem and always leads to unmaintainable systems - a bit like our test scripts really. But maybe you are saying that 'it is not a problem because all we need is discipline', which I would agree with but we have historically not had that discipline in the test scripts and I'm not going to bet on that changing any-time soon. So I would rather use methods that strongly encourage that right thing to happen, for example an approach to pre-requisites that makes the easiest method the right method, encourages reuse and discourages duplication etc. Regards to all Chris From chris at whamcloud.com Wed May 16 12:18:35 2012 From: chris at whamcloud.com (Chris) Date: Wed, 16 May 2012 13:18:35 +0100 Subject: [Twg] stand alone testing In-Reply-To: References: Message-ID: <4FB39B1B.1070204@whamcloud.com> On 15/05/2012 19:39, Bruce Korb wrote: > Hi Chris, > > On 5/15/12 10:02 AM, Chris wrote: >>> This does not prevent any of these phases from being in a library >>> for use by any test, but it would mean that a given test cannot >>> presuppose that some other test had been run before it. >> One problem with this approach and looking at the history of the current >> bash code is that it will lead to duplication via copy and paste because >> people won't write libraries, bash does not lend itself to this and so >> solutions will be messy. > Bash *does* lend itself to libraries, but it is not a well understood > technique. Shell scripts, by and large, do tend to be messy, but that > is really much more a matter of education and technique than it is > of inherent capabilities. > > Due to scripting misuse, using another tool less prone to messy source > seems like a reasonable approach. But the issue is not that of > inadequacies > of bash. > > I absolutely agree Bruce and what we should learn from the last 10 years of Lustre and by looking at 37000 lines of bash code is that people don't do the right big picture thing when it comes to test scripts and so I want us to adopt approaches that tend to cause the right thing to happen. If we declare that all test scripts and modifications will now be written in a structured, clear, well documented manner where duplicate functionality is pulled out into libraries, variables are never duplicate, naming is consistent... then I would have less concerns and we would have more degrees of freedom. But can't actually do that and so methods that encourage reuse and discourage code replication are a must. I also absolutely think we need to begin the work of creating a whole new methodology (not to replace the bash but to provide for new tests) that provides another tool less prone to messy source and structure. Getting this right will be difficult because we need to provide for all the freedom that developers require whilst constraining them into something that over the next 10 years leads to an ordered outcome. Chris From Denis_Kondratenko at xyratex.com Wed May 16 13:37:22 2012 From: Denis_Kondratenko at xyratex.com (Denis Kondratenko) Date: Wed, 16 May 2012 14:37:22 +0100 Subject: [Twg] stand alone testing References: <4FB39B1B.1070204@whamcloud.com> Message-ID: <74A7CBF22EB3FB4AB198F266ED2D2B210CACCC@XY01EX21.xy01.xyratex.com> An HTML attachment was scrubbed... URL: From Roman_Grigoryev at xyratex.com Wed May 16 13:57:24 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Wed, 16 May 2012 17:57:24 +0400 Subject: [Twg] [more info] Separated test execution In-Reply-To: <4FB17224.1050105@xyratex.com> References: <4FB17224.1050105@xyratex.com> Message-ID: <4FB3B244.7080309@xyratex.com> Hi all, I did (wit Alexanders help) 2 test executions on our latest lustre build with limited set of test suites. * First tests execution*: executing all tests with with ONLY keyword one-by-one. *Second execution*: executing all tests with with ONLY keyword one-by-one and reformat lustre partition (/usr/lib64/lustre/tests/llmountcleanup.sh and FORMAT=yes sh /usr/lib64/lustre/tests/llmount.sh). With these executions ways we should detect all tests dependencies (exclude false pass, but this it other problem). I prepared table with results for both executions and differences between them. Crossed tests - tests which are in ALWAYS_EXLUDED list. So, it is 15 test which was failed with ONLY/ ONLY+REFORMAT. Test replay-single.44a.test is killed by timeout, and can be excluded from this list. Tests sanity-quota.18b and sanity.129 failed because end of space(and looks like formatting fix it). So, 12 test have dependencies and should be fixed. I think this is good news. suite with reformat without reformat diff sanity sanity.200h.test sanity.200c.test sanity.201b.test sanity.200d.test sanity.51c.test sanity.42d.test sanity.201c.test sanity.200b.test sanity.201b.test sanity.200d.test sanity.51c.test sanity.42d.test sanity.129.test sanity.201c.test +200h +200c +200b -129 sanityn sanityn.14b.test sanityn.1b.test sanityn.1c.test sanityn.28.test sanityn.29.test sanityn.14b.test sanityn.1b.test sanityn.1c.test sanityn.28.test sanityn.29.test no diff sanity-quota none sanity-quota.18b.test -18b conf-sanity none none no diff ost-pools none none no diff lustre-rsync-test none none no diff insanity insanity.10.test insanity.10.test no diff replay-vbr none none no diff replay-dual none none no diff replay-ost-single none none no diff recovery-small recovery-small.3.test recovery-small.5.test recovery-small.52.test recovery-small.2.test recovery-small.3.test recovery-small.5.test recovery-small.52.test recovery-small.2.test no diff replay-single replay-single.44a.test replay-single.44a.test no diff Thanks, Roman On 05/15/2012 12:59 AM, Alexander Lezhoev wrote: > Hi there, > > Let me raise the question about Lustre tests separated execution. > We've discussed this problem already, but I'd like to clear up some > details. > > Usually we run all tests sequentially, but in the automation tool we > are using we need to run tests separately, with ONLY parameter. This > allows us to have full control over the test execution: terminate hung > tests by timeout or restore environment in case of the file system > damage. At the moment some of tests are designed to be run sequentially. > > By our estimation, there are about 30 tests need for the improvement. > If we settle this question, we can significantly improve > test-framework automation potential. > Please share your opinions about this question and help to make a > decision about it. > The questions are > > 1. Do we want to have an ability to run each test independently? > 2. What is more acceptable - unite sequential tests into complex ones > or supplement exists test with additional code steps? > > > > Some technical details: > > Typical problem is sanityn test_1 > > test_1a() { > touch $DIR1/f1 > [ -f $DIR2/f1 ] || error > } > run_test 1a "check create on 2 mtpt's ==========================" > > test_1b() { > chmod 777 $DIR2/f1 > $CHECKSTAT -t file -p 0777 $DIR1/f1 || error > chmod a-x $DIR2/f1 > } > run_test 1b "check attribute updates on 2 mtpt's ===============" > > test_1c() { > $CHECKSTAT -t file -p 0666 $DIR1/f1 || error > } > run_test 1c "check after remount attribute updates on 2 mtpt's =" > > test_1d() { > rm $DIR2/f1 > $CHECKSTAT -a $DIR1/f1 || error > } > run_test 1d "unlink on one mountpoint removes file on other ====" > > > They cannot be run separately, because the next index uses the code of > previous one. This means all tests should be run in groups of letter > indexes, or they should be refactored to run independently. > > Some of tests have been already refactored to run "letters" > separately, but we have to make a rule which we should follow and use > for further refactoring. > There are three decisions we can take about this situation > > * Join the code of all test steps into single test with > corresponding number. So we will have one test_1 instead of > test_1a .. test_1d in the described case. > * Move the code of steps to corresponding functions which will be > called from each step. In other words the next indexed test will > duplicate some functionality of previous one. > * Do nothing and decide that "letters" mustn't be executed > independently, but only in "number" group. > > > The first variant could be implemented as follows. > > test_1() { > > touch $DIR1/f1 > > [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed" > > chmod 777 $DIR2/f1 > > $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates > on 2 mtpt's failed" > > chmod a-x $DIR2/f1 > > $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount > attribute updates on 2 mtpt's failed" > > rm $DIR2/f1 > > $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file > on other failed" > > } > run_test 1 "check attributes updates on 2 mtpt's" > > This approach has disadvantage that such kind of refactoring will lead > to reduction of test numbering and it will hard to work with > regression history of the refactored tests. > The second case of refactoring can look like this: > > test_1a() { > test_1_create > } > run_test 1a "check create on 2 mtpt's ==========================" > > test_1b() { > test_1_create > test_1_check_attr > } > run_test 1b "check attribute updates on 2 mtpt's ===============" > > test_1c() { > test_1_create > test_1_check_attr > test_1_check_attr2 > } > run_test 1c "check after remount attribute updates on 2 mtpt's =" > > test_1d() { > test_1_create > test_1_check_attr > test_1_check_attr2 > test_1_unlink > } > run_test 1d "unlink on one mountpoint removes file on other ====" > > I've omitted functions code - their content is obvious. > This disadvantage of this approach --- summary increase of tests > run-time (the next test duplicates code of previous one). But the > necessity of all these tests is doubtful here, because the last one > includes first three tests. > > Very similar situation is for recovery-small 1, 2, 3 and 4, 5. > > test_1() { > drop_request "mcreate $DIR/f1" || return 1 > drop_reint_reply "mcreate $DIR/f2" || return 2 > } > run_test 1 "mcreate: drop req, drop rep" > > test_2() { > drop_request "tchmod 111 $DIR/f2" || return 1 > drop_reint_reply "tchmod 666 $DIR/f2" || return 2 > } > run_test 2 "chmod: drop req, drop rep" > > test_3() { > drop_request "statone $DIR/f2" || return 1 > drop_reply "statone $DIR/f2" || return 2 > } > run_test 3 "stat: drop req, drop rep" > > These three tests are actually steps of a single test scenario, > because they work with the results of previous ones. > > We can separate these tests: > > test_1() { > test_1_mcreate > } > run_test 1 "mcreate: drop req, drop rep" > > test_2() { > test_1_mcreate > test_2_chmod > } > run_test 2 "chmod: drop req, drop rep" > > test_3() { > test_1_mcreate > test_3_stat > } > run_test 3 "stat: drop req, drop rep" > > or join them into one and remove test_2 and test_3. > > test_1() { > drop_request "mcreate $DIR/f1" || return 1 > drop_reint_reply "mcreate $DIR/f2" || return 2 > drop_request "tchmod 111 $DIR/f2" || return 3 > drop_reint_reply "tchmod 666 $DIR/f2" || return 4 > drop_request "statone $DIR/f2" || return 5 > drop_reply "statone $DIR/f2" || return 6 > } > run_test 1 "mcreate, chmod,stat: drop req, drop,req" > > > Another big example are sanity tests 200 and 201. Here is the part of > the resulting code after refactoring, so we can separately run each > letter index: > > test_200a() { > test_200_create_pool > test_201_remove_pool > } > run_test 200a "Create new pool ==========================================" > > test_200b() { > test_200_create_pool > test_200_add_targets > test_201_remove_all_targets > test_201_remove_pool > } > > . . . > > test_201b() { > test_200_create_pool > test_200_add_targets > test_200_dir_set_pool > test_200_check_dir_pool > test_200_check_file_alloc > test_200_create_files > test_200_create_relative_path_files > test_201_remove_all_targets > test_201_remove_pool > } > run_test 201b "Remove all targets from a pool ==========================" > > test_201c() { > test_200_create_pool > test_201_remove_pool > } > run_test 201c "Remove a pool ============================================" > > > We have to include cleanup steps here to make possible to run letter > indexes independently. With that cleanup steps, 200a and 201c became > absolutely equal and need to be reduced. Same situation is for 200h > and 201b. > > > > Sorry for so long email and thanks to Kyr > (Kyrylo_Shatskyy at xyratex.com) for it's preparing. > > -- > Alexander Lezhoev. > Morpheus test team. > Xyratex. > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Thu May 17 00:36:53 2012 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Wed, 16 May 2012 17:36:53 -0700 Subject: [Twg] TWG Requirements Message-ID: <4FB44825.8060804@llnl.gov> LLNL's list: 1. Avoid RPC timeouts 2. Improved LNET robustness 3. Dynamic LNET configuration 4. Complex File Layouts 5. OST migration/rebalancing 6. Single Client Performance 7. Scalable Fault Management 8. Improved Configuration Robustness 9. Better Userspace Tools 10. File create performance Chris Morrone From carrier at cray.com Thu May 17 11:54:01 2012 From: carrier at cray.com (John Carrier) Date: Thu, 17 May 2012 11:54:01 +0000 Subject: [Twg] TWG development requirements voting Message-ID: We still need your help identifying the top requirements, which the TWG will propose that OpenSFS fund for 2013. Our goal is to issue an RFP and complete contract negotiations by SC12. All organizations in the Lustre community are welcome to submit a list. I've received priority lists from the following organizations: Fermi, IU, CEA, Bull, Whamcloud, Terascala, and LLNL. Please let me know ASAP if you submitted a list and are not mentioned above. Dave and I had hoped that we would wrap up voting before this week's TWG call. But at this point, I don't have lists from 4 of the 5 OpenSFS promoters (Cray, DDN, ORNL, Xyratex). Therefore, I propose that we wrap up the voting this week so that I can publish the tally next week for discussion at our 5/24 meeting. Here's the "new" schedule: Fri 5/18 - final deadline to submit top 10 requirments in priority order Mon 5/21 - TWG to publish summary of votes Thur 5/24 - TWG meeting to resolve summary and make recommendation to board Voting is really just a matter of submitting to discuss at lists.opensfs.org a prioritized list of top10 features from our gathered requirements http://goo.gl/cZSWG. Although we will include all lists in our analysis, voting priority will be given to OpenSFS members to resolve ties. The OpenSFS board has ultimate authority for recommending which features should be funded for development. Please let us know if you have any questions. John & Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From spitzcor at cray.com Thu May 17 14:00:41 2012 From: spitzcor at cray.com (Cory Spitz) Date: Thu, 17 May 2012 09:00:41 -0500 Subject: [Twg] [Discuss] TWG development requirements voting In-Reply-To: References: Message-ID: <4FB50489.7000705@cray.com> > Voting is really just a matter of submitting to discuss at lists.opensfs.org a prioritized list of top10 features from our gathered requirements http://goo.gl/cZSWG. That link points to last year's requirements. Please go to http://goo.gl/63u9Q for this year's requirements. Thanks, -Cory On 05/17/2012 06:54 AM, John Carrier wrote: > We still need your help identifying the top requirements, which the TWG > will propose that OpenSFS fund for 2013. Our goal is to issue an RFP > and complete contract negotiations by SC12. All organizations in the > Lustre community are welcome to submit a list. > > > > I’ve received priority lists from the following organizations: Fermi, > IU, CEA, Bull, Whamcloud, Terascala, and LLNL. > > Please let me know ASAP if you submitted a list and are not mentioned > above. > > > > Dave and I had hoped that we would wrap up voting before this week’s TWG > call. But at this point, I don’t have lists from 4 of the 5 OpenSFS > promoters (Cray, DDN, ORNL, Xyratex). Therefore, I propose that we > wrap up the voting this week so that I can publish the tally next week > for discussion at our 5/24 meeting. Here’s the “new” schedule: > > Fri 5/18 – final deadline to submit top 10 requirments in priority order > > Mon 5/21 – TWG to publish summary of votes > > Thur 5/24 – TWG meeting to resolve summary and make recommendation to board > > > > Voting is really just a matter of submitting to > discuss at lists.opensfs.org a > prioritized list of top10 features from our gathered requirements > http://goo.gl/cZSWG. Although we will include all lists in our > analysis, voting priority will be given to OpenSFS members to resolve > ties. The OpenSFS board has ultimate authority for recommending which > features should be funded for development. > > > > Please let us know if you have any questions. > > > > John & Dave > > > > > > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org From adilger at whamcloud.com Thu May 17 16:24:03 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 17 May 2012 10:24:03 -0600 Subject: [Twg] [Discuss] TWG development requirements voting In-Reply-To: <4FB50489.7000705@cray.com> References: <4FB50489.7000705@cray.com> Message-ID: <3B58C32D-6A4F-4FE0-AFF1-C3933DF7524F@whamcloud.com> On 2012-05-17, at 8:00 AM, Cory Spitz wrote: >> Voting is really just a matter of submitting to discuss at lists.opensfs.org a prioritized list of top10 features from our gathered requirements http://goo.gl/cZSWG. > > That link points to last year's requirements. Please go to > http://goo.gl/63u9Q for this year's requirements. Will there be a call this week, or are we waiting on priority lists? > On 05/17/2012 06:54 AM, John Carrier wrote: >> We still need your help identifying the top requirements, which the TWG >> will propose that OpenSFS fund for 2013. Our goal is to issue an RFP >> and complete contract negotiations by SC12. All organizations in the >> Lustre community are welcome to submit a list. >> >> >> >> I’ve received priority lists from the following organizations: Fermi, >> IU, CEA, Bull, Whamcloud, Terascala, and LLNL. >> >> Please let me know ASAP if you submitted a list and are not mentioned >> above. >> >> >> >> Dave and I had hoped that we would wrap up voting before this week’s TWG >> call. But at this point, I don’t have lists from 4 of the 5 OpenSFS >> promoters (Cray, DDN, ORNL, Xyratex). Therefore, I propose that we >> wrap up the voting this week so that I can publish the tally next week >> for discussion at our 5/24 meeting. Here’s the “new” schedule: >> >> Fri 5/18 – final deadline to submit top 10 requirments in priority order >> >> Mon 5/21 – TWG to publish summary of votes >> >> Thur 5/24 – TWG meeting to resolve summary and make recommendation to board >> >> >> >> Voting is really just a matter of submitting to >> discuss at lists.opensfs.org a >> prioritized list of top10 features from our gathered requirements >> http://goo.gl/cZSWG. Although we will include all lists in our >> analysis, voting priority will be given to OpenSFS members to resolve >> ties. The OpenSFS board has ultimate authority for recommending which >> features should be funded for development. >> >> >> >> Please let us know if you have any questions. >> >> >> >> John & Dave >> >> >> >> >> >> _______________________________________________ >> discuss mailing list >> discuss at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From carrier at cray.com Thu May 17 16:53:34 2012 From: carrier at cray.com (carrier@cray.com (Google Docs)) Date: Thu, 17 May 2012 16:53:34 +0000 Subject: [Twg] TWG Requirements Priorities (twg@lists.opensfs.org) Message-ID: <14dae93407d9bf546b04c03e46a8@google.com> I've shared an item with you: TWG Requirements Priorities https://docs.google.com/spreadsheet/ccc?key=0Al6kUqlvuNX8dHNSc0pkVW92cmk4NFRRcjZhOWt0ZGc&invite=COmfoo4N It's not an attachment -- it's stored online at Google Docs. To open this document, just click the link above. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dillowda at ornl.gov Thu May 17 17:47:25 2012 From: dillowda at ornl.gov (David Dillow) Date: Thu, 17 May 2012 13:47:25 -0400 Subject: [Twg] [Discuss] TWG development requirements voting In-Reply-To: References: Message-ID: <1337276845.2895.2.camel@frustration.ornl.gov> On Thu, 2012-05-17 at 07:54 -0400, John Carrier wrote: > We still need your help identifying the top requirements, which the > TWG will propose that OpenSFS fund for 2013. Our goal is to issue an > RFP and complete contract negotiations by SC12. All organizations in > the Lustre community are welcome to submit a list. [snip] > Dave and I had hoped that we would wrap up voting before this week’s > TWG call. But at this point, I don’t have lists from 4 of the 5 > OpenSFS promoters (Cray, DDN, ORNL, Xyratex). Therefore, I propose > that we wrap up the voting this week so that I can publish the tally > next week for discussion at our 5/24 meeting. Here’s the “new” > schedule: > > Fri 5/18 – final deadline to submit top 10 requirments in priority > order > > Mon 5/21 – TWG to publish summary of votes After today's meeting, we have modified the schedule: Wed, 5/23 is now the final deadline to submit the prioritized requirements. TWG will publish a summary of the voting prior to our meeting on 5/24. > Thur 5/24 – TWG meeting to resolve summary and make recommendation to > board This potion of the schedule remains unchanged. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From adilger at whamcloud.com Thu May 17 17:50:58 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 17 May 2012 11:50:58 -0600 Subject: [Twg] TWG Requirements Priorities In-Reply-To: <14dae93407d9bf546b04c03e46a8@google.com> References: <14dae93407d9bf546b04c03e46a8@google.com> Message-ID: On 2012-05-17, at 10:53 AM, carrier at cray.com (Google Docs) wrote: > I've shared TWG Requirements Priorities > Click to open: > https://docs.google.com/spreadsheet/ccc?key=0Al6kUqlvuNX8dHNSc0pkVW92cmk4NFRRcjZhOWt0ZGc&invite=COmfoo4N > > Google Docs makes it easy to create, store and share online documents, spreadsheets and presentations. Just an FYI, I added columns for the Weighted sum, which is (13 - score) for each item. I chose this over (11 - score) since I figured that more votes should count for something compared to fewer. I also tried (11^2 - score^2) to give more weighting to higher placement in the list (at least for us the last few items were not really top priorities anymore), but this doesn't change the rankings very much (a couple of items near the threshold changed places. It will be interesting to see if the rankings of the remaining TWG members affect the ranking in any significant way. The one drawback is that the remaining members can rank based on current standings rather than purely based on their own priorities. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Nathan_Rutman at xyratex.com Thu May 17 18:45:09 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Thu, 17 May 2012 11:45:09 -0700 Subject: [Twg] [Discuss] TWG development requirements voting In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A92102F10A6C@XYUS-EX22.xyus.xyratex.com> Sorry John, apparently this never made it out my mailbox :( Begin forwarded message: From: Nathan Rutman Subject: OpenSFS 2012 Priorities Date: May 3, 2012 1:22:50 PM PDT To: twg at lists.opensfs.org Cc: Nathan Rutman Xyratex list: 1. Scalable fault management - faster detection and reporting of problems, vital for monitoring and our quality story 2. Avoid RPC timeouts - similar to above, better user feel. 3. Improved Lustre tests - quality 4. QOS - better interactive user feel 5. HSM 6. OST migration / space rebalancing 7. Asynchronous file replication 8. IPv6 support 9. directory traversal / attribute retreival - better interactive user feel 10. LNET channel bonding On May 17, 2012, at 4:54 AM, John Carrier wrote: We still need your help identifying the top requirements, which the TWG will propose that OpenSFS fund for 2013. Our goal is to issue an RFP and complete contract negotiations by SC12. All organizations in the Lustre community are welcome to submit a list. I’ve received priority lists from the following organizations: Fermi, IU, CEA, Bull, Whamcloud, Terascala, and LLNL. Please let me know ASAP if you submitted a list and are not mentioned above. Dave and I had hoped that we would wrap up voting before this week’s TWG call. But at this point, I don’t have lists from 4 of the 5 OpenSFS promoters (Cray, DDN, ORNL, Xyratex). Therefore, I propose that we wrap up the voting this week so that I can publish the tally next week for discussion at our 5/24 meeting. Here’s the “new” schedule: Fri 5/18 – final deadline to submit top 10 requirments in priority order Mon 5/21 – TWG to publish summary of votes Thur 5/24 – TWG meeting to resolve summary and make recommendation to board Voting is really just a matter of submitting to discuss at lists.opensfs.org a prioritized list of top10 features from our gathered requirements http://goo.gl/cZSWG. Although we will include all lists in our analysis, voting priority will be given to OpenSFS members to resolve ties. The OpenSFS board has ultimate authority for recommending which features should be funded for development. Please let us know if you have any questions. John & Dave _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu May 17 19:29:26 2012 From: carrier at cray.com (John Carrier) Date: Thu, 17 May 2012 19:29:26 +0000 Subject: [Twg] TWG Requirements Priorities In-Reply-To: References: <14dae93407d9bf546b04c03e46a8@google.com> Message-ID: Folks, this document is a shared resource that gets updated synchronously. If you want to play with different weightings, please pull down a local copy (as excel, to preserve the formulas) and make your changes to the local copy. Thanks, --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Andreas Dilger Sent: Thursday, May 17, 2012 10:51 AM To: twg at lists.opensfs.org Subject: Re: [Twg] TWG Requirements Priorities On 2012-05-17, at 10:53 AM, carrier at cray.com (Google Docs) wrote: > I've shared TWG Requirements Priorities > Click to open: > https://docs.google.com/spreadsheet/ccc?key=0Al6kUqlvuNX8dHNSc0pkVW92cmk4NFRRcjZhOWt0ZGc&invite=COmfoo4N > > Google Docs makes it easy to create, store and share online documents, spreadsheets and presentations. Just an FYI, I added columns for the Weighted sum, which is (13 - score) for each item. I chose this over (11 - score) since I figured that more votes should count for something compared to fewer. I also tried (11^2 - score^2) to give more weighting to higher placement in the list (at least for us the last few items were not really top priorities anymore), but this doesn't change the rankings very much (a couple of items near the threshold changed places. It will be interesting to see if the rankings of the remaining TWG members affect the ranking in any significant way. The one drawback is that the remaining members can rank based on current standings rather than purely based on their own priorities. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From dillowda at ornl.gov Wed May 23 21:38:24 2012 From: dillowda at ornl.gov (David Dillow) Date: Wed, 23 May 2012 17:38:24 -0400 Subject: [Twg] ORNL's prioritized requirement list Message-ID: <1337809104.4867.7.camel@frustration.ornl.gov> 1 File create performance 2 Directory traversal performance 3 Avoiding RPC timeouts 4 Scalable fault management 5 LNET channel bonding 6 Single client performance 7 Single shared file performance 8 Dynamic LNET configuration 9 Improved Lustre tests 10 OST migration/rebalancing -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From carrier at cray.com Thu May 24 15:56:02 2012 From: carrier at cray.com (John Carrier) Date: Thu, 24 May 2012 15:56:02 +0000 Subject: [Twg] Cray priorities list Message-ID: Cray has focused on requirements in the Availability, Storage Management, and Performance categories.   We believe the requirements and features described in the Availability and Storage Management are foundational in nature and, as a result, we need to start working on them now to ensure feature landings in years to come.    We also see the requirements in the LNET category as improving Lustre resiliency and, as such, we consider them as additions to the Availability category.   We are hopeful that the new metadata features will greatly improve metadata performance.  Nonetheless, we recognize that we must never be complacent that Lustre performance is "good enough" and, as new performance improvements land, we must look for the next critical bottleneck.  Finally, we think the recent discussions of Lustre validation and test tools suggest that community investment is warranted to help create a consistent solution. Our list of top priorities follows.  Our preference is that at least one requirement in each of the three above categories should be represented in the RFP.   Nonetheless, we have grouped the requirements together by category to make the list easier to parse. 1. Avoid RPC timeouts 2. Scalable fault management 3. File create performance 4. Single client performance 5. HSM and storage management infrastructure 6. OST migration/rebalancing 7. Asynchronous file replication (mirroring) 8. Improved LNET robustness 9. LNET channel bonding 10. Improved Lustre tests From carrier at cray.com Thu May 24 16:28:31 2012 From: carrier at cray.com (John Carrier) Date: Thu, 24 May 2012 16:28:31 +0000 Subject: [Twg] updated priorities spreadsheet Message-ID: Alex has been working on the spreadsheet. Here's the link to the new document. https://docs.google.com/spreadsheet/ccc?key=0Al6kUqlvuNX8dDYzVldpNTRPYUozbzZHU2hfTkxqZFE BTW, I found that I could see the results more easily by downloading it to my local machine as an excel spreadsheet. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Sat May 26 22:33:38 2012 From: carrier at cray.com (John Carrier) Date: Sat, 26 May 2012 22:33:38 +0000 Subject: [Twg] new priority spreadsheet Message-ID: On the TWG call Thursday 5/24, we agreed that we would recommend to the board that OpenSFS focus on new feature development to address our top requirements in Availability, Storage Management, and Performance. As I was writing the text summarizing this recommendation, I noticed that Alex's spreadsheet had grouped some member organizations in the community group and included some other members in the group of promoters. At our 5/17 meeting, we had agreed to consider two groupings--promoters and non-promoters--to reflect the representation on the OpenSFS Board. As a result, I have modified Alex's tables to reflect this organization for analysis. This new grouping of organizations does not qualitatively alter our conclusions and our recommendation to the Board will be unchanged. While working on the spreadsheets, I also decided to consolidate the different sheets Alex had created to make viewing the results clearer. Also, with Alex's help, I have formatted the columns so that each page in the spreadsheet will print on a single page. The new spreadsheet (http://goo.gl/Lqg7s) now has three pages : "Priorities" lists all of our votes. It shows the vote counts and the sum of squares analysis that Andreas had proposed. "Scores" uses Alex's method of converting priorities 1-10 to scores 100-10. The table includes the scores summed by promoter, non-promoter, and all members. "Sorted Scores" contains four tables that show the feature ranking by promoter scores, non-promoter scores, all member scores, and total votes cast. I have had trouble accessing alternate pages in the spreadsheet with Firefox and so did my final edits with Chrome. For printing, another alternative is to download the sheet in Excel or OpenOffice and then print from these apps on your system. Please let me know if you have any questions or comments. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Sat May 26 22:55:09 2012 From: carrier at cray.com (John Carrier) Date: Sat, 26 May 2012 22:55:09 +0000 Subject: [Twg] Cray priorities list In-Reply-To: <266C57B7-8EBA-462D-97D6-66F6FFA129C9@whamcloud.com> References: <266C57B7-8EBA-462D-97D6-66F6FFA129C9@whamcloud.com> Message-ID: Andreas requested that Cray order our list in priority order. 1. Avoid RPC timeouts 2. File create performance 3. HSM and storage management infrastructure 4. Single client performance 5. Scalable fault management 6. Improved LNET robustness 7. OST migration/rebalancing 8. LNET channel bonding 9. Asynchronous file replication (mirroring) 10. Improved Lustre tests This new order is reflected in the new spreadsheet update. The change does not qualitatively change our recommendations to the board. Thanks, --jc -----Original Message----- From: Andreas Dilger [mailto:adilger at whamcloud.com] Sent: Thursday, May 24, 2012 9:20 AM To: John Carrier Subject: Re: [Twg] Cray priorities list On 2012-05-24, at 9:56 AM, John Carrier wrote: > Our list of top priorities follows. Our preference is that at least one requirement in each of the three above categories should be represented in the RFP. Nonetheless, we have grouped the requirements together by category to make the list easier to parse. > > 1. Avoid RPC timeouts > 2. Scalable fault management > 3. File create performance > 4. Single client performance > 5. HSM and storage management infrastructure > 6. OST migration/rebalancing > 7. Asynchronous file replication (mirroring) > 8. Improved LNET robustness > 9. LNET channel bonding > 10. Improved Lustre tests John, the other lists were provided in priority order, and the spreadsheet is using this to select top-ranked aggregate features. Could you please add these to the spreadsheet in ranked order, or alternately I guess you could use "5" as the rank for all items. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From carrier at cray.com Tue May 29 06:05:14 2012 From: carrier at cray.com (John Carrier) Date: Tue, 29 May 2012 06:05:14 +0000 Subject: [Twg] TWG meeting 5/29/2012 Message-ID: Hi all, At last week's regularly scheduled TWG meeting, we agreed to add an additional meeting this week so that we can discuss our recommendation to the OpenSFS Board. Meeting time is 3:00pm ET / 12:00pm PT, Tuesday 5/29/2012. Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 Following our last meeting, we created a draft recommendation based on our review of the initial spreadsheet: https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 Our gathered requirements are here: http://goo.gl/63u9Q The spreadsheet summarizing our priorities is here: http://goo.gl/Lqg7s Our agenda for this meeting is to discuss the draft recommendation and the steps for submitting this to the Board. Please note that we plan to have our regular Thursday meeting this week (5/31/2012, 12:30ET/9:30PT). If the Board approves our recommendation, we want to have an RFP ready for ISC. Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Tue May 29 16:23:35 2012 From: carrier at cray.com (John Carrier) Date: Tue, 29 May 2012 16:23:35 +0000 Subject: [Twg] DDN requirements Message-ID: I received this list from DDN over the weekend. I note that the first two items on their list were not captured during our requirements gathering this spring. I think we should add them to our requirements document, but we need to discuss how they impact our recommendations to the OpenSFS Board. I have added these to the spreadsheet and taken the liberty of matching "patch-less server support" with our existing "better support for newer kernels". --jc DDN's Lustre requirements for OpenSFS TWG 1) Architecture/Developer documentation Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. 2) Improved small and medium I/O performance (compare to GPFS) 3) Single client performance while preserving scalability (even at a single thread per client) 4) OST migration/rebalancing (compare to GPFS) 5) LNET channel bonding 6) Patch-less server support 7) Single MDS File create performance 8) Single MDS Directory traversal and attribute retrieval performance 9) Improved Lustre tests and testing infrastructure 10) Quality of service (mechanism and policies) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dillowda at ornl.gov Tue May 29 19:45:02 2012 From: dillowda at ornl.gov (David Dillow) Date: Tue, 29 May 2012 15:45:02 -0400 Subject: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations Message-ID: <1338320702.2398.39.camel@obelisk.thedillows.org> We discussed the recommendation to the board as well as testing and release management topics during today's additional TWG meeting. We will be incorporating the recommendation into the requirements document as we did last year. We will note the community's interest in improving the test infrastructure as maintaining patch lists for bug fix releases, but will refer those topics to the CWG. We will be sending out updated documents later this evening. Due to the poor attendance for today's call, we have extended the schedule slightly; we will be forwarding the recommendation to the board shortly after Thursday's meeting. If you have wish to have your comments heard, please make every effort to attend this week's TWG meeting. Thursday, 12:30pm ET/9:30 PT. 715-726-4994 or toll-free 866-304-8294 Meeting ID and password: 72090 Current draft recommendation based on our review of the initial spreadsheet (to be revised and redistributed this evening): https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 Gathered requirements: http://goo.gl/63u9Q The spreadsheet summarizing our priorities: http://goo.gl/Lqg7s Thanks, John Carrier Dave Dillow -------------- next part -------------- _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From adilger at whamcloud.com Tue May 29 22:01:18 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Tue, 29 May 2012 16:01:18 -0600 Subject: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations In-Reply-To: <1338320702.2398.39.camel@obelisk.thedillows.org> References: <1338320702.2398.39.camel@obelisk.thedillows.org> Message-ID: <5520FFBB-7CBB-439F-BBE6-157EE2FF7935@whamcloud.com> On 2012-05-29, at 1:45 PM, David Dillow wrote: > We discussed the recommendation to the board as well as testing and > release management topics during today's additional TWG meeting. > We will be incorporating the recommendation into the requirements > document as we did last year. We will note the community's interest in > improving the test infrastructure as maintaining patch lists for bug fix > releases, but will refer those topics to the CWG. > > We will be sending out updated documents later this evening. > > Due to the poor attendance for today's call, we have extended the > schedule slightly; we will be forwarding the recommendation to the board > shortly after Thursday's meeting. If you have wish to have your comments > heard, please make every effort to attend this week's TWG meeting. Apologies on missing the meeting. I didn't have anything in my calendar, and I guess I thought there would be a meeting invitation sent out. I'll see if I have any further feedback on the updated doc. > Thursday, 12:30pm ET/9:30 PT. > 715-726-4994 or toll-free 866-304-8294 > Meeting ID and password: 72090 > > > Current draft recommendation based on our review of the initial > spreadsheet (to be revised and redistributed this evening): > https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 > > Gathered requirements: http://goo.gl/63u9Q > The spreadsheet summarizing our priorities: http://goo.gl/Lqg7s > > Thanks, > John Carrier > Dave Dillow > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From Nathan_Rutman at xyratex.com Tue May 29 23:54:46 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 29 May 2012 16:54:46 -0700 Subject: [Twg] [Discuss] DDN requirements In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A92102F10AC3@XYUS-EX22.xyus.xyratex.com> On May 29, 2012, at 9:23 AM, John Carrier wrote: I received this list from DDN over the weekend. I note that the first two items on their list were not captured during our requirements gathering this spring. I think we should add them to our requirements document, but we need to discuss how they impact our recommendations to the OpenSFS Board. I have added these to the spreadsheet and taken the liberty of matching “patch-less server support” with our existing “better support for newer kernels”. --jc DDN's Lustre requirements for OpenSFS TWG 1) Architecture/Developer documentation Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. Interestingly, this first item is not on anyone else's list, however I also think it is important for the dissemination of Lustre knowledge and to help grow the Lustre contributor community. I also believe Oak Ridge feels strongly about this, as the producers of the first Lustre Internals doc. I think an expansion and update of this is worthy of funding consideration. 2) Improved small and medium I/O performance (compare to GPFS) 3) Single client performance while preserving scalability (even at a single thread per client) 4) OST migration/rebalancing (compare to GPFS) 5) LNET channel bonding 6) Patch-less server support 7) Single MDS File create performance 8) Single MDS Directory traversal and attribute retrieval performance 9) Improved Lustre tests and testing infrastructure 10) Quality of service (mechanism and policies) _______________________________________________ discuss mailing list discuss at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Tue May 29 23:59:00 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Tue, 29 May 2012 16:59:00 -0700 Subject: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations In-Reply-To: <5520FFBB-7CBB-439F-BBE6-157EE2FF7935@whamcloud.com> References: <1338320702.2398.39.camel@obelisk.thedillows.org> <5520FFBB-7CBB-439F-BBE6-157EE2FF7935@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F10AC5@XYUS-EX22.xyus.xyratex.com> My apologies as well. I added a paragraph at the bottom of the Draft http://goo.gl/7ncPT we can discuss on Thurs. Also for completeness we should talk about this: 1) Architecture/Developer documentation Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. On May 29, 2012, at 3:01 PM, Andreas Dilger wrote: > On 2012-05-29, at 1:45 PM, David Dillow wrote: >> We discussed the recommendation to the board as well as testing and >> release management topics during today's additional TWG meeting. >> We will be incorporating the recommendation into the requirements >> document as we did last year. We will note the community's interest in >> improving the test infrastructure as maintaining patch lists for bug fix >> releases, but will refer those topics to the CWG. >> >> We will be sending out updated documents later this evening. >> >> Due to the poor attendance for today's call, we have extended the >> schedule slightly; we will be forwarding the recommendation to the board >> shortly after Thursday's meeting. If you have wish to have your comments >> heard, please make every effort to attend this week's TWG meeting. > > Apologies on missing the meeting. I didn't have anything in my calendar, and I guess I thought there would be a meeting invitation sent out. I'll see if I have any further feedback on the updated doc. > >> Thursday, 12:30pm ET/9:30 PT. >> 715-726-4994 or toll-free 866-304-8294 >> Meeting ID and password: 72090 >> >> >> Current draft recommendation based on our review of the initial >> spreadsheet (to be revised and redistributed this evening): >> https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 >> >> Gathered requirements: http://goo.gl/63u9Q >> The spreadsheet summarizing our priorities: http://goo.gl/Lqg7s >> >> Thanks, >> John Carrier >> Dave Dillow >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From morrone2 at llnl.gov Wed May 30 00:28:55 2012 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 29 May 2012 17:28:55 -0700 Subject: [Twg] [Discuss] DDN requirements In-Reply-To: References: Message-ID: <4FC569C7.10701@llnl.gov> Yes, I think that internals documentation would be excellent. FYI, there will be some OSD API documentation that results from LLNL's OSD contract work. Chris On 05/29/2012 05:07 PM, Ian Colle wrote: > I think LLNL would also appreciate making #1 a priority. > > Chris Morrone – can you confirm? > > > Ian R. Colle > HPC Project Manager, Whamcloud > Cell: +1.303.601.7713 > Email: ian at whamcloud.com > > [http://images.wisestamp.com/apps/buttons/linkedinbutton_option_3.png] > [http://twitter-badges.s3.amazonaws.com/follow_me-a.png] > > From: Nathan Rutman> > Date: Tue, 29 May 2012 16:54:46 -0700 > To: John Carrier> > Cc:>,> > Subject: Re: [Discuss] DDN requirements > > > On May 29, 2012, at 9:23 AM, John Carrier wrote: > > I received this list from DDN over the weekend. I note that the first two items on their list were not captured during our requirements gathering this spring. I think we should add them to our requirements document, but we need to discuss how they impact our recommendations to the OpenSFS Board. > > I have added these to the spreadsheet and taken the liberty of matching “patch-less server support” with our existing “better support for newer kernels”. > > --jc > > DDN's Lustre requirements for OpenSFS TWG > > 1) Architecture/Developer documentation > Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. > > Interestingly, this first item is not on anyone else's list, however I also think it is important for the dissemination of Lustre knowledge and to help grow the Lustre contributor community. I also believe Oak Ridge feels strongly about this, as the producers of the first Lustre Internals doc. I think an expansion and update of this is worthy of funding consideration. > > > > 2) Improved small and medium I/O performance (compare to GPFS) > 3) Single client performance while preserving scalability (even at a single thread per client) > 4) OST migration/rebalancing (compare to GPFS) > 5) LNET channel bonding > 6) Patch-less server support > 7) Single MDS File create performance > 8) Single MDS Directory traversal and attribute retrieval performance > 9) Improved Lustre tests and testing infrastructure > 10) Quality of service (mechanism and policies) > > > > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org > > _______________________________________________ discuss mailing list discuss at lists.opensfs.orghttp://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org > From ian at whamcloud.com Wed May 30 00:07:21 2012 From: ian at whamcloud.com (Ian Colle) Date: Tue, 29 May 2012 18:07:21 -0600 Subject: [Twg] [Discuss] DDN requirements In-Reply-To: <73AED5C780AE05478241DB067651A92102F10AC3@XYUS-EX22.xyus.xyratex.com> Message-ID: I think LLNL would also appreciate making #1 a priority. Chris Morrone ­ can you confirm? Ian R. Colle HPC Project Manager, Whamcloud Cell: +1.303.601.7713 Email: ian at whamcloud.com From: Nathan Rutman Date: Tue, 29 May 2012 16:54:46 -0700 To: John Carrier Cc: , Subject: Re: [Discuss] DDN requirements On May 29, 2012, at 9:23 AM, John Carrier wrote: > I received this list from DDN over the weekend. I note that the first two > items on their list were not captured during our requirements gathering this > spring. I think we should add them to our requirements document, but we need > to discuss how they impact our recommendations to the OpenSFS Board. > > I have added these to the spreadsheet and taken the liberty of matching > ³patch-less server support² with our existing ³better support for newer > kernels². > > --jc > > DDN's Lustre requirements for OpenSFS TWG > > 1) Architecture/Developer documentation > Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. Interestingly, this first item is not on anyone else's list, however I also think it is important for the dissemination of Lustre knowledge and to help grow the Lustre contributor community. I also believe Oak Ridge feels strongly about this, as the producers of the first Lustre Internals doc. I think an expansion and update of this is worthy of funding consideration. > > 2) Improved small and medium I/O performance (compare to GPFS) > 3) Single client performance while preserving scalability (even at a single > thread per client) > 4) OST migration/rebalancing (compare to GPFS) > 5) LNET channel bonding > 6) Patch-less server support > 7) Single MDS File create performance > 8) Single MDS Directory traversal and attribute retrieval performance > 9) Improved Lustre tests and testing infrastructure > 10) Quality of service (mechanism and policies) > > > > _______________________________________________ > discuss mailing list > discuss at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/discuss-opensfs.org _______________________________________________ discuss mailing list discuss at lists.opensfs.orghttp://lists.opensfs.org/listinfo.cgi/discuss-opens fs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Wed May 30 04:20:13 2012 From: carrier at cray.com (John Carrier) Date: Wed, 30 May 2012 04:20:13 +0000 Subject: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations In-Reply-To: <73AED5C780AE05478241DB067651A92102F10AC5@XYUS-EX22.xyus.xyratex.com> References: <1338320702.2398.39.camel@obelisk.thedillows.org> <5520FFBB-7CBB-439F-BBE6-157EE2FF7935@whamcloud.com> <73AED5C780AE05478241DB067651A92102F10AC5@XYUS-EX22.xyus.xyratex.com> Message-ID: For everyone's benefit, Nathan added the following text to the recommendation: [NZR] Also, an LNET channel bonding requirement was ranked highly by OpenSFS promoters; however this feature may address availability (if graceful degradation facilities are included) as well as improved single-client performance, and so may be considered under those categories. Andreas also pointed out that "Dynamic LNET configuration" was also rated highly. If taken as a whole, we have three LNET features that people have rated highly. Instead of cramming LNET into Availability, let's keep them separate and make four recommendations to the board: availability, performance, storage management and LNET. Remember that we are making a recommendation to the Board. They will decide our budget and tell us the priority for the RFPs that we prepare. --jc -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Nathan Rutman Sent: Tuesday, May 29, 2012 4:59 PM To: twg at lists.opensfs.org Subject: Re: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations My apologies as well. I added a paragraph at the bottom of the Draft http://goo.gl/7ncPT we can discuss on Thurs. Also for completeness we should talk about this: 1) Architecture/Developer documentation Well documented internals of MDS, MGS, OSS, OST, OSD API, etc. On May 29, 2012, at 3:01 PM, Andreas Dilger wrote: > On 2012-05-29, at 1:45 PM, David Dillow wrote: >> We discussed the recommendation to the board as well as testing and >> release management topics during today's additional TWG meeting. >> We will be incorporating the recommendation into the requirements >> document as we did last year. We will note the community's interest in >> improving the test infrastructure as maintaining patch lists for bug fix >> releases, but will refer those topics to the CWG. >> >> We will be sending out updated documents later this evening. >> >> Due to the poor attendance for today's call, we have extended the >> schedule slightly; we will be forwarding the recommendation to the board >> shortly after Thursday's meeting. If you have wish to have your comments >> heard, please make every effort to attend this week's TWG meeting. > > Apologies on missing the meeting. I didn't have anything in my calendar, and I guess I thought there would be a meeting invitation sent out. I'll see if I have any further feedback on the updated doc. > >> Thursday, 12:30pm ET/9:30 PT. >> 715-726-4994 or toll-free 866-304-8294 >> Meeting ID and password: 72090 >> >> >> Current draft recommendation based on our review of the initial >> spreadsheet (to be revised and redistributed this evening): >> https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 >> >> Gathered requirements: http://goo.gl/63u9Q >> The spreadsheet summarizing our priorities: http://goo.gl/Lqg7s >> >> Thanks, >> John Carrier >> Dave Dillow >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From carrier at cray.com Thu May 31 06:48:20 2012 From: carrier at cray.com (John Carrier) Date: Thu, 31 May 2012 06:48:20 +0000 Subject: [Twg] TWG meeting 5/31/2012: last chance for input on recommendations In-Reply-To: <1338320702.2398.39.camel@obelisk.thedillows.org> References: <1338320702.2398.39.camel@obelisk.thedillows.org> Message-ID: I have updated the requirements document for discussion at our meeting on 5/31 with our recommendation to the board as well as the new requirements from DDN (small i/o performance, internals documentation): https://docs.google.com/document/d/1gXs0VPaCNEU3l9OuXYmBCyxKEm3dkIV60smCVAD5Un8/edit#heading=h.n0c122ds7q9l Dave and I are editors. Please add comments to the doc or send them to the reflector. Meeting reminder: Thursday, 12:30pm ET/9:30 PT. 715-726-4994 or toll-free 866-304-8294 Meeting ID and password: 72090 Thanks, --jc -----Original Message----- From: discuss-bounces at lists.opensfs.org [mailto:discuss-bounces at lists.opensfs.org] On Behalf Of David Dillow Sent: Tuesday, May 29, 2012 12:45 PM To: twg at lists.opensfs.org; discuss at lists.opensfs.org Subject: [Discuss] TWG meeting 5/31/2012: last chance for input on recommendations We discussed the recommendation to the board as well as testing and release management topics during today's additional TWG meeting. We will be incorporating the recommendation into the requirements document as we did last year. We will note the community's interest in improving the test infrastructure as maintaining patch lists for bug fix releases, but will refer those topics to the CWG. We will be sending out updated documents later this evening. Due to the poor attendance for today's call, we have extended the schedule slightly; we will be forwarding the recommendation to the board shortly after Thursday's meeting. If you have wish to have your comments heard, please make every effort to attend this week's TWG meeting. Thursday, 12:30pm ET/9:30 PT. 715-726-4994 or toll-free 866-304-8294 Meeting ID and password: 72090 Current draft recommendation based on our review of the initial spreadsheet (to be revised and redistributed this evening): https://docs.google.com/document/d/1zwGkID8yENgdmmp9lsPgg9CInJXVbZoIfTjglKEB4RE/edit?pli=1 Gathered requirements: http://goo.gl/63u9Q The spreadsheet summarizing our priorities: http://goo.gl/Lqg7s Thanks, John Carrier Dave Dillow