[Twg] [Lustre-devel] your opinion about testing

[Nathan Rutman]

On Apr 12, 2012, at 6:33 AM, Chris wrote:


	Hi,
	

		On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would
		like to address.
		
		discussion:
		   Xyratex started by suggesting that the test suite included with every
		   Lustre release needed reworking.  This led to a group conversation
		   that focused on the following areas:
		   
		   1) refactor unused tests
		      Test scripts have grown unwieldly and do not form a coherent
		      test package.  As a result, there are some tests that are
		      used regularly and others that ineffective and skipped.  We
		      should remove the unused tests.

	Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass!  I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed.
	
	It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled.
	

There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run.



		   2) create single test suite
		      Each release has its own test suite and assumes that the client
		      and server will use the same test version.  It is becoming common
		      today to have different client/server versions.  In this case, it
		      is unclear which tests from which release package should be used.
		      Instead, we should have a single test suite that accommodates
		      different combinations of client and server versions.  

	What is the problem statement here that leads to the conclusion that a single test suite should be produced?
	

The test suite from 1.8.7 is not the same as from 2.2.  Therefore, when running the tests on a 1.8.7 client talking to a 2.2 server, which versions' test suite should be run?  An independent test RPM, divorced from the Lustre versioning, would provide a definitive answer.  Yes, I know the tests change with code changes; they should change in the "master" test repo, and a mechanism should be implemented to skip known failures with older versions of non-interop (e.g. 1.8.7-1.8.7) and interop cases.  This mechanism should probably involve pairs of server-client versions and simple lists of tests.
This also solves the problem of having to backport each test to the old test suite as new tests are developed.  



	It can't be interoperability because the suites do run together ( b1_8 with b2_x <https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter>  ) and where interoperability issues exist tickets are raised and the issues resolved. It's worth noting that the tests behind the link do not perform significantly worse than a straight master test cycle and that most of the issues are Lustre not test-interoperability.
	
	Perhaps a complete problem statement will allow us to understand what the ROI would be from a single test suite.
	

		   3) extract individual tests from the scripts
		      Many of the tests are run as a suite.  There needs to be some
		      mechanism to allow running components of the suite independently.
		      To do this, we will need to make clear the dependencies of the
		      subtests so that users know a test can be run only after another
		      component of the test has executed.  Also mentioned was the need
		      for a common header on all test results to facilitate
		      post-processing.

	What is to be achieved here that ONLY/EXCEPT does not allow, using a mixture of ONLY and EXCEPT any combination of tests can be run.
	

That would be the ideal.  Unfortunately, it simply is not the case today.  Plenty of tests depend utterly on the tests before them doing some kind of setup or "first half" of a test.  Definition of what constitutes a "test" and a "sub-test" muddies the issue somewhat.



	On the topic of meta data I think placing machine readable extensible data in a comments section above every test to store this data would make a lot of sense. We probably need nothing more than a set of key:value pairings. If people have seen good working solutions that should be considered it would be useful if they posted them.
	

Yes, I'm a big proponent of YAML.  Beyond this keyword descriptions of the test would allow automatic testing of specific areas, "acl" or "fopen" or "quotas" etc.


		   4) client failure shouldn't stop the tests
		      The goal is never to have test failures, nonethelees failures
		      currently can stall test progress.  The suites need to be written
		      to allow for client failures.        

	Can someone elaborate on what 'client failure' means, perhaps it is obvious but from my experience test failures do not stop the test suites.
	

Some tests hang or crash the client machine running the test script.  An external mechanism is needed in that case to power cycle the machine and restart the tests.  Architectural redesign of the framework deployment should recognize and avoid this problem.


	Can I also say that the goal is not to never have test failures, because if that were true we would just have no tests.

Not test failure, client failure as above.


	The goal is to have a set of tests that detect and flag every possible short coming in the operation of Lustre, this goal would then mean that Lustre would need to be perfect to pass test.
	

		   5) increase code coverage
		      There are new tests that we should consider adding to the Lustre
		      suite in order to increase code coverage. Suggestions were xfstest
		      and some of the MPI tests. 

	Before we can increase code coverage we need to systematically understand what coverage we have today, this might well be aided by creating the appropriate metadata in 3 (above). Once we have a measurement we can objectively plan an increase of the coverage, and then measure our effectiveness in the future.
	

Excellent point.


		Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well.  Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture?
		

	My statement was not bold it was just a statement of fact. The current test-framework is not a suitable platform for fulfilling the requirements of tomorrow. We have to be building for the next 10 years which will include exascale, does anybody really feel that the current bash based, un-scalable and relatively unstructured test facility that we have today is that platform?

I think not.  The discussion here is a starting point specifically for identifying the current problems, with the hope of driving a an evolutionary way of getting there.  Perhaps via OpenSFS we can drive this as a prioritized task of broad interest.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120412/374e39a6/attachment.html>