[Twg] [Lustre-devel] your opinion about testing

[Nathan Rutman]

Chris and I talked a little at LUG about the Lustre testing suite, and came up with a some thoughts about how to get out of the current mess:

1. New language / library
Bash has several problems: slow, non-portable, obscure syntax constructions, difficulty doing math, and most importantly it's not function-oriented.  A replacement language that called clearly-defined functions with clear parameters allows for the building of a function library.  Ideally current bash scripts could call the new library functions as we slowly replace the old framework. 

2. Start new tests with new framework.
New tests would be required to use the new framework, thus building up the library.

3. Remote operations via lctl messages.
The tests frequently fail to work on many clusters due to passwordless ssh requirements.  Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent.  A "testing upcall" procedure would be registered on the remote node and would receive the messages.  Obviously this could be a more broadly useful mechanism.  Due to security issues some sites may chose not to register an upcall. A response mechanism is also required.

Roman and I additionally had some more thoughts:

4. Clarify local / remote operations
Often tests are written assuming a particular configuration (eg. llmount.sh) or version, and all tests execute local script data remotely.  These must be rewritten as encapsulated library functions which may execute remote procedure calls.  A library on the remote node would be responsible for executing the procedure.

5. Clarify test independence
Currently many tests need all tests in a particular range to run, in sequence, in order to correctly set up and clean up a particular environment.  The pre-requisites and post-requisites for each test need to be broken out from the tests themselves.

6. Add meta-information to tests
Tags bearing descriptions and other characteristics of the tests should be included with each test to allow for automated test selection.  E.g. label:quota could be used to run all quotas tests.

7. Remove version dependency
Having the Lustre tests tied to the Lustre source makes it difficult to run interoperability tests consistently (e.g. use old or new version of tests?).  Instead, we can separate out the tests into an independent set, and have interoperability matrixes for tests that should or should not be run.  This also eliminates the need to backport tests between Lustre versions.

8. Splitting packages
Lustre, lustre test suite, LNET, and ldiskfs should all be separated into different packages.  Dependencies between package versions should be addressed by normal RPM methods.  This is probably useful for a variety of different projects: zfs,  the lustre-client-in-upstream-kernel, etc.




On Apr 13, 2012, at 6:29 AM, Chris wrote:


	Nathan,
	
	


		discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests. 

			Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass!  I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed.
			
			It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled.
			

		There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run.

	I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly.
	
	I would also suggest that focusing on increasing test coverage and quality will create greater return then removing redundant tests, although if someone wanted to spend time seeking out the redundant tests it would not do any harm.
	



				   2) create single test suite
				      Each release has its own test suite and assumes that the client
				      and server will use the same test version.  It is becoming common
				      today to have different client/server versions.  In this case, it
				      is unclear which tests from which release package should be used.
				      Instead, we should have a single test suite that accommodates
				      different combinations of client and server versions.  

			What is the problem statement here that leads to the conclusion that a single test suite should be produced?
			

		The test suite from 1.8.7 is not the same as from 2.2.  Therefore, when running the tests on a 1.8.7 client talking to a 2.2 server, which versions' test suite should be run?  An independent test RPM, divorced from the Lustre versioning, would provide a definitive answer.  Yes, I know the tests change with code changes; they should change in the "master" test repo, and a mechanism should be implemented to skip known failures with older versions of non-interop (e.g. 1.8.7-1.8.7) and interop cases.  This mechanism should probably involve pairs of server-client versions and simple lists of tests.
		This also solves the problem of having to backport each test to the old test suite as new tests are developed.  


	In your example you would use 1.8.7 test scripts on the client, and 2.2 test scripts on the server. When the client makes an RPC to the server script it is making a functional request, the receiving script then performs that function in a way applicable to the server version in question.
	
	Whilst keeping the scripts separate does require back porting it also removes the need for the already unwieldy scripts to contain code to make them version aware. I would say with some certainty that separate scripts with explicit back-porting achieves a more reliable result than attempting to extend the bash scripts to cover multiple versions.
	
	Also I don't think we could merge 47000 lines of bash, and then verify results. Verification would require that we see each test pass and fail. We would also be unable to remove and verify the 1.8 parts of the new common test-framework once 1.8 becomes an unsupported product, which will happen one day.
	
	Don't get me wrong, if the test-framework was written in a modern, structured and object orientated language a single validation and verification framework might be possible, but today we don't have that.
	
	
	


			It can't be interoperability because the suites do run together ( b1_8 with b2_x <https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter>  ) and where interoperability issues exist tickets are raised and the issues resolved. It's worth noting that the tests behind the link do not perform significantly worse than a straight master test cycle and that most of the issues are Lustre not test-interoperability.
			
			Perhaps a complete problem statement will allow us to understand what the ROI would be from a single test suite.
			

				   3) extract individual tests from the scripts
				      Many of the tests are run as a suite.  There needs to be some
				      mechanism to allow running components of the suite independently.
				      To do this, we will need to make clear the dependencies of the
				      subtests so that users know a test can be run only after another
				      component of the test has executed.  Also mentioned was the need
				      for a common header on all test results to facilitate
				      post-processing.

			What is to be achieved here that ONLY/EXCEPT does not allow, using a mixture of ONLY and EXCEPT any combination of tests can be run.
			

		That would be the ideal.  Unfortunately, it simply is not the case today.  Plenty of tests depend utterly on the tests before them doing some kind of setup or "first half" of a test.  Definition of what constitutes a "test" and a "sub-test" muddies the issue somewhat.

	I think there is confusion here about dependencies and extracting the tests. If we extracted the tests the dependencies would still exist, and if we understood and solved the dependencies of each test then ONLY/EXCEPT would allow us to do exactly what was needed.
	
	So we are in agreement that we need to understand and document using meta-data the dependencies, but I don't really understand what extracting the tests from the scripts really buys us. I don't think it does harm, if the tools are put in place to allow developers to still run 'suites', but it certainly doesn't solve the dependency issue either.
	



			On the topic of meta data I think placing machine readable extensible data in a comments section above every test to store this data would make a lot of sense. We probably need nothing more than a set of key:value pairings. If people have seen good working solutions that should be considered it would be useful if they posted them.
			

		Yes, I'm a big proponent of YAML.  Beyond this keyword descriptions of the test would allow automatic testing of specific areas, "acl" or "fopen" or "quotas" etc.
		

	I'll try an propose something to this list.
	

		Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture? 

			My statement was not bold it was just a statement of fact. The current test-framework is not a suitable platform for fulfilling the requirements of tomorrow. We have to be building for the next 10 years which will include exascale, does anybody really feel that the current bash based, un-scalable and relatively unstructured test facility that we have today is that platform?

		I think not.  The discussion here is a starting point specifically for identifying the current problems, with the hope of driving a an evolutionary way of getting there.  Perhaps via OpenSFS we can drive this as a prioritized task of broad interest.
		
		

	We can all agree that we don't have the luxury of a big-bang change over to a new method, but evolution in this case needs to be a new species that will overtime become dominant and eventually replace the current bashous-thalensis!
	
	Chris
	
	


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120430/b8e5a4a3/attachment.html>