[Twg] Separated test execution

[Nathan Rutman]

On May 15, 2012, at 10:41 AM, Andreas Dilger wrote:


	



		We have to include cleanup steps here to make possible to run letter indexes independently. With that cleanup steps, 200a and 201c became absolutely equal and need to be reduced. Same situation is for 200h and 201b.
		


	The problem with doing a full setup and cleanup for each subtest is that this can make the test runtime longer than necessary. In previous cases where there was a dependency between subtests, it is better to have the "setup" step do a "check or setup", and if the previous subtest has already run, there is no additional work done for the setup.  Similarly, if every subtest does its own cleanup, but the next test sets everything back up again, it is just a waste of testing time.
	


Another big problem is that these two states are not equal.  After remounting Lustre does not have the same state (memory structures, cache, variable settings) that it has after running a test.  If we only ever run tests after remounting, how do we know Lustre can successfully run more than one operation in a row?


	

	I also don't think each subtest should do its own cleanup, unless the subtest itself could cause problems with other subtests because it significantly fills the OST(s) or the MDT(s).  The cleanup for "normal" tests should be deferred until after the end of all subtests (e.g. the "rm -rf $DIR/[df][0-9]*" for sanity.sh and sanityn.sh).  The pool tests themselves already have a "cleanup_pools" test that is run after all of the pools tests are completed.


I think tests should clean up after themselves, if they leave any persistent fingerprints -- changes in debug level, parameter settings, fills space > eg 10%, etc.   That way, tests can be run individually, in a random order, repeatedly, etc.  (BTW, I don't think there's a way to run a test more than once in a row with current test-framework.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120515/5f482a17/attachment.html>