[Twg] Separated test execution

[Roman Grigoryev]

On 05/16/2012 03:30 AM, Nathan Rutman wrote:
> 
> On May 15, 2012, at 10:41 AM, Andreas Dilger wrote:
> 
>>>
>>>
>>> We have to include cleanup steps here to make possible to run letter
>>> indexes independently. With that cleanup steps, 200a and 201c became
>>> absolutely equal and need to be reduced. Same situation is for 200h
>>> and 201b.
>>
>> The problem with doing a full setup and cleanup for each subtest is
>> that this can make the test runtime longer than necessary. In previous
>> cases where there was a dependency between subtests, it is better to
>> have the "setup" step do a "check or setup", and if the previous
>> subtest has already run, there is no additional work done for the
>> setup.  Similarly, if every subtest does its own cleanup, but the next
>> test sets everything back up again, it is just a waste of testing time.
> 
> Another big problem is that these two states are not equal.  After
> remounting Lustre does not have the same state (memory structures,
> cache, variable settings) that it has after running a test.  If we only
> ever run tests after remounting, how do we know Lustre can successfully
> run more than one operation in a row?
> 
>>
>> I also don't think each subtest should do its own cleanup, unless the
>> subtest itself could cause problems with other subtests because it
>> significantly fills the OST(s) or the MDT(s).  The cleanup for
>> "normal" tests should be deferred until after the end of all subtests
>> (e.g. the "rm -rf $DIR/[df][0-9]*" for sanity.sh and sanityn.sh).  The
>> pool tests themselves already have a "cleanup_pools" test that is run
>> after all of the pools tests are completed.
> 
> I think tests should clean up after themselves, if they leave any
> persistent fingerprints -- changes in debug level, parameter settings,
> fills space > eg 10%, etc.   That way, tests can be run individually, in
> a random order, repeatedly, etc.  (BTW, I don't think there's a way to
> run a test more than once in a row with current test-framework.)

This feature is very simple to implement via using ONLY parameter and
external script with knowledge of test names.

Thanks,
	Roman