[Twg] Separated test execution

[Roman Grigoryev]

On 05/15/2012 08:15 PM, Nicholas Henke wrote:
> On 05/14/2012 03:59 PM, Alexander Lezhoev wrote:
>> Hi there,
>>
> 
> ..snipped...
>>
>> They cannot be run separately, because the next index uses the code of
>> previous one. This means all tests should be run in groups of letter
>> indexes, or they should be refactored to run independently.
> 
> Is there really a hard 'style' that has the letter indices meaning they
> depend on previous tests ? It seem to vary by test, including where a
> test 6d is just the 'Dth' style of running test6.

No any definition haven't placed in documentation about it till now.
Framework don't do different processing. Every tests creator do it by
his own understanding.

>>
>> Some of tests have been already refactored to run “letters”
>> separately, but we have to make a rule which we should follow and use
>> for further refactoring.
>> There are three decisions we can take about this situation
>>
>>   * Join the code of all test steps into single test with
>>     corresponding number. So we will have one test_1 instead of
>>     test_1a .. test_1d in the described case.
>>
> 
> It would certainly be much cleaner to have a single test contain all of
> the steps it needs. It is then repeatable and includes all of the setup
> and cleanup required to complete the test.

Absolutely agree. I think, we should not afraid existence of some
percent of copy-paste code in tests it they became cleaner.

>>
>>
>> The first variant could be implemented as follows.
>>
>> test_1() {
>>
>> touch $DIR1/f1
>>
>> [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed"
>>
>> chmod 777 $DIR2/f1
>>
>> $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates
>> on 2 mtpt's failed"
>>
>> chmod a-x $DIR2/f1
>>
>> $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount
>> attribute updates on 2 mtpt's failed"
>>
>> rm $DIR2/f1
>>
>> $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file
>> on other failed"
>>
>> }
>> run_test 1 "check attributes updates on 2 mtpt's"
>>
>> This approach has disadvantage that such kind of refactoring will lead
>> to reduction of test numbering and it will hard to work with
>> regression history of the refactored tests.
>> The second case of refactoring can look like this:
>>
> I'm not sure this is a huge negative for me. The test is very clear and
> it gives us precise errors on each step -- something we should make sure
> is necessary.
> 
> As we rework these tests, we'll be changing them enough that we'll
> effectively wipe some of the regression history. Perhaps it is enough to
> change the regression history to just look at test numbering, ignoring
> letters ?

Other possible approach: start to use test id. Now they could be same as
test name and can migrate with test core, include case of merging tests
and merged test could have field which have original tests ids.

Thanks,
	Roman

> 
> 
> ...snipped...
>> We can separate these tests:
>>
>> test_1() {
>>    test_1_mcreate
>> }
>> run_test 1 "mcreate: drop req, drop rep"
>>
>> test_2() {
>>    test_1_mcreate
>>    test_2_chmod
>> }
>> run_test 2 "chmod: drop req, drop rep"
>>
>> test_3() {
>>    test_1_mcreate
>>    test_3_stat
>> }
>> run_test 3 "stat: drop req, drop rep"
> 
> I think this makes it much more complex and harder to code, not to
> mention adds overhead. Each instance of 'run_test' invokes a host of
> setup & cleanup code that can add significantly to the test runtime.
> 
>>
>> or join them into one and remove test_2 and test_3.
>>
>> test_1() {
>>    drop_request "mcreate $DIR/f1"  || return 1
>>    drop_reint_reply "mcreate $DIR/f2"    || return 2
>>    drop_request "tchmod 111 $DIR/f2"  || return 3
>>    drop_reint_reply "tchmod 666 $DIR/f2"    || return 4
>>    drop_request "statone $DIR/f2" || return 5
>>    drop_reply "statone $DIR/f2"   || return 6
>> }
>> run_test 1 "mcreate, chmod,stat: drop req, drop,req"
>>
> 
> The return codes are not typically enough to distinguish errors from a
> test log. The style above with 'error "MSG with values $x $y"' seems
> more rich and helpful.
> 
> Cheers,
> Nic
> 
> 
> 
> _______________________________________________
> twg mailing list
> twg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org