[Twg] Separated test execution

[Nicholas Henke]

On 05/14/2012 03:59 PM, Alexander Lezhoev wrote:
> Hi there,
>

..snipped...
>
> They cannot be run separately, because the next index uses the code of 
> previous one. This means all tests should be run in groups of letter 
> indexes, or they should be refactored to run independently.

Is there really a hard 'style' that has the letter indices meaning they 
depend on previous tests ? It seem to vary by test, including where a 
test 6d is just the 'Dth' style of running test6.
>
> Some of tests have been already refactored to run "letters" 
> separately, but we have to make a rule which we should follow and use 
> for further refactoring.
> There are three decisions we can take about this situation
>
>   * Join the code of all test steps into single test with
>     corresponding number. So we will have one test_1 instead of
>     test_1a .. test_1d in the described case.
>

It would certainly be much cleaner to have a single test contain all of 
the steps it needs. It is then repeatable and includes all of the setup 
and cleanup required to complete the test.
>
>
> The first variant could be implemented as follows.
>
> test_1() {
>
> touch $DIR1/f1
>
> [ -f $DIR2/f1 ] || error "check create on 2 mtpt's failed"
>
> chmod 777 $DIR2/f1
>
> $CHECKSTAT -t file -p 0777 $DIR1/f1 || error "check attribute updates 
> on 2 mtpt's failed"
>
> chmod a-x $DIR2/f1
>
> $CHECKSTAT -t file -p 0666 $DIR1/f1 || error "check after remount 
> attribute updates on 2 mtpt's failed"
>
> rm $DIR2/f1
>
> $CHECKSTAT -a $DIR1/f1 || error "unlink on one mountpoint removes file 
> on other failed"
>
> }
> run_test 1 "check attributes updates on 2 mtpt's"
>
> This approach has disadvantage that such kind of refactoring will lead 
> to reduction of test numbering and it will hard to work with 
> regression history of the refactored tests.
> The second case of refactoring can look like this:
>
I'm not sure this is a huge negative for me. The test is very clear and 
it gives us precise errors on each step -- something we should make sure 
is necessary.

As we rework these tests, we'll be changing them enough that we'll 
effectively wipe some of the regression history. Perhaps it is enough to 
change the regression history to just look at test numbering, ignoring 
letters ?


...snipped...
> We can separate these tests:
>
> test_1() {
> test_1_mcreate
> }
> run_test 1 "mcreate: drop req, drop rep"
>
> test_2() {
> test_1_mcreate
> test_2_chmod
> }
> run_test 2 "chmod: drop req, drop rep"
>
> test_3() {
> test_1_mcreate
> test_3_stat
> }
> run_test 3 "stat: drop req, drop rep"

I think this makes it much more complex and harder to code, not to 
mention adds overhead. Each instance of 'run_test' invokes a host of 
setup & cleanup code that can add significantly to the test runtime.

>
> or join them into one and remove test_2 and test_3.
>
> test_1() {
>    drop_request "mcreate $DIR/f1"  || return 1
>    drop_reint_reply "mcreate $DIR/f2"    || return 2
>    drop_request "tchmod 111 $DIR/f2"  || return 3
>    drop_reint_reply "tchmod 666 $DIR/f2"    || return 4
>    drop_request "statone $DIR/f2" || return 5
>    drop_reply "statone $DIR/f2"   || return 6
> }
> run_test 1 "mcreate, chmod,stat: drop req, drop,req"
>

The return codes are not typically enough to distinguish errors from a 
test log. The style above with 'error "MSG with values $x $y"' seems 
more rich and helpful.

Cheers,
Nic

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120515/6e6cbab5/attachment.html>