[Twg] stand alone testing

[Chris]

On 15/05/2012 17:16, Bruce Korb wrote:
> On 5/15/12 9:02 AM, Chris wrote:
>>>    1. Do we want to have an ability to run each test independently?
> Yes.
>
>> At first I thought this was a good target but actually after some
>> thought I think quite the opposite is true.
>>
>> Having each discrete step a separate test is very useful, for example.
>>
>> test 1: create a file
>> test 2: rename the file
>> test 3: chmod the file
>> test 4: test file can't be deleted
>> test 5: copy the file to file2
>> test 6: compare file to file2
>> test 7: chmod the file and file 2
>> test 8: test file and file 2 can be deleted
>>
>> These tests are linked
> Alternatively, these tests are one.  A test to verify "chmod"
> on a renamed file would be in several phases:
>     phase 1) create a file || die cannot create
>     phase 2) rename it || die cannot rename
>     phase 3) chmod it || die cannot chmod
>     cleanup) remove file || report cleanup failure
The problem with this is that you need a machine digestable method of 
reporting back the failure. This test failing can be because of 4 
things, so 4 failures might mean 4 different things. With the model 
above you would know exactly what had failed, and can run earlier tests 
separately.


> This does not prevent any of these phases from being in a library
> for use by any test, but it would mean that a given test cannot
> presuppose that some other test had been run before it.
One problem with this approach and looking at the history of the current 
bash code is that it will lead to duplication via copy and paste because 
people won't write libraries, bash does not lend itself to this and so 
solutions will be messy.

>
> In order to reduce redundant testing, as an example you would not
> want to regularly run the "create a file" stand alone test, given
> that few other tests would go far if that feature were broken.
> But there is too much confusion if every separately identifiable
> test does not stand on its own.
Does this not mean that instead of requiring to know about dependency we 
now require an overlap list so that we don't duplicate the work.

We could expand framework to never call a test twice (this would be 
pretty easy) then we could have.

test_1 {
   create a file
}
run_test 1

test_2 {
   run_test 1    (only does something if test_1 has not already been called)
   rename the file
}
run_test 2


test_3 {
   run_test 2    (only does something if test_2 has not already been called)
   chmod the file
}
run_test 3

This way no duplication of code, no duplication of execution and very easy to read and


Chris