[Twg] stand alone testing

[Roman Grigoryev]

On 05/15/2012 09:02 PM, Chris wrote:
> On 15/05/2012 17:16, Bruce Korb wrote:
>> On 5/15/12 9:02 AM, Chris wrote:
>>>>    1. Do we want to have an ability to run each test independently?
>> Yes.

I think, it should very important for developers and for tests failure
evaluators, because they often run limited set of tests.

>>
>>> At first I thought this was a good target but actually after some
>>> thought I think quite the opposite is true.
>>>
>>> Having each discrete step a separate test is very useful, for example.
>>>
>>> test 1: create a file
>>> test 2: rename the file
>>> test 3: chmod the file
>>> test 4: test file can't be deleted
>>> test 5: copy the file to file2
>>> test 6: compare file to file2
>>> test 7: chmod the file and file 2
>>> test 8: test file and file 2 can be deleted
>>>
>>> These tests are linked
>> Alternatively, these tests are one.  A test to verify "chmod"
>> on a renamed file would be in several phases:
>>     phase 1) create a file || die cannot create
>>     phase 2) rename it || die cannot rename
>>     phase 3) chmod it || die cannot chmod
>>     cleanup) remove file || report cleanup failure
> The problem with this is that you need a machine digestable method of
> reporting back the failure. This test failing can be because of 4
> things, so 4 failures might mean 4 different things. With the model
> above you would know exactly what had failed, and can run earlier tests
> separately.

Just debug messages should be enough, i think. For example, if I asked
framework to execute testsC (which failed) and framework would start
executing testA, it is not straight behavior.

If we use your logic, evaluator must understand that steps are dependent
and see messages from all previous steps(because error may be happens on
previous step and only indicated on current step).

In general, you are suggest to use  one big tests instead of set of
tests, which could be only evaluated as one entity. Every separated part
stop be a test and became other entity, .f.e subtest, test step, test case.

> 
>> This does not prevent any of these phases from being in a library
>> for use by any test, but it would mean that a given test cannot
>> presuppose that some other test had been run before it.
> One problem with this approach and looking at the history of the current
> bash code is that it will lead to duplication via copy and paste because
> people won't write libraries, bash does not lend itself to this and so
> solutions will be messy.

Duplicating in tests is not so big problem, i think. As we  discussed
previously, this is disciple.

Thanks,
	Roman

> 
>>
>> In order to reduce redundant testing, as an example you would not
>> want to regularly run the "create a file" stand alone test, given
>> that few other tests would go far if that feature were broken.
>> But there is too much confusion if every separately identifiable
>> test does not stand on its own.
> Does this not mean that instead of requiring to know about dependency we
> now require an overlap list so that we don't duplicate the work.
> 
> We could expand framework to never call a test twice (this would be
> pretty easy) then we could have.
> 
> test_1 {
>   create a file
> }
> run_test 1
> 
> test_2 {
>   run_test 1    (only does something if test_1 has not already been called)
>   rename the file
> }
> run_test 2
> 
> 
> test_3 {
>   run_test 2    (only does something if test_2 has not already been called)
>   chmod the file
> }
> run_test 3
> 
> This way no duplication of code, no duplication of execution and very
> easy to read and
> 
> 
> Chris
> 
> _______________________________________________
> twg mailing list
> twg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org