[Twg] twg Digest, Vol 19, Issue 11

[Chris]

> Hi there,
>
> Let me raise the question about Lustre tests separated execution. We've
> discussed this problem already, but I'd like to clear up some details.
>
> Usually we run all tests sequentially, but in the automation tool we are
> using we need to run tests separately, with ONLY parameter. This allows
> us to have full control over the test execution: terminate hung tests by
> timeout or restore environment in case of the file system damage. At the
> moment some of tests are designed to be run sequentially.
>
> By our estimation, there are about 30 tests need for the  improvement.
> If we settle this question, we can significantly improve test-framework
> automation potential.
> Please share your opinions about this question and help to make a
> decision about it.
> The questions are
>
>   1. Do we want to have an ability to run each test independently?
At first I thought this was a good target but actually after some 
thought I think quite the opposite is true.

Having each discrete step a separate test is very useful, for example.

test 1: create a file
test 2: rename the file
test 3: chmod the file
test 4: test file can't be deleted
test 5: copy the file to file2
test 6: compare file to file2
test 7: chmod the file and file 2
test 8: test file and file 2 can be deleted

These tests are linked and it is useful to know which failed, but you 
can't start anywhere other than test1 and proceed to test 8. Having 
specific setup would be artificial, and having a requirement that they 
could not be link would cause people to group discrete tests into bigger 
tests.
>   2. What is more acceptable - unite sequential tests into complex ones
>      or supplement exists test with additional code steps?
>
I release don't think we should make tests being atomic a requirement 
for the above, and because it will lead to test becoming more complex, 
or the rather strange situation where test 2 (above) calls test 1 
silently if it was not already run, (test 3 calls test 2....) which I 
think is what you are suggesting in your example.

I think that both the solutions in your mail add complexity and remove 
readability from the test environment and when a test fails you do not 
know what actually failed in a systematic way.  I don't really think 
that merging tests together or having each test silently do all the 
functionally of the previous tests is adding value.

A piece of information that might help us come up with the right 
solution would be the driving force behind being able to run individual 
tests? Why do you want to be able to do it.

Chris