[Twg] stand alone testing

[Chris]

On 15/05/2012 19:31, Roman Grigoryev wrote:
> On 05/15/2012 09:02 PM, Chris wrote:
>> On 15/05/2012 17:16, Bruce Korb wrote:
>>> On 5/15/12 9:02 AM, Chris wrote:
>>>>>     1. Do we want to have an ability to run each test independently?
>>> Yes.
> I think, it should very important for developers and for tests failure
> evaluators, because they often run limited set of tests.
I absolutely agree that a developer needs to be able to specify just the 
test that he or she wants, but what is the disadvantage if the test 
framework / system executes the required prerequisites before hand. The 
whole test cycle takes no longer (the prerequisite steps have to occur 
one way or the other), the results provide better granularity in the 
event of a prerequisite failing and the tests scripts are almost 
certainly simpler and easy to read.

I would also suggest that looking at the history of the scripts people 
won't use libraries to ensure that setup / clear-up code only executes 
once what will happen is that setup/clearup will get executed for every 
test and so execution of groups of tests will take longer.

My example below illustrates how we could simple provide pre-requisite 
test support using what is effectively simple inheritance, very simply 
and very readably and in a way that makes the desired outcome the likely 
outcome.  This approach means that when a change is required it only 
needs to be made once and works well within the standards that we've 
actually used to right test scripts for the last 10 years not the 
standards we wish we had used.

What is the disadvantage of this simple explicit (light inheritance) 
example below, expanding the framework to never call a test twice would 
be a no-brainer.

test_1a {
   create a file
}
run_test 1a

test_1b {
   run_test 1a    (only does something if test_1a has not already been 
called)
   rename the file
}
run_test 1b

test_1c {
   run_test 1b    (only does something if test_1b has not already been 
called)
   chmod the file
}
run_test 1c

There is no duplication of code and duplicating code would be harder 
work than not duplicating it, no duplication of execution and it is very 
easy to read. We could get the framework to do it automatically, i.e. if 
test_1c was called without 1b having been called it could call 1b... but 
that would make the code less clear to the reader.

>
>>> This does not prevent any of these phases from being in a library
>>> for use by any test, but it would mean that a given test cannot
>>> presuppose that some other test had been run before it.
>> One problem with this approach and looking at the history of the current
>> bash code is that it will lead to duplication via copy and paste because
>> people won't write libraries, bash does not lend itself to this and so
>> solutions will be messy.
> Duplicating in tests is not so big problem, i think. As we  discussed
> previously, this is disciple.

I am not sure if you are saying 'it is not a problem if tests are 
duplicated' in which case I will absolutely say the duplication of code 
is always a problem and always leads to unmaintainable systems - a bit 
like our test scripts really.

But maybe you are saying that 'it is not a problem because all we need 
is discipline', which I would agree with but we have historically not 
had that discipline in the test scripts and I'm not going to bet on that 
changing any-time soon. So I would rather use methods that strongly 
encourage that right thing to happen, for example an approach to 
pre-requisites that makes the easiest method the right method, 
encourages reuse and discourages duplication etc.

Regards to all

Chris