[Twg] [Lustre-devel] your opinion about testing

[Chris]

Nathan,

>
> discussion: Xyratex started by suggesting that the test suite included 
> with every Lustre release needed reworking. This led to a group 
> conversation that focused on the following areas: 1) refactor unused 
> tests Test scripts have grown unwieldly and do not form a coherent 
> test package. As a result, there are some tests that are used 
> regularly and others that ineffective and skipped. We should remove 
> the unused tests.
>> Removing tests that are genuinely ineffective is a good idea, but we 
>> must be careful because historically tests have been skipped for many 
>> reasons including the non-function of a useful test, or even the fact 
>> that Lustre could not be made to pass!  I think this refactor process 
>> should be framed more as an examination of currently skipped steps to 
>> understand which should be fixed+re-enabled and which should be removed.
>>
>> It's also worth noting that the ratio of 'used regularly' to skipped 
>> is probably 50:1, skipped tests have no runtime cost and so the ROI 
>> is probably not significant. In fact the ROI is probably only going 
>> to be worthwhile if skipped tests can be made useful and re-enabled.
> There's also a component here of removing redundant tests -- Robert 
> Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs 
> SLOW=YES as virtually identical, but YES takes hours to run.
I've not seen Robert's code coverage figures, did he measure static code 
coverage and dynamic behaviours, because I would hope and certainly not 
presume otherwise that SLOW=YES exercises the dynamic behaviours and 
states much more thoroughly.

I would also suggest that focusing on increasing test coverage and 
quality will create greater return then removing redundant tests, 
although if someone wanted to spend time seeking out the redundant tests 
it would not do any harm.
>
>>
>>>     2) create single test suite
>>>        Each release has its own test suite and assumes that the client
>>>        and server will use the same test version.  It is becoming common
>>>        today to have different client/server versions.  In this case, it
>>>        is unclear which tests from which release package should be used.
>>>        Instead, we should have a single test suite that accommodates
>>>        different combinations of client and server versions.
>> What is the problem statement here that leads to the conclusion that 
>> a single test suite should be produced?
> The test suite from 1.8.7 is not the same as from 2.2.  Therefore, 
> when running the tests on a 1.8.7 client talking to a 2.2 server, 
> which versions' test suite should be run?  An independent test RPM, 
> divorced from the Lustre versioning, would provide a definitive 
> answer.  Yes, I know the tests change with code changes; they should 
> change in the "master" test repo, and a mechanism should be 
> implemented to skip known failures with older versions of non-interop 
> (e.g. 1.8.7-1.8.7) and interop cases.  This mechanism should probably 
> involve pairs of server-client versions and simple lists of tests.
> This also solves the problem of having to backport each test to the 
> old test suite as new tests are developed.
>
In your example you would use 1.8.7 test scripts on the client, and 2.2 
test scripts on the server. When the client makes an RPC to the server 
script it is making a functional request, the receiving script then 
performs that function in a way applicable to the server version in 
question.

Whilst keeping the scripts separate does require back porting it also 
removes the need for the already unwieldy scripts to contain code to 
make them version aware. I would say with some certainty that separate 
scripts with explicit back-porting achieves a more reliable result than 
attempting to extend the bash scripts to cover multiple versions.

Also I don't think we could merge 47000 lines of bash, and then verify 
results. Verification would require that we see each test pass and fail. 
We would also be unable to remove and verify the 1.8 parts of the new 
common test-framework once 1.8 becomes an unsupported product, which 
will happen one day.

Don't get me wrong, if the test-framework was written in a modern, 
structured and object orientated language a single validation and 
verification framework might be possible, but today we don't have that.


>>
>> It can't be interoperability because the suites do run together ( 
>> b1_8 with b2_x 
>> <https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter> 
>> ) and where interoperability issues exist tickets are raised and the 
>> issues resolved. It's worth noting that the tests behind the link do 
>> not perform significantly worse than a straight master test cycle and 
>> that most of the issues are Lustre not test-interoperability.
>>
>> Perhaps a complete problem statement will allow us to understand what 
>> the ROI would be from a single test suite.
>>>     3) extract individual tests from the scripts
>>>        Many of the tests are run as a suite.  There needs to be some
>>>        mechanism to allow running components of the suite independently.
>>>        To do this, we will need to make clear the dependencies of the
>>>        subtests so that users know a test can be run only after another
>>>        component of the test has executed.  Also mentioned was the need
>>>        for a common header on all test results to facilitate
>>>        post-processing.
>> What is to be achieved here that ONLY/EXCEPT does not allow, using a 
>> mixture of ONLY and EXCEPT any combination of tests can be run.
> That would be the ideal.  Unfortunately, it simply is not the case 
> today.  Plenty of tests depend utterly on the tests before them doing 
> some kind of setup or "first half" of a test.  Definition of what 
> constitutes a "test" and a "sub-test" muddies the issue somewhat.
I think there is confusion here about dependencies and extracting the 
tests. If we extracted the tests the dependencies would still exist, and 
if we understood and solved the dependencies of each test then 
ONLY/EXCEPT would allow us to do exactly what was needed.

So we are in agreement that we need to understand and document using 
meta-data the dependencies, but I don't really understand what 
extracting the tests from the scripts really buys us. I don't think it 
does harm, if the tools are put in place to allow developers to still 
run 'suites', but it certainly doesn't solve the dependency issue either.
>
>>
>> On the topic of meta data I think placing machine readable extensible 
>> data in a comments section above every test to store this data would 
>> make a lot of sense. We probably need nothing more than a set of 
>> key:value pairings. If people have seen good working solutions that 
>> should be considered it would be useful if they posted them.
> Yes, I'm a big proponent of YAML.  Beyond this keyword descriptions of 
> the test would allow automatic testing of specific areas, "acl" or 
> "fopen" or "quotas" etc.
I'll try an propose something to this list.
> Obviously this is not as bold as Chris' statement, which, by the way, 
> I am happy to entertain as well. Do others have anything to add to the 
> list above, or thoughts on proceeding with a completely new architecture?
>> My statement was not bold it was just a statement of fact. The 
>> current test-framework is not a suitable platform for fulfilling the 
>> requirements of tomorrow. We have to be building for the next 10 
>> years which will include exascale, does anybody really feel that the 
>> current bash based, un-scalable and relatively unstructured test 
>> facility that we have today is that platform?
> I think not.  The discussion here is a starting point specifically for 
> identifying the current problems, with the hope of driving a an 
> evolutionary way of getting there.  Perhaps via OpenSFS we can drive 
> this as a prioritized task of broad interest.
>
We can all agree that we don't have the luxury of a big-bang change over 
to a new method, but evolution in this case needs to be a new species 
that will overtime become dominant and eventually replace the current 
bashous-thalensis!

Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120413/60bf2dcc/attachment.html>