[Twg] [Lustre-devel] your opinion about testing

[Chris]

Hi,
> On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would
> like to address.
>
> discussion:
>     Xyratex started by suggesting that the test suite included with every
>     Lustre release needed reworking.  This led to a group conversation
>     that focused on the following areas:
>
>     1) refactor unused tests
>        Test scripts have grown unwieldly and do not form a coherent
>        test package.  As a result, there are some tests that are
>        used regularly and others that ineffective and skipped.  We
>        should remove the unused tests.
Removing tests that are genuinely ineffective is a good idea, but we 
must be careful because historically tests have been skipped for many 
reasons including the non-function of a useful test, or even the fact 
that Lustre could not be made to pass!  I think this refactor process 
should be framed more as an examination of currently skipped steps to 
understand which should be fixed+re-enabled and which should be removed.

It's also worth noting that the ratio of 'used regularly' to skipped is 
probably 50:1, skipped tests have no runtime cost and so the ROI is 
probably not significant. In fact the ROI is probably only going to be 
worthwhile if skipped tests can be made useful and re-enabled.

>     2) create single test suite
>        Each release has its own test suite and assumes that the client
>        and server will use the same test version.  It is becoming common
>        today to have different client/server versions.  In this case, it
>        is unclear which tests from which release package should be used.
>        Instead, we should have a single test suite that accommodates
>        different combinations of client and server versions.
What is the problem statement here that leads to the conclusion that a 
single test suite should be produced?

It can't be interoperability because the suites do run together ( b1_8 
with b2_x 
<https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter> 
) and where interoperability issues exist tickets are raised and the 
issues resolved. It's worth noting that the tests behind the link do not 
perform significantly worse than a straight master test cycle and that 
most of the issues are Lustre not test-interoperability.

Perhaps a complete problem statement will allow us to understand what 
the ROI would be from a single test suite.
>     3) extract individual tests from the scripts
>        Many of the tests are run as a suite.  There needs to be some
>        mechanism to allow running components of the suite independently.
>        To do this, we will need to make clear the dependencies of the
>        subtests so that users know a test can be run only after another
>        component of the test has executed.  Also mentioned was the need
>        for a common header on all test results to facilitate
>        post-processing.
What is to be achieved here that ONLY/EXCEPT does not allow, using a 
mixture of ONLY and EXCEPT any combination of tests can be run.

On the topic of meta data I think placing machine readable extensible 
data in a comments section above every test to store this data would 
make a lot of sense. We probably need nothing more than a set of 
key:value pairings. If people have seen good working solutions that 
should be considered it would be useful if they posted them.
>     4) client failure shouldn't stop the tests
>        The goal is never to have test failures, nonethelees failures
>        currently can stall test progress.  The suites need to be written
>        to allow for client failures.
Can someone elaborate on what 'client failure' means, perhaps it is 
obvious but from my experience test failures do not stop the test suites.

Can I also say that the goal is not to never have test failures, because 
if that were true we would just have no tests. The goal is to have a set 
of tests that detect and flag every possible short coming in the 
operation of Lustre, this goal would then mean that Lustre would need to 
be perfect to pass test.
>     5) increase code coverage
>        There are new tests that we should consider adding to the Lustre
>        suite in order to increase code coverage. Suggestions were xfstest
>        and some of the MPI tests.
Before we can increase code coverage we need to systematically 
understand what coverage we have today, this might well be aided by 
creating the appropriate metadata in 3 (above). Once we have a 
measurement we can objectively plan an increase of the coverage, and 
then measure our effectiveness in the future.
> Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well.  Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture?
>
My statement was not bold it was just a statement of fact. The current 
test-framework is not a suitable platform for fulfilling the 
requirements of tomorrow. We have to be building for the next 10 years 
which will include exascale, does anybody really feel that the current 
bash based, un-scalable and relatively unstructured test facility that 
we have today is that platform?

Thanks

Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120412/4246276e/attachment.html>