<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap="">On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would
like to address.
discussion:
Xyratex started by suggesting that the test suite included with every
Lustre release needed reworking. This led to a group conversation
that focused on the following areas:
1) refactor unused tests
Test scripts have grown unwieldly and do not form a coherent
test package. As a result, there are some tests that are
used regularly and others that ineffective and skipped. We
should remove the unused tests.</pre>
</blockquote>
Removing tests that are genuinely ineffective is a good idea, but we
must be careful because historically tests have been skipped for
many reasons including the non-function of a useful test, or even
the fact that Lustre could not be made to pass! I think this
refactor process should be framed more as an examination of
currently skipped steps to understand which should be
fixed+re-enabled and which should be removed.<br>
<br>
It's also worth noting that the ratio of 'used regularly' to skipped
is probably 50:1, skipped tests have no runtime cost and so the ROI
is probably not significant. In fact the ROI is probably only going
to be worthwhile if skipped tests can be made useful and re-enabled.<br>
<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 2) create single test suite
Each release has its own test suite and assumes that the client
and server will use the same test version. It is becoming common
today to have different client/server versions. In this case, it
is unclear which tests from which release package should be used.
Instead, we should have a single test suite that accommodates
different combinations of client and server versions.
</pre>
</blockquote>
What is the problem statement here that leads to the conclusion that
a single test suite should be produced?<br>
<br>
It can't be interoperability because the suites do run together ( <a
href="https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter">b1_8
with b2_x</a> ) and where interoperability issues exist tickets
are raised and the issues resolved. It's worth noting that the tests
behind the link do not perform significantly worse than a straight
master test cycle and that most of the issues are Lustre not
test-interoperability.<br>
<br>
Perhaps a complete problem statement will allow us to understand
what the ROI would be from a single test suite.<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 3) extract individual tests from the scripts
Many of the tests are run as a suite. There needs to be some
mechanism to allow running components of the suite independently.
To do this, we will need to make clear the dependencies of the
subtests so that users know a test can be run only after another
component of the test has executed. Also mentioned was the need
for a common header on all test results to facilitate
post-processing.</pre>
</blockquote>
What is to be achieved here that ONLY/EXCEPT does not allow, using a
mixture of ONLY and EXCEPT any combination of tests can be run.<br>
<br>
On the topic of meta data I think placing machine readable
extensible data in a comments section above every test to store this
data would make a lot of sense. We probably need nothing more than a
set of key:value pairings. If people have seen good working
solutions that should be considered it would be useful if they
posted them.<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 4) client failure shouldn't stop the tests
The goal is never to have test failures, nonethelees failures
currently can stall test progress. The suites need to be written
to allow for client failures. </pre>
</blockquote>
Can someone elaborate on what 'client failure' means, perhaps it is
obvious but from my experience test failures do not stop the test
suites.<br>
<br>
Can I also say that the goal is not to never have test failures,
because if that were true we would just have no tests. The goal is
to have a set of tests that detect and flag every possible short
coming in the operation of Lustre, this goal would then mean that
Lustre would need to be perfect to pass test.<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 5) increase code coverage
There are new tests that we should consider adding to the Lustre
suite in order to increase code coverage. Suggestions were xfstest
and some of the MPI tests.
</pre>
</blockquote>
Before we can increase code coverage we need to systematically
understand what coverage we have today, this might well be aided by
creating the appropriate metadata in 3 (above). Once we have a
measurement we can objectively plan an increase of the coverage, and
then measure our effectiveness in the future.<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap="">Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture?
</pre>
</blockquote>
My statement was not bold it was just a statement of fact. The
current test-framework is not a suitable platform for fulfilling the
requirements of tomorrow. We have to be building for the next 10
years which will include exascale, does anybody really feel that the
current bash based, un-scalable and relatively unstructured test
facility that we have today is that platform?<br>
<br>
Thanks<br>
<br>
Chris<br>
<br>
</body>
</html>