<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Apr 12, 2012, at 6:33 AM, Chris wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap="">On the 4/5 TWG concall we discussed this further, and came up with 5 items that we would
like to address.
discussion:
Xyratex started by suggesting that the test suite included with every
Lustre release needed reworking. This led to a group conversation
that focused on the following areas:
1) refactor unused tests
Test scripts have grown unwieldly and do not form a coherent
test package. As a result, there are some tests that are
used regularly and others that ineffective and skipped. We
should remove the unused tests.</pre>
</blockquote>
Removing tests that are genuinely ineffective is a good idea, but we
must be careful because historically tests have been skipped for
many reasons including the non-function of a useful test, or even
the fact that Lustre could not be made to pass! I think this
refactor process should be framed more as an examination of
currently skipped steps to understand which should be
fixed+re-enabled and which should be removed.<br>
<br>
It's also worth noting that the ratio of 'used regularly' to skipped
is probably 50:1, skipped tests have no runtime cost and so the ROI
is probably not significant. In fact the ROI is probably only going
to be worthwhile if skipped tests can be made useful and re-enabled.<br></div></blockquote>There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run.</div><div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 2) create single test suite
Each release has its own test suite and assumes that the client
and server will use the same test version. It is becoming common
today to have different client/server versions. In this case, it
is unclear which tests from which release package should be used.
Instead, we should have a single test suite that accommodates
different combinations of client and server versions.
</pre>
</blockquote>
What is the problem statement here that leads to the conclusion that
a single test suite should be produced?<br></div></blockquote>The test suite from 1.8.7 is not the same as from 2.2. Therefore, when running the tests on a 1.8.7 client talking to a 2.2 server, which versions' test suite should be run? An independent test RPM, divorced from the Lustre versioning, would provide a definitive answer. Yes, I know the tests change with code changes; they should change in the "master" test repo, and a mechanism should be implemented to skip known failures with older versions of non-interop (e.g. 1.8.7-1.8.7) and interop cases. This mechanism should probably involve pairs of server-client versions and simple lists of tests.</div><div>This also solves the problem of having to backport each test to the old test suite as new tests are developed. </div><div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<br>
It can't be interoperability because the suites do run together ( <a href="https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter">b1_8
with b2_x</a> ) and where interoperability issues exist tickets
are raised and the issues resolved. It's worth noting that the tests
behind the link do not perform significantly worse than a straight
master test cycle and that most of the issues are Lustre not
test-interoperability.<br>
<br>
Perhaps a complete problem statement will allow us to understand
what the ROI would be from a single test suite.<br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 3) extract individual tests from the scripts
Many of the tests are run as a suite. There needs to be some
mechanism to allow running components of the suite independently.
To do this, we will need to make clear the dependencies of the
subtests so that users know a test can be run only after another
component of the test has executed. Also mentioned was the need
for a common header on all test results to facilitate
post-processing.</pre>
</blockquote>
What is to be achieved here that ONLY/EXCEPT does not allow, using a
mixture of ONLY and EXCEPT any combination of tests can be run.<br></div></blockquote>That would be the ideal. Unfortunately, it simply is not the case today. Plenty of tests depend utterly on the tests before them doing some kind of setup or "first half" of a test. Definition of what constitutes a "test" and a "sub-test" muddies the issue somewhat.</div><div><br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<br>
On the topic of meta data I think placing machine readable
extensible data in a comments section above every test to store this
data would make a lot of sense. We probably need nothing more than a
set of key:value pairings. If people have seen good working
solutions that should be considered it would be useful if they
posted them.<br></div></blockquote>Yes, I'm a big proponent of YAML. Beyond this keyword descriptions of the test would allow automatic testing of specific areas, "acl" or "fopen" or "quotas" etc.<br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 4) client failure shouldn't stop the tests
The goal is never to have test failures, nonethelees failures
currently can stall test progress. The suites need to be written
to allow for client failures. </pre>
</blockquote>
Can someone elaborate on what 'client failure' means, perhaps it is
obvious but from my experience test failures do not stop the test
suites.<br></div></blockquote>Some tests hang or crash the client machine running the test script. An external mechanism is needed in that case to power cycle the machine and restart the tests. Architectural redesign of the framework deployment should recognize and avoid this problem.</div><div><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<br>
Can I also say that the goal is not to never have test failures,
because if that were true we would just have no tests.</div></blockquote>Not test failure, client failure as above.<br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"> The goal is
to have a set of tests that detect and flag every possible short
coming in the operation of Lustre, this goal would then mean that
Lustre would need to be perfect to pass test.<br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 5) increase code coverage
There are new tests that we should consider adding to the Lustre
suite in order to increase code coverage. Suggestions were xfstest
and some of the MPI tests.
</pre>
</blockquote>
Before we can increase code coverage we need to systematically
understand what coverage we have today, this might well be aided by
creating the appropriate metadata in 3 (above). Once we have a
measurement we can objectively plan an increase of the coverage, and
then measure our effectiveness in the future.<br></div></blockquote>Excellent point.<br><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap="">Obviously this is not as bold as Chris' statement, which, by the way, I am happy to entertain as well. Do others have anything to add to the list above, or thoughts on proceeding with a completely new architecture?
</pre>
</blockquote>
My statement was not bold it was just a statement of fact. The
current test-framework is not a suitable platform for fulfilling the
requirements of tomorrow. We have to be building for the next 10
years which will include exascale, does anybody really feel that the
current bash based, un-scalable and relatively unstructured test
facility that we have today is that platform?</div></blockquote></div><div>I think not. The discussion here is a starting point specifically for identifying the current problems, with the hope of driving a an evolutionary way of getting there. Perhaps via OpenSFS we can drive this as a prioritized task of broad interest.</div><div><br><div><br><div><br></div></div></div></body></html>