<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Nathan,<br>
<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com"
type="cite"><br>
<div>
discussion: Xyratex started by suggesting that the test suite
included with every Lustre release needed reworking. This led to
a group conversation that focused on the following areas: 1)
refactor unused tests Test scripts have grown unwieldly and do
not form a coherent test package. As a result, there are some
tests that are used regularly and others that ineffective and
skipped. We should remove the unused tests.
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite"> </blockquote>
Removing tests that are genuinely ineffective is a good
idea, but we must be careful because historically tests have
been skipped for many reasons including the non-function of
a useful test, or even the fact that Lustre could not be
made to pass! I think this refactor process should be
framed more as an examination of currently skipped steps to
understand which should be fixed+re-enabled and which should
be removed.<br>
<br>
It's also worth noting that the ratio of 'used regularly' to
skipped is probably 50:1, skipped tests have no runtime cost
and so the ROI is probably not significant. In fact the ROI
is probably only going to be worthwhile if skipped tests can
be made useful and re-enabled.<br>
</div>
</blockquote>
There's also a component here of removing redundant tests --
Robert Read a few years ago a LUG demonstrated code coverage of
SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours
to run.</div>
</blockquote>
I've not seen Robert's code coverage figures, did he measure static
code coverage and dynamic behaviours, because I would hope and
certainly not presume otherwise that SLOW=YES exercises the dynamic
behaviours and states much more thoroughly.<br>
<br>
I would also suggest that focusing on increasing test coverage and
quality will create greater return then removing redundant tests,
although if someone wanted to spend time seeking out the redundant
tests it would not do any harm.<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div><br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 2) create single test suite
Each release has its own test suite and assumes that the client
and server will use the same test version. It is becoming common
today to have different client/server versions. In this case, it
is unclear which tests from which release package should be used.
Instead, we should have a single test suite that accommodates
different combinations of client and server versions.
</pre>
</blockquote>
What is the problem statement here that leads to the
conclusion that a single test suite should be produced?<br>
</div>
</blockquote>
The test suite from 1.8.7 is not the same as from 2.2.
Therefore, when running the tests on a 1.8.7 client talking to
a 2.2 server, which versions' test suite should be run? An
independent test RPM, divorced from the Lustre versioning, would
provide a definitive answer. Yes, I know the tests change with
code changes; they should change in the "master" test repo, and
a mechanism should be implemented to skip known failures with
older versions of non-interop (e.g. 1.8.7-1.8.7) and interop
cases. This mechanism should probably involve pairs of
server-client versions and simple lists of tests.</div>
<div>This also solves the problem of having to backport each test
to the old test suite as new tests are developed. </div>
<div><br>
</div>
</blockquote>
In your example you would use 1.8.7 test scripts on the client, and
2.2 test scripts on the server. When the client makes an RPC to the
server script it is making a functional request, the receiving
script then performs that function in a way applicable to the server
version in question.<br>
<br>
Whilst keeping the scripts separate does require back porting it
also removes the need for the already unwieldy scripts to contain
code to make them version aware. I would say with some certainty
that separate scripts with explicit back-porting achieves a more
reliable result than attempting to extend the bash scripts to cover
multiple versions.<br>
<br>
Also I don't think we could merge 47000 lines of bash, and then
verify results. Verification would require that we see each test
pass and fail. We would also be unable to remove and verify the 1.8
parts of the new common test-framework once 1.8 becomes an
unsupported product, which will happen one day.<br>
<br>
Don't get me wrong, if the test-framework was written in a modern,
structured and object orientated language a single validation and
verification framework might be possible, but today we don't have
that.<br>
<br>
<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
It can't be interoperability because the suites do run
together ( <a moz-do-not-send="true"
href="https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter">b1_8
with b2_x</a> ) and where interoperability issues exist
tickets are raised and the issues resolved. It's worth
noting that the tests behind the link do not perform
significantly worse than a straight master test cycle and
that most of the issues are Lustre not
test-interoperability.<br>
<br>
Perhaps a complete problem statement will allow us to
understand what the ROI would be from a single test suite.<br>
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite">
<pre wrap=""> 3) extract individual tests from the scripts
Many of the tests are run as a suite. There needs to be some
mechanism to allow running components of the suite independently.
To do this, we will need to make clear the dependencies of the
subtests so that users know a test can be run only after another
component of the test has executed. Also mentioned was the need
for a common header on all test results to facilitate
post-processing.</pre>
</blockquote>
What is to be achieved here that ONLY/EXCEPT does not allow,
using a mixture of ONLY and EXCEPT any combination of tests
can be run.<br>
</div>
</blockquote>
That would be the ideal. Unfortunately, it simply is not the
case today. Plenty of tests depend utterly on the tests before
them doing some kind of setup or "first half" of a test.
Definition of what constitutes a "test" and a "sub-test"
muddies the issue somewhat.</div>
</blockquote>
I think there is confusion here about dependencies and extracting
the tests. If we extracted the tests the dependencies would still
exist, and if we understood and solved the dependencies of each test
then ONLY/EXCEPT would allow us to do exactly what was needed.<br>
<br>
So we are in agreement that we need to understand and document using
meta-data the dependencies, but I don't really understand what
extracting the tests from the scripts really buys us. I don't think
it does harm, if the tools are put in place to allow developers to
still run 'suites', but it certainly doesn't solve the dependency
issue either.<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com"
type="cite">
<div><br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
On the topic of meta data I think placing machine readable
extensible data in a comments section above every test to
store this data would make a lot of sense. We probably need
nothing more than a set of key:value pairings. If people
have seen good working solutions that should be considered
it would be useful if they posted them.<br>
</div>
</blockquote>
Yes, I'm a big proponent of YAML. Beyond this keyword
descriptions of the test would allow automatic testing of
specific areas, "acl" or "fopen" or "quotas" etc.<br>
</div>
</blockquote>
I'll try an propose something to this list.<br>
<blockquote
cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com"
type="cite">Obviously this is not as bold as Chris' statement,
which, by the way, I am happy to entertain as well. Do others have
anything to add to the list above, or thoughts on proceeding with
a completely new architecture?
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<blockquote
cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org"
type="cite"> </blockquote>
My statement was not bold it was just a statement of fact.
The current test-framework is not a suitable platform for
fulfilling the requirements of tomorrow. We have to be
building for the next 10 years which will include exascale,
does anybody really feel that the current bash based,
un-scalable and relatively unstructured test facility that
we have today is that platform?</div>
</blockquote>
</div>
<div>I think not. The discussion here is a starting point
specifically for identifying the current problems, with the hope
of driving a an evolutionary way of getting there. Perhaps via
OpenSFS we can drive this as a prioritized task of broad
interest.<br>
<br>
</div>
</blockquote>
We can all agree that we don't have the luxury of a big-bang change
over to a new method, but evolution in this case needs to be a new
species that will overtime become dominant and eventually replace
the current bashous-thalensis!<br>
<br>
Chris<br>
<br>
</body>
</html>