<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Chris and I talked a little at LUG about the Lustre testing suite, and came up with a some thoughts about how to get out of the current mess:<div><br></div><div>1. New language / library</div><div>Bash has several problems: slow, non-portable, obscure syntax constructions, difficulty doing math, and most importantly it's not function-oriented. A replacement language that called clearly-defined functions with clear parameters allows for the building of a function library. Ideally current bash scripts could call the new library functions as we slowly replace the old framework. </div><div><br></div><div>2. Start new tests with new framework.</div><div>New tests would be required to use the new framework, thus building up the library.</div><div><br></div><div>3. Remote operations via lctl messages.</div><div>The tests frequently fail to work on many clusters due to passwordless ssh requirements. Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent. A "testing upcall" procedure would be registered on the remote node and would receive the messages. Obviously this could be a more broadly useful mechanism. Due to security issues some sites may chose not to register an upcall. A response mechanism is also required.</div><div><br></div><div>Roman and I additionally had some more thoughts:</div><div><br></div><div>4. Clarify local / remote operations</div><div><div>Often tests are written assuming a particular configuration (eg. llmount.sh) or version, and all tests execute local script data remotely. These must be rewritten as encapsulated library functions which may execute remote procedure calls. A library on the remote node would be responsible for executing the procedure.</div></div><div><div><br></div></div><div>5. Clarify test independence</div><div>Currently many tests need all tests in a particular range to run, in sequence, in order to correctly set up and clean up a particular environment. The pre-requisites and post-requisites for each test need to be broken out from the tests themselves.</div><div><br></div><div>6. Add meta-information to tests</div><div><div>Tags bearing descriptions and other characteristics of the tests should be included with each test to allow for automated test selection. E.g. label:quota could be used to run all quotas tests.</div><div><br></div><div>7. Remove version dependency</div><div>Having the Lustre tests tied to the Lustre source makes it difficult to run interoperability tests consistently (e.g. use old or new version of tests?). Instead, we can separate out the tests into an independent set, and have interoperability matrixes for tests that should or should not be run. This also eliminates the need to backport tests between Lustre versions.</div><div><br></div><div><div>8. Splitting packages</div><div>Lustre, lustre test suite, LNET, and ldiskfs should all be separated into different packages. Dependencies between package versions should be addressed by normal RPM methods. This is probably useful for a variety of different projects: zfs, the lustre-client-in-upstream-kernel, etc.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><div>On Apr 13, 2012, at 6:29 AM, Chris wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000">
Nathan,<br>
<br>
<blockquote cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com" type="cite"><br>
<div>
discussion: Xyratex started by suggesting that the test suite
included with every Lustre release needed reworking. This led to
a group conversation that focused on the following areas: 1)
refactor unused tests Test scripts have grown unwieldly and do
not form a coherent test package. As a result, there are some
tests that are used regularly and others that ineffective and
skipped. We should remove the unused tests.
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite"> </blockquote>
Removing tests that are genuinely ineffective is a good
idea, but we must be careful because historically tests have
been skipped for many reasons including the non-function of
a useful test, or even the fact that Lustre could not be
made to pass! I think this refactor process should be
framed more as an examination of currently skipped steps to
understand which should be fixed+re-enabled and which should
be removed.<br>
<br>
It's also worth noting that the ratio of 'used regularly' to
skipped is probably 50:1, skipped tests have no runtime cost
and so the ROI is probably not significant. In fact the ROI
is probably only going to be worthwhile if skipped tests can
be made useful and re-enabled.<br>
</div>
</blockquote>
There's also a component here of removing redundant tests --
Robert Read a few years ago a LUG demonstrated code coverage of
SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours
to run.</div>
</blockquote>
I've not seen Robert's code coverage figures, did he measure static
code coverage and dynamic behaviours, because I would hope and
certainly not presume otherwise that SLOW=YES exercises the dynamic
behaviours and states much more thoroughly.<br>
<br>
I would also suggest that focusing on increasing test coverage and
quality will create greater return then removing redundant tests,
although if someone wanted to spend time seeking out the redundant
tests it would not do any harm.<br>
<blockquote cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com" type="cite">
<div><br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 2) create single test suite
Each release has its own test suite and assumes that the client
and server will use the same test version. It is becoming common
today to have different client/server versions. In this case, it
is unclear which tests from which release package should be used.
Instead, we should have a single test suite that accommodates
different combinations of client and server versions.
</pre>
</blockquote>
What is the problem statement here that leads to the
conclusion that a single test suite should be produced?<br>
</div>
</blockquote>
The test suite from 1.8.7 is not the same as from 2.2.
Therefore, when running the tests on a 1.8.7 client talking to
a 2.2 server, which versions' test suite should be run? An
independent test RPM, divorced from the Lustre versioning, would
provide a definitive answer. Yes, I know the tests change with
code changes; they should change in the "master" test repo, and
a mechanism should be implemented to skip known failures with
older versions of non-interop (e.g. 1.8.7-1.8.7) and interop
cases. This mechanism should probably involve pairs of
server-client versions and simple lists of tests.</div>
<div>This also solves the problem of having to backport each test
to the old test suite as new tests are developed. </div>
<div><br>
</div>
</blockquote>
In your example you would use 1.8.7 test scripts on the client, and
2.2 test scripts on the server. When the client makes an RPC to the
server script it is making a functional request, the receiving
script then performs that function in a way applicable to the server
version in question.<br>
<br>
Whilst keeping the scripts separate does require back porting it
also removes the need for the already unwieldy scripts to contain
code to make them version aware. I would say with some certainty
that separate scripts with explicit back-porting achieves a more
reliable result than attempting to extend the bash scripts to cover
multiple versions.<br>
<br>
Also I don't think we could merge 47000 lines of bash, and then
verify results. Verification would require that we see each test
pass and fail. We would also be unable to remove and verify the 1.8
parts of the new common test-framework once 1.8 becomes an
unsupported product, which will happen one day.<br>
<br>
Don't get me wrong, if the test-framework was written in a modern,
structured and object orientated language a single validation and
verification framework might be possible, but today we don't have
that.<br>
<br>
<br>
<blockquote cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com" type="cite">
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
It can't be interoperability because the suites do run
together ( <a moz-do-not-send="true" href="https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&lustre_branch_id%5B%5D=249f16b8-04a9-11e1-bb5f-52540025f9af&lustre_branch_id%5B%5D=24a6947e-04a9-11e1-bb5f-52540025f9af&commit=Apply+Filter">b1_8
with b2_x</a> ) and where interoperability issues exist
tickets are raised and the issues resolved. It's worth
noting that the tests behind the link do not perform
significantly worse than a straight master test cycle and
that most of the issues are Lustre not
test-interoperability.<br>
<br>
Perhaps a complete problem statement will allow us to
understand what the ROI would be from a single test suite.<br>
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite">
<pre wrap=""> 3) extract individual tests from the scripts
Many of the tests are run as a suite. There needs to be some
mechanism to allow running components of the suite independently.
To do this, we will need to make clear the dependencies of the
subtests so that users know a test can be run only after another
component of the test has executed. Also mentioned was the need
for a common header on all test results to facilitate
post-processing.</pre>
</blockquote>
What is to be achieved here that ONLY/EXCEPT does not allow,
using a mixture of ONLY and EXCEPT any combination of tests
can be run.<br>
</div>
</blockquote>
That would be the ideal. Unfortunately, it simply is not the
case today. Plenty of tests depend utterly on the tests before
them doing some kind of setup or "first half" of a test.
Definition of what constitutes a "test" and a "sub-test"
muddies the issue somewhat.</div>
</blockquote>
I think there is confusion here about dependencies and extracting
the tests. If we extracted the tests the dependencies would still
exist, and if we understood and solved the dependencies of each test
then ONLY/EXCEPT would allow us to do exactly what was needed.<br>
<br>
So we are in agreement that we need to understand and document using
meta-data the dependencies, but I don't really understand what
extracting the tests from the scripts really buys us. I don't think
it does harm, if the tools are put in place to allow developers to
still run 'suites', but it certainly doesn't solve the dependency
issue either.<br>
<blockquote cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com" type="cite">
<div><br>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
On the topic of meta data I think placing machine readable
extensible data in a comments section above every test to
store this data would make a lot of sense. We probably need
nothing more than a set of key:value pairings. If people
have seen good working solutions that should be considered
it would be useful if they posted them.<br>
</div>
</blockquote>
Yes, I'm a big proponent of YAML. Beyond this keyword
descriptions of the test would allow automatic testing of
specific areas, "acl" or "fopen" or "quotas" etc.<br>
</div>
</blockquote>
I'll try an propose something to this list.<br>
<blockquote cite="mid:73AED5C780AE05478241DB067651A92102F10959@XYUS-EX22.xyus.xyratex.com" type="cite">Obviously this is not as bold as Chris' statement,
which, by the way, I am happy to entertain as well. Do others have
anything to add to the list above, or thoughts on proceeding with
a completely new architecture?
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000">
<blockquote cite="mid:mailman.4558.1334185245.28966.twg-opensfs.org@lists.opensfs.org" type="cite"> </blockquote>
My statement was not bold it was just a statement of fact.
The current test-framework is not a suitable platform for
fulfilling the requirements of tomorrow. We have to be
building for the next 10 years which will include exascale,
does anybody really feel that the current bash based,
un-scalable and relatively unstructured test facility that
we have today is that platform?</div>
</blockquote>
</div>
<div>I think not. The discussion here is a starting point
specifically for identifying the current problems, with the hope
of driving a an evolutionary way of getting there. Perhaps via
OpenSFS we can drive this as a prioritized task of broad
interest.<br>
<br>
</div>
</blockquote>
We can all agree that we don't have the luxury of a big-bang change
over to a new method, but evolution in this case needs to be a new
species that will overtime become dominant and eventually replace
the current bashous-thalensis!<br>
<br>
Chris<br>
<br>
</div>
</blockquote></div><br></div></div></body></html>