[Twg] [Discuss] [Lustre-devel] your opinion about testing

[Nathan Rutman]

I really don't want to rathole here on the security implications of sending messages through LNET.   

I'll repost: this is a set of concrete proposals for improving the test situation with Lustre.  
My goal with this list is to spark other ideas, and see if we can get some momentum toward addressing the problems with the current testing framework.


> Chris and I talked a little at LUG about the Lustre testing suite, and came up with a some thoughts about how to get out of the current mess:
> 
> 1. New language / library
> Bash has several problems: slow, non-portable, obscure syntax constructions, difficulty doing math, and most importantly it's not function-oriented.  A replacement language that called clearly-defined functions with clear parameters allows for the building of a function library.  Ideally current bash scripts could call the new library functions as we slowly replace the old framework. 
> 
> 2. Start new tests with new framework.
> New tests would be required to use the new framework, thus building up the library.
> 
> 3. Remote operations via lctl messages.
> The tests frequently fail to work on many clusters due to passwordless ssh requirements.  Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent.  A "testing upcall" procedure would be registered on the remote node and would receive the messages.  Obviously this could be a more broadly useful mechanism.  Due to security issues some sites may chose not to register an upcall. A response mechanism is also required.
> 
> Roman and I additionally had some more thoughts:
> 
> 4. Clarify local / remote operations
> Often tests are written assuming a particular configuration (eg. llmount.sh) or version, and all tests execute local script data remotely.  These must be rewritten as encapsulated library functions which may execute remote procedure calls.  A library on the remote node would be responsible for executing the procedure.
> 
> 5. Clarify test independence
> Currently many tests need all tests in a particular range to run, in sequence, in order to correctly set up and clean up a particular environment.  The pre-requisites and post-requisites for each test need to be broken out from the tests themselves.
> 
> 6. Add meta-information to tests
> Tags bearing descriptions and other characteristics of the tests should be included with each test to allow for automated test selection.  E.g. label:quota could be used to run all quotas tests.
> 
> 7. Remove version dependency
> Having the Lustre tests tied to the Lustre source makes it difficult to run interoperability tests consistently (e.g. use old or new version of tests?).  Instead, we can separate out the tests into an independent set, and have interoperability matrixes for tests that should or should not be run.  This also eliminates the need to backport tests between Lustre versions.
> 
> 8. Splitting packages
> Lustre, lustre test suite, LNET, and ldiskfs should all be separated into different packages.  Dependencies between package versions should be addressed by normal RPM methods.  This is probably useful for a variety of different projects: zfs,  the lustre-client-in-upstream-kernel, etc.




On May 1, 2012, at 10:07 AM, Bruce Korb wrote:

> Hi,
> 
> On 4/30/12 3:37 PM, Nathan Rutman wrote:
>>> 
>>> It is not immediately clear how this fixes anything.  Such a facility
>>> built into lnet needs to have exactly the same configuration issues as
>>> ssh, munge, or whatever.  If it DOESN'T have the same issues, then it
>>> sounds like it would be a very serious security risk.
>> What it fixes is the ease of testing, by not requiring a separate
>> communication path outside of LNET.  No router problems, no ssh'isms; if
>> your client can talk to your servers, you're good to go.
>> Obviously it doesn't resolve any security issues, and may introduce new
>> ones.
> 
> It would introduce new ones.  By the time you are done addressing them,
> you've
> likely spent the effort required to set up ssh daemons.  For folks that
> find
> it difficult to configure, perhaps make a separate "lustre-testing" package
> that does all the configury necessary to set up ssh for a specific testing
> user.
> I think developing that would be far easier than rolling your own new
> mechanism.
> 
>> 
>>> 
>>> Such a feature needs to at least be off by default and only enabled by
>>> a 
>>> human deciding that such a remote-access feature is acceptable.
>> Agreed. Hopefully it might be easier to qualify a particular upcall
>> program as "safe",
> 
> Installing the package would be a good human indicator.
> You'd want such a package to be easy to turn off, too,
> so it is only operational when testing is known to be active.
> 
>> rather than allowing arbitrary code execution as ssh does,
> 
> ssh can be constrained.  Constrain it to a particular testing user.
> Over time, that user could have a testing harness shell that constrains
> the command set however you like.  Having done this, I can say I found
> it fairly trivial.
> 
>