[Twg] [Discuss] [Lustre-devel] your opinion about testing

[Nathan Rutman]

On May 2, 2012, at 8:44 AM, Roman Grigoryev wrote:

> Hi Nathan,
> 
> I think it is very interesting possibility but, but from QA/QE logic
> there is problem : we are planing to use for test control messages
> channel which is under test too. So, we have to have separated lnet test
> for prove that lnet is working ok before starting full testing session.

To some degree.  Most (all?) of the test-framework current tests are actually Lustre tests, and not LNET tests, so I don't think this will have much of an impact.  If there's some minor additional LNET load due to the control message overhead, I would expect LNET to handle this reasonably -- LNET has historically been quite robust.  This actually ties in to the "8. Splitting packages" point; if LNET and Lustre are separate packages, I expect Lustre testing could depend on proper LNET setup and operation.

The bigger issue to me is that this does require a network and/or Lustre FS to already be set up -- configured, routed, running at least LNET.  This is a fairly big change from the current situation, where we assume we can start/stop/reconfigure things willy-nilly. However, I also think that it's probably a good thing to separate the configuration / setup of the cluster from the test execution in most cases.

> Also we must keep possibility to execute tests (maybe not all) without
> this feature, in classic way, f.e. via ssh.

I think that's probably true; in any case it needs to be encapsulated in a library layer.  I think in general terms the simplification of running the tests in complex environments will be worth the possibly reduced scope -- but others please chime in!  This is my opinion of writing/using the tests as a developer, and may not match other experiences.

> 
> Thanks,
> 	Roman
> 
> 
> On 05/01/2012 02:37 AM, Nathan Rutman wrote:
>>> 
>>>> 3. Remote operations via lctl messages.
>>>> The tests frequently fail to work on many clusters due to passwordless ssh requirements.  Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent.  A "testing upcall" procedure would be registered on the remote node and would receive the messages.  Obviously this could be a more broadly useful mechanism.  Due to security issues some sites may chose not to register an upcall. A response mechanism is also required.
>>> 
>>> It is not immediately clear how this fixes anything.  Such a facility 
>>> built into lnet needs to have exactly the same configuration issues as 
>>> ssh, munge, or whatever.  If it DOESN'T have the same issues, then it 
>>> sounds like it would be a very serious security risk.
>> What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET.  No router problems, no ssh'isms; if your client can talk to your servers, you're good to go.  
>> Obviously it doesn't resolve any security issues, and may introduce new ones.
>> 
>>> 
>>> Such a feature needs to at least be off by default and only enabled by a 
>>> human deciding that such a remote-access feature is acceptable.
>> Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here.  
>> The goal is to fix the testing, and I think this is one useful component of that.
>> _______________________________________________