[Twg] [Discuss] [Lustre-devel] your opinion about testing

[Andreas Dilger]

On 2012-05-02, at 10:19 AM, Nathan Rutman wrote:
> On May 2, 2012, at 8:44 AM, Roman Grigoryev wrote:
>> I think it is very interesting possibility but, but from QA/QE logic
>> there is problem : we are planing to use for test control messages
>> channel which is under test too. So, we have to have separated lnet test
>> for prove that lnet is working ok before starting full testing session.
> 
> To some degree.  Most (all?) of the test-framework current tests are actually Lustre tests, and not LNET tests, so I don't think this will have much of an impact.  If there's some minor additional LNET load due to the control message overhead, I would expect LNET to handle this reasonably -- LNET has historically been quite robust.  This actually ties in to the "8. Splitting packages" point; if LNET and Lustre are separate packages, I expect Lustre testing could depend on proper LNET setup and operation.
> 
> The bigger issue to me is that this does require a network and/or Lustre FS to already be set up -- configured, routed, running at least LNET. 

LNET and LST can be run independently of Lustre.

>  This is a fairly big change from the current situation, where we assume we can start/stop/reconfigure things willy-nilly. However, I also think that it's probably a good thing to separate the configuration / setup of the cluster from the test execution in most cases.

Right, and this is done, or ongoing in the existing tests (e.g. check_and_setup_lustre()).

>> Also we must keep possibility to execute tests (maybe not all) without
>> this feature, in classic way, f.e. via ssh.
> 
> I think that's probably true; in any case it needs to be encapsulated in a library layer.

Sure, like do_facet() or do_nodes().  Any test, or even helper function, calling SSH/RSH/pdsh directly should be considered broken.

>  I think in general terms the simplification of running the tests in complex environments will be worth the possibly reduced scope -- but others please chime in!  This is my opinion of writing/using the tests as a developer, and may not match other experiences.
> 
>> 
>> Thanks,
>> 	Roman
>> 
>> 
>> On 05/01/2012 02:37 AM, Nathan Rutman wrote:
>>>> 
>>>>> 3. Remote operations via lctl messages.
>>>>> The tests frequently fail to work on many clusters due to passwordless ssh requirements.  Instead, remote commands / function calls should be sent to other Lustre nodes via a new LNET command, similar to 'lctl ping', that allows arbitrary data to be sent.  A "testing upcall" procedure would be registered on the remote node and would receive the messages.  Obviously this could be a more broadly useful mechanism.  Due to security issues some sites may chose not to register an upcall. A response mechanism is also required.
>>>> 
>>>> It is not immediately clear how this fixes anything.  Such a facility 
>>>> built into lnet needs to have exactly the same configuration issues as 
>>>> ssh, munge, or whatever.  If it DOESN'T have the same issues, then it 
>>>> sounds like it would be a very serious security risk.
>>> What it fixes is the ease of testing, by not requiring a separate communication path outside of LNET.  No router problems, no ssh'isms; if your client can talk to your servers, you're good to go.  
>>> Obviously it doesn't resolve any security issues, and may introduce new ones.
>>> 
>>>> 
>>>> Such a feature needs to at least be off by default and only enabled by a 
>>>> human deciding that such a remote-access feature is acceptable.
>>> Agreed. Hopefully it might be easier to qualify a particular upcall program as "safe", rather than allowing arbitrary code execution as ssh does, but that's not the intent here.  
>>> The goal is to fix the testing, and I think this is one useful component of that.
>>> _______________________________________________
> _______________________________________________
> twg mailing list
> twg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org


Cheers, Andreas
--
Andreas Dilger                       Whamcloud, Inc.
Principal Lustre Engineer            http://www.whamcloud.com/