[Twg] [Lustre-devel] your opinion about testing

[Nathan Rutman]

On Apr 13, 2012, at 10:49 AM, Robert Read wrote:


	Hi,
	
	On Apr 13, 2012, at 6:29 , Chris wrote:
	
	

		Nathan,
		



			discussion: Xyratex started by suggesting that the test suite included with every Lustre release needed reworking. This led to a group conversation that focused on the following areas: 1) refactor unused tests Test scripts have grown unwieldly and do not form a coherent test package. As a result, there are some tests that are used regularly and others that ineffective and skipped. We should remove the unused tests.
			


				Removing tests that are genuinely ineffective is a good idea, but we must be careful because historically tests have been skipped for many reasons including the non-function of a useful test, or even the fact that Lustre could not be made to pass!  I think this refactor process should be framed more as an examination of currently skipped steps to understand which should be fixed+re-enabled and which should be removed.
				


				It's also worth noting that the ratio of 'used regularly' to skipped is probably 50:1, skipped tests have no runtime cost and so the ROI is probably not significant. In fact the ROI is probably only going to be worthwhile if skipped tests can be made useful and re-enabled.
				

			There's also a component here of removing redundant tests -- Robert Read a few years ago a LUG demonstrated code coverage of SLOW=NO vs SLOW=YES as virtually identical, but YES takes hours to run.
			

		I've not seen Robert's code coverage figures, did he measure static code coverage and dynamic behaviours, because I would hope and certainly not presume otherwise that SLOW=YES exercises the dynamic behaviours and states much more thoroughly.
		


	I don't have that data anymore, but it is true that we saw virtually no difference in static code coverage with SLOW=yes. However we did see significantly more bugs with SLOW=yes, and so we always did testing in slow mode. 
	



http://wiki.lustre.org/images/e/ee/QualityInitiativeTalk.pdf




	Also, to give credit where it's due,  Li Wei did the actual coverage analysis. 
	
	robert
	
	


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/twg_lists.opensfs.org/attachments/20120413/be3dffbe/attachment.html>