[cdwg] FW: Lustre Development Community Tree Maintenance Q2 2012 report

[Peter Jones]

Hi Terri

The chart you are referring to is an aggregated summary of all the 
testing results for master in our test results database (maloo) broken 
down by test suite and git tag (which we create ~ every two weeks).

The numbers (x/y) show the number of test runs that passed with no 
failures (x) and the total number of test runs. Each test suite executes 
many tests. The number of tests run should increase over time as new 
tests are added in conjunction with bugfixes.

Some test suites have been removed because they were deemed to be 
obsolete (e.g. liblustre) and some have been added (e.g. mds-survey)

The best way to analyze this report is dynamically online at 
https://maloo.whamcloud.com/reports. There you are able to drill into 
the individual test runs and see what configuration was run and the 
details of any failures (with links to JIRA)

Our goal is work through all the test failures (which are mostly due to 
either tests that were written with certain assumptions that are not 
reliably valid in our present-day testing environment or to certain 
configurations which the test environment does not fully support yet) so 
that all tests will show as green unless there is an actual bug introduced.

As to how well this correlates to the quality of the work, that is open 
to debate. Personally, I think that, while I would not look at this data 
in isolation, I think that it has the advantage of being largely 
comparable to earlier Lustre releases and less affected by unrelated 
factors. For example, if we looked at the number of bugs reported 
against a given release, say, this could be affected by the the total 
number of sites running Lustre, or the introduction of new hardware that 
breaks former coding assumptions and so overall quality improvements 
could be offset by "noise" (or vice versa if people are slow to adopt 
newer releases and so do not discover and report bugs, say)

I'll be happy to answer any further questions that you have either via 
email or on the upcoming CDWG call on Thursday.

Regards

Peter

On 12-07-24 9:55 AM, Quinn, Terri wrote:
> CDWG,
>
> Perhaps someone could help me better understand this report. I have a few questions but I'll start with this one. What should I glean from the Quality Metrics included in the report on pages 4 and 5? What I see are the results of a number of tests. Some passed some did not. This is all understandable, but what does this say about the quality of the work? I might conclude that since I see less red as time goes on that things are getting better. However I also note that less tests are being performed.
>
> Regards, Terri
>
>
> Terri Quinn
> Principal Deputy Department Head
> Integrated Computing and Communications
> Lawrence Livermore National Laboratory
> quinn1 at llnl.gov
> cell 925 321-2879
> office 925 423-2385
>
>
> -----Original Message-----
> From: cdwg-bounces at lists.opensfs.org [mailto:cdwg-bounces at lists.opensfs.org] On Behalf Of Hamilton, Pam
> Sent: Thursday, July 12, 2012 10:11 PM
> To: 'cdwg at lists.opensfs.org'; OpenSFS Execs (Execs at lists.opensfs.org)
> Subject: [cdwg] Lustre Development Community Tree Maintenance Q2 2012 report
>
> Hi all,
>
> Attached is the Q2 2012 quarterly report from Whamcloud for the Lustre Development Community Tree Maintenance contract. Please send any questions/comments to cdwg at lists.opensfs.org.
>
> Regards,
> Pam
> ___________________________________
> Pam Hamilton
> Lawrence Livermore National Lab
> P.O. Box 808, L-556
> Livermore, CA  94551-9900
> E-Mail:  pgh at llnl.gov
> Phone:  925-423-1332          Fax:  925-423-8719
>
>
>
>
> _______________________________________________
> cdwg mailing list
> cdwg at lists.opensfs.org
> http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org

-- 
Peter Jones
Whamcloud, Inc.
www.whamcloud.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opensfs.org/pipermail/cdwg-opensfs.org/attachments/20120724/97d6049e/attachment-0001.htm>