[Openbenchmark] Benchmarking Working Group

Sat Mar 24 08:35:53 PDT 2012

Henry,

The benchmark, any benchmark is as good as the profile of the workload it wants to represent as close as possible as many applications behavior. It is obvious that there are applications running on Lustre that are not representative to typical usage of Lustre FS. The art here is to find the best workload generator that will represent the majority of the apps using Lustre. Yet when a users selects a Lustre FS server he needs to give them a clear picture if Lustre is the right FS as well as which Lustre vendor has a better implementation. It is almost impossible for a user to use a benchmark for a single type of application. What majority of users do is build their own micro-benchmark specific to the application they us, of course if they have a real workload characterization of the application. In my experience many apps do not have a definition of the workload and prefer to select servers using a more general benchmark such as sfs, SPC, and others. This is the value of published benchmark data that are general enough to allow to compare different implementations. As a disclaimer a benchmark cannot ever represent all the behavior of all the apps using a FS but can represent accurately the average goodness of a FS implementation assuming a typical average usage of the FS. That being said I need the help of the LUG community to build the best most representative average workload definition for sfs2013.
I hope this explains the reason behind using general benchmarks and sfs2013 in particular

Sorin Faibish
Distinguished Engineer
EMC - Office of CTO

-----Original Message-----
From: Henry Newman [mailto:hsn at instrumental.com] 
Sent: Saturday, March 24, 2012 10:39 AM
To: faibish, sorin; Jeffrey_Layton at Dell.com; Richard.Vanderbilt at netapp.com; openbenchmark at lists.opensfs.org; discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org
Subject: RE: Benchmarking Working Group

Have some SPEC data is interesting in my opinion only if the workload maps to user application.  Much of Lustre usage from what I have seen is defensive I/O and therefore is write intensive.  How does that map is the question at hand.

Thanks

hsn

________________________________________
Henry Newman | CEO/CTO
Instrumental, Inc | High Performance Innovation
1450 Energy Park Drive Suite 375
St. Paul, MN 55108-5274
Direct:      651-280-4801
Fax:         651-280-4839
Toll-free:   1.800.886.6188
E-Mail:      hsn at instrumental.com
Web:         http://www.instrumental.com

STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and 
proprietary material for the sole use of the intended recipient. Any review 
or distribution by others is strictly prohibited. If you are not the 
intended recipient please contact the sender and delete all copies.

-----Original Message-----
From: openbenchmark-bounces at lists.opensfs.org [mailto:openbenchmark-bounces at lists.opensfs.org] On Behalf Of faibish_sorin at emc.com
Sent: Friday, March 23, 2012 4:35 PM
To: Jeffrey_Layton at Dell.com; Richard.Vanderbilt at netapp.com; openbenchmark at lists.opensfs.org; discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org
Subject: Re: [Openbenchmark] Benchmarking Working Group

-----Original Message-----
From: Jeffrey_Layton at Dell.com [mailto:Jeffrey_Layton at Dell.com] 
Sent: Friday, March 23, 2012 4:55 PM
To: faibish, sorin; Richard.Vanderbilt at netapp.com; openbenchmark at lists.opensfs.org; discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org
Subject: RE: Benchmarking Working Group

Sorin,

Is the intent to add some benchmarks that utilize the ideas and concepts of sfs2013 to the list? Or is the intent to possible add sfs2013 itself to the list of benchmarks?
[SF] My intention was to include Lustre as one of the workloads of sfs2013 as a special profile to the benchmark. Regardless, I was approved by sfs committee to add a Lustre specific profile to sfs2013. I hope with the help of the Lustre community to create a realistic and representative profile to be included in sfs2013 additional to other benchmarks that this group will decide to do. The advantage of such a benchmark will be that vendors can publish results officially to the SPEC.org reviewed by a peer audit group. Hope this makes sense

Sorin Faibish
Distinguished Engineer
EMC - Office of CTO

Thanks!

Dr. Jeff Layton
HPC Enterprise Technologist
Dell | Research Computing 
mobile + 1 678 427-5819
jeffrey_layton at dell.com

-----Original Message-----
From: openbenchmark-bounces at lists.opensfs.org [mailto:openbenchmark-bounces at lists.opensfs.org] On Behalf Of faibish_sorin at emc.com
Sent: Friday, March 23, 2012 1:12 PM
To: Richard.Vanderbilt at netapp.com; openbenchmark at lists.opensfs.org; discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org
Subject: Re: [Openbenchmark] Benchmarking Working Group

Per the discussion during the call, attached please find the documentation of sfs2013 benchmark (currently called netmist).

Please take a look and let me know if it is of interest to continue on this path of action for the Lustre benchmark. I will be attending the LUG and plan to introduce sfs2013 benchmark. Thank you for your understanding and patience

Sorin Faibish
Distinguished Engineer
EMC - Office of CTO

_____________________________________________
From:   Vanderbilt, Richard [mailto:Richard.Vanderbilt at netapp.com]
Sent:   Monday, March 19, 2012 12:25 PM
To:     openbenchmark at lists.opensfs.org; discuss at lists.opensfs.org; lustre-discuss at lists.lustre.org; faibish, sorin
Subject:        [Openbenchmark] Benchmarking Working Group
When:   Friday, March 23, 2012 11:30 AM-12:30 PM (GMT-05:00) Eastern Time (US & Canada).
Where:  1(866)844-9418 id 4088226016

The OpenSFS Benchmarking Workgroup aims to provide an I/O benchmark suite to satisfy the benchmarking requirements of the scalable parallel file system users and facilities. Towards this end, as a first step, the workgroup aims to characterize the I/O workloads from small- to very large-scale scalable parallel file systems deployed at various high-performance computing and parallel computing facilities and institutions. Using these gathered characteristics, the workgroup will then collect and build the required I/O benchmarks to emulate these workloads and provide required documentation about the benchmark suite.

The workgroup meeting will reoccur every Friday at 8:30am Pacific time. 1 hour each. The dial-in is as follows: 1(866)844-9418 id# 4088226016.

Also note: we will be having our first face to face meeting on Sunday April 22nd from 2:00-5:00pm at the LUG.  Details to come.

On Tuesday April 24 4:10-5:00pm there will be a Benchmark panel discussion moderated by Sarp Oral. Panelist will be Jeremy Filizetti (DOD), Andrew Uselton (NERSC), Eric Barton (Whamcloud), Nathan Rutman (Xyratex), Stephane Thiell (CEA), and Stephen Simms (IU).

If you are interested in contributing to this work group, please email Sarp Oral at oralhs at ornl.gov<mailto:oralhs at ornl.gov> or myself at richard.vanderbilt at netapp.com<mailto:richard.vanderbilt at netapp.com>.

Sincerely,

Sarp Oral, PhD
National Center for Computational Sciences Oak Ridge National Laboratory oralhs at ornl.gov<mailto:oralhs at ornl.gov>
865-574-2173

Richard Vanderbilt  Co-chair
Netapp Inc.
(408)822-6016

  << File: ATT00001.txt >>

_______________________________________________
Openbenchmark mailing list
Openbenchmark at lists.opensfs.org
http://lists.opensfs.org/listinfo.cgi/openbenchmark-opensfs.org
STRICTLY PERSONAL AND CONFIDENTIAL. This email may contain confidential and proprietary material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies.