From Nathan_Rutman at xyratex.com Fri Apr 1 22:05:43 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 1 Apr 2011 15:05:43 -0700 Subject: [Twg] Xyratex plans In-Reply-To: References: Message-ID: <73AED5C780AE05478241DB067651A921023783F1@XYUS-EX22.xyus.xyratex.com> Addressing concerns raised in the twg call yesterday: We (Xyratex internally) are all agreed that we need to provide more 'transparency' to the community as to what we are doing with Lustre and our intentions regarding contributions to the community. We will be setting up a Lustre community page on our website where we will be publishing our roadmap, white papers and other relevant content. Our roadmap will be available at LUG 2011, but in the meantime here is a brief list of the Lustre topics that Xyratex is interested in pursuing. Paths forward to all these items will be published in the roadmap. Many of these align with stated OpenSFS goals; apologies for not being able to publish this information sooner. * 4MB RPCs * Persistent, fast OSS write flash caches * Metadata flash caches * NRS * Fully-wide striping (e.g. all or almost all OSTs) * Client metadata concurrency (remove MDC semaphore) * Simplified SOM (using metadata flash) * MDS threading improvements * 100B files per FS * 1B files per directory * scalable fsck * end-to-end data integrity * larger LUN sizes * FS replication ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter_bojanic at xyratex.com Sun Apr 3 12:25:42 2011 From: peter_bojanic at xyratex.com (Peter Bojanic) Date: Sun, 3 Apr 2011 09:25:42 -0300 Subject: [Twg] Xyratex plans In-Reply-To: <73AED5C780AE05478241DB067651A921023783F1@XYUS-EX22.xyus.xyratex.com> References: <73AED5C780AE05478241DB067651A921023783F1@XYUS-EX22.xyus.xyratex.com> Message-ID: <5CBBD3EC-6DCB-407D-A94A-718914F35CFD@xyratex.com> Thanks, Nathan. Just to add to this: Xyratex is a public company and must manage public communications with a view to its broader business and product portfolio. In our Q1 FY2011 earnings call on March 31, 2011, our CEO Steve Barber has officially announced Xyratex's intentions to move forward with Lustre and to deliver new products in support of this growth market. We may now discuss elements of our plan much more freely with the HPC community, in particular our goals for Lustre in support of our business. We look forward to a much more open collaboration involving Lustre. Cheers, Bojanic On Apr 1, 2011, at 19:05 , Nathan Rutman wrote: > Addressing concerns raised in the twg call yesterday: > > We (Xyratex internally) are all agreed that we need to provide more 'transparency' to the community as to what we are > doing with Lustre and our intentions regarding contributions to the community. We will be setting up a > Lustre community page on our website where we will be publishing our roadmap, white papers and other relevant content. > > Our roadmap will be available at LUG 2011, but in the meantime here is a brief list of the Lustre topics that Xyratex is interested in pursuing. Paths forward to all these items will be published in the roadmap. Many of these align with stated OpenSFS goals; apologies for not being able to publish this information sooner. > > 4MB RPCs > Persistent, fast OSS write flash caches > Metadata flash caches > NRS > Fully-wide striping (e.g. all or almost all OSTs) > Client metadata concurrency (remove MDC semaphore) > Simplified SOM (using metadata flash) > MDS threading improvements > 100B files per FS > 1B files per directory > scalable fsck > end-to-end data integrity > larger LUN sizes > FS replication > > > ______________________________________________________________________ > This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. > > Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. > > Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. > > The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. > ______________________________________________________________________ > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssseager at gmail.com Mon Apr 4 22:29:14 2011 From: ssseager at gmail.com (Shay Seager) Date: Mon, 4 Apr 2011 15:29:14 -0700 Subject: [Twg] Google Docs Info Sheet Message-ID: <34CFF477-E363-4A64-9BBE-6ECC4DF12FE3@gmail.com> Hello! OpenSFS recently started using Google Docs as a way to share and store internal documents. There have been many questions about how to use Google Docs, so the communication support working group created a brief on the quickest ways to maximize your Google Doc experience, check it out at this link https://docs.google.com/document/pub?id=1jo0aB9Z1zISqQwaUJC1ICaJu72mFhRWHyBR5ZVl8FDI This is a working document so all feedback is welcome and encouraged. If you would like to edit this document please send me an e-mail and I will send you an invite. Otherwise you can send the communication working group or shay at opensfs.org your comments and we will incorporate them. Thank you for using Google Docs! Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org From ssseager at gmail.com Tue Apr 5 18:13:37 2011 From: ssseager at gmail.com (Shay Seager) Date: Tue, 5 Apr 2011 11:13:37 -0700 Subject: [Twg] LUG 2011 OpenSFS meeting - UPDATE Message-ID: Hello Working Groups, Next week is LUG 2011. On April 14th from 1-5pm EST OpenSFS will be having a group meeting, open to all OpenSFS community participants and those in due-dilligence. The location has been announced, we will be meeting in the Key West room. The agenda is posted http://www.opensfs.org/?page_id=46, this is a working document, so if you have any topics to be addressed please e-mail shay at opensfs.org and I will incorporate your suggestions. Thank you so much for your time and effort and we look forward to seeing you all at LUG 2011! Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Wed Apr 6 20:17:43 2011 From: carrier at cray.com (John Carrier) Date: Wed, 6 Apr 2011 15:17:43 -0500 Subject: [Twg] updates Message-ID: A couple of updates for you 1) There will be no meeting tomorrow (4/7). Dave and I have been at the OFA conference and will be traveling during our call. 2) The twg-rfp produced RFP documents based on the twg's recommendations which the board approved. Expect to see RFPs from OpenSFS announced on the Lustre reflectors before LUG. 3) EOFS has produced its own set of requirements, which are attached. Dave and I think they mesh well with the requirements we have put together and want to work with the EOFS to create a set of unified requirements. Dave and I will be at LUG. Hope to see you there. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: EOFS_Lustre_priorities.pdf Type: application/pdf Size: 50606 bytes Desc: EOFS_Lustre_priorities.pdf URL: From ssseager at gmail.com Thu Apr 14 04:47:26 2011 From: ssseager at gmail.com (Shay Seager) Date: Wed, 13 Apr 2011 21:47:26 -0700 Subject: [Twg] Call-in for OpenSFS Meeting at LUG 4/13/11 1-5pm EST (10am-2pm PST) 866-914-3976 passcode 534986# Message-ID: Hello All, Tomorrow from 1-5pm EST (10am-2pmPST) OpenSFS will be hosting a meeting for participants in the Key West Room at the Marriot World Center, Orlando, FL. If you are not in Florida but would like to participate in the meeting, please call into 866-914-3976 passcode 534986# at 10am Pacific/1pm Eastern. For more information about the event please see http://www.opensfs.org/?page_id=46. Please don't hesitate to reply to this e-mail with any questions. We look forward to your participation. Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org From carrier at cray.com Thu Apr 21 06:36:38 2011 From: carrier at cray.com (John Carrier) Date: Thu, 21 Apr 2011 01:36:38 -0500 Subject: [Twg] TWG concall reminder Message-ID: Hi all, Just a reminder of our regularly scheduled TWG concall (Thursday 4/21) at 12:30ET/9:30PT. With the RFP in progress, Dave and I want to discuss our expectations for code delivery. What exactly is the "collateral" we should expect for patches and new features in order to ensure that we deliver the highest quality code? We will open the agenda for additional topics at the start of the meeting. Concall info: Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 NOTE that this is a new ID and password! (Dial-up numbers are the same, but the old ID/PW were retired.) Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Apr 21 07:22:02 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 21 Apr 2011 01:22:02 -0600 Subject: [Twg] TWG concall reminder In-Reply-To: References: Message-ID: On 2011-04-21, at 12:36 AM, John Carrier wrote: > Just a reminder of our regularly scheduled TWG concall (Thursday 4/21) at 12:30ET/9:30PT. > > With the RFP in progress, Dave and I want to discuss our expectations for code delivery. What exactly is the “collateral” we should expect for patches and new features in order to ensure that we deliver the highest quality code? Does that mean the call is closed to people who are responding to the RFP? > We will open the agenda for additional topics at the start of the meeting. > > Concall info: > Dial: 1-715-726-4994, 1-866-304-8294 > ID & Password: 72090 > > NOTE that this is a new ID and password! (Dial-up numbers are the same, but the old ID/PW were retired.) Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From carrier at cray.com Thu Apr 21 07:44:23 2011 From: carrier at cray.com (John Carrier) Date: Thu, 21 Apr 2011 02:44:23 -0500 Subject: [Twg] TWG concall reminder In-Reply-To: References: Message-ID: Hi Andreas, Good question. I think this should be an open discussion. We will not discuss the RFPs directly, just what OpenSFS expects to accompany any code delivered at the end of the statements of work for the RFPs. So just to be clear, there will be no discussion of the ongoing RFP. Thanks, --jc -----Original Message----- From: Andreas Dilger [mailto:adilger at whamcloud.com] Sent: Thursday, April 21, 2011 12:22 AM To: John Carrier Cc: twg at lists.opensfs.org; discuss at lists.opensfs.org; Shay Seager Subject: Re: [Twg] TWG concall reminder On 2011-04-21, at 12:36 AM, John Carrier wrote: > Just a reminder of our regularly scheduled TWG concall (Thursday 4/21) at 12:30ET/9:30PT. > > With the RFP in progress, Dave and I want to discuss our expectations for code delivery. What exactly is the "collateral" we should expect for patches and new features in order to ensure that we deliver the highest quality code? Does that mean the call is closed to people who are responding to the RFP? > We will open the agenda for additional topics at the start of the meeting. > > Concall info: > Dial: 1-715-726-4994, 1-866-304-8294 > ID & Password: 72090 > > NOTE that this is a new ID and password! (Dial-up numbers are the same, but the old ID/PW were retired.) Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From carrier at cray.com Thu Apr 21 19:38:28 2011 From: carrier at cray.com (John Carrier) Date: Thu, 21 Apr 2011 14:38:28 -0500 Subject: [Twg] TWG meeting minutes for 2011-04-20 Message-ID: Attached are the meeting minutes from today's TWG call where we started the discussion of what OpenSFS will require for release collateral and delivery of patches that result from our RFPs. Please post questions/comments to discuss at lists.opensfs.org Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 04/21/2011 Concall: start : 9:30a PT, adjourn : 10:10a PT Next meeting: Thursday, 4/28/2011 @ 9:30a PDT/12:30p EDT Dial-in numbers are 715-726-4994 or 866-304-8294. Meeting ID and password are 72090 (note the new code) Attending : Name Organization email ----------------- -------------- ---------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Steve Simms IU ssimms at indiana.edu Joshua Walgenbach IU jjw at indiana.edu Chris Morrone LLNL morrone2 at llnl.gov Shay Seager OpenSFS shay at opensfs.org Dave Dillow ORNL dillowda at ornl.gov Andreas Dilger Whamcloud adilger at whamcloud.com Nathan Rutman Xyratex nathan_rutman at xyratex.com Agenda ------------------------------------ release collateral Discussion -------------------------------- The release material is independent of who is hosting the canonical branch or who is managing the next release. OpenSFS is paying for code development and we need to set expectations for what is required to land the new features in a release branch. Baseline requirements are - design documents - test plan - results of tests Sun/Oracle used a procedure where the developer would * push the patches into a development branch of the source tree * request two inspectors to review the patches * run tests against the private branch * report results of the tests to the gatekeeper The gatekeeper would then negotiate with the developer whether the feature was ready for inclusion. For OpenSFS, it was suggested that we include scale testing to the list of tests to be run before posting the results to the gatekeeper. Note that this list is similar to the submission guidelines offered by Whamcloud on their wiki (see "Patch landing process" at http://wiki.whamcloud.com/display/PUB/Submitting+Changes). >> OpenSFS should provide our own reference to these steps to clearly set our expectations of the vendors. The guidelines should be release neutral, but acknowledge that they should be tailored to the steps of the current release manager. In addition, Sun/Oracle had checklists that the developer needed to consider before actually submitting the patch. These included considerations of impact of the patch on recovery, scaling and interoperability. There were also coding style checklists to ensure that the code was compatible with the kernel. (Whamcloud has a copy of the coding style manual at http://wiki.whamcloud.com/display/PUB/Coding+Guidelines ). As part of the coding guidelines, OpenSFS expects appropriate copyright statements for the new features. These typically follow the license of whatever module is being modified. >> OpenSFS should provide a copy of both types of checklists to the vendors. Andreas and Nathan said they would send the historical Lustre submission checklists to the TWG reflector. The following are the proposed deliverables which should be included in the RFP contracts : 1) We require design documents available for community review The TWG is responsible for approving the designs after confirming that they meet the requirements in the RFP and then recommending that implementation of the designs begins. 2) We require the code to land upstream This means we will follow standard Lustre guidelines (as described above) for delivering the patches to the canonical tree. * We expect the vendor to request two qualified, impartial inspectors rather than the TWG itself to review the feature submissions. (We acknowledge that the TWG itself does not have the expertise to do the review, though some of its members do.) * Patches need to be sized to facilate easy review. [Do we have guidelines for patch size?] * At the time of patch submission there needs to be a test plan. Results need to be posted publically to the patch as tests are completed. * We expect the ongoing development branch to be publically readable. Though we desire a stable top of tree, we expect vendors to provide tags covering significant milestones (eg, test completion) so that the tagged stable branch can be inspected and tested by the community. adjourn 10:10 PT From adilger at whamcloud.com Fri Apr 22 00:25:34 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 21 Apr 2011 18:25:34 -0600 Subject: [Twg] TWG meeting minutes for 2011-04-20 In-Reply-To: References: Message-ID: On 2011-04-21, at 1:38 PM, John Carrier wrote: > Attached are the meeting minutes from today’s TWG call where we started the discussion of what OpenSFS will require for release collateral and delivery of patches that result from our RFPs. > > Please post questions/comments to discuss at lists.opensfs.org I looked around and in fact do not have any of the TSP HLD/DLD design checklists. IIRC they were kept on the Lustre internal wiki, and I couldn't find any of them attached to bugzilla, only Sun "process review" checklists for things like who will support the feature, if it was tested, etc. https://bugzilla.lustre.org/attachment.cgi?id=19272 is one such example, but this isn't really what I was looking for. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From peter_bojanic at xyratex.com Fri Apr 22 03:43:45 2011 From: peter_bojanic at xyratex.com (Peter Bojanic) Date: Fri, 22 Apr 2011 00:43:45 -0300 Subject: [Twg] TWG meeting minutes for 2011-04-20 In-Reply-To: References: Message-ID: John, > * We expect the vendor to request two qualified, impartial > inspectors rather than the TWG itself to review the feature > submissions. (We acknowledge that the TWG itself does not > have the expertise to do the review, though some of its > members do.) Please forgive me if this topic has been discussed more thoroughly by the TWG in the past. OpenSFS might consider formulating this list of trusted inspectors on behalf of the community. Inspectors are volunteers from the community and are selected based on merit. To be an inspector, you submit information about: - summary of overall Lustre and development experience - overall years of Lustre experience - years of experience per subsystem - number of accepted commits to Lustre (per subsystem?) Inspectors should also have some number of endorsements from the established group of inspectors. All volunteers go into a public registry. OpenSFS establishes qualification criteria and then transparently selects a group of qualified volunteers. It doesn't matter what organization you work for but you must commit a certain number of inspection hours per month for OpenSFS community projects and chances are you'll be inspecting work WhamClould already have a process refinement that requires two-level inspections, which OpenSFS might consider adopting. Cheers, Bojanic On Apr 21, 2011, at 16:38 , John Carrier wrote: > Attached are the meeting minutes from today’s TWG call where we started the discussion of what OpenSFS will require for release collateral and delivery of patches that result from our RFPs. > > Please post questions/comments to discuss at lists.opensfs.org > > Thanks, > > --jc > <2011-04-21 OpenSFS TWG minutes.txt>_______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From peter_bojanic at xyratex.com Fri Apr 22 04:28:12 2011 From: peter_bojanic at xyratex.com (Peter Bojanic) Date: Fri, 22 Apr 2011 01:28:12 -0300 Subject: [Twg] TWG meeting minutes for 2011-04-20 In-Reply-To: References: Message-ID: On Apr 21, 2011, at 21:25 , Andreas Dilger wrote: > On 2011-04-21, at 1:38 PM, John Carrier wrote: >> Attached are the meeting minutes from today’s TWG call where we started the discussion of what OpenSFS will require for release collateral and delivery of patches that result from our RFPs. >> >> Please post questions/comments to discuss at lists.opensfs.org > > I looked around and in fact do not have any of the TSP HLD/DLD design checklists. IIRC they were kept on the Lustre internal wiki, and I couldn't find any of them attached to bugzilla, only Sun "process review" checklists for things like who will support the feature, if it was tested, etc. > > https://bugzilla.lustre.org/attachment.cgi?id=19272 is one such example, but this isn't really what I was looking for. The material is licensed from the SEI, which is why it was never published to lustre.org. Probably an experienced group of engineers could reconstruct a public, Lustre-centric design and inspection checklist with a few rounds of edits. Bojanic > > Cheers, Andreas > -- > Andreas Dilger > Principal Engineer > Whamcloud, Inc. > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________