From hamilton5 at llnl.gov Mon Jan 3 22:55:11 2011 From: hamilton5 at llnl.gov (Hamilton, Pam) Date: Mon, 3 Jan 2011 14:55:11 -0800 Subject: [Twg] Ideas for consideration Message-ID: Dear OpenSFS Execs, The Support Working Group has had discussions regarding the board's request to show rapid progress by getting a support RFP and development RFP out ASAP. We have two other ideas which can show progress immediately that we'd like the board to consider. First, we could start with a community-developed OpenSFS Lustre release as a bootstrapping step. We'd propose that we target a release based on Oracle's 2.1 release (which is currently in development), for roughly half way through 2011. Our collaborative effort to test and stabilize 2.1 would add legitimacy to that release, and encourage the community to begin migrating to the Lustre 2 branch. Note that a final OpenSFS release/development/support model can and likely will be quite different from this "birthing" step, but will hopefully be able to fully leverage any infrastructure deployed to support it. We could proceed almost immediately by performing the following steps: 1) Create a branch named "2.0.59.0-opensfs" on github, which is based at Oracle's 2.0.59.0 tag. This is a "pre-release" version on the branch that will be tagged 2.1 by Oracle in the next few months. We can easily create a new branch with all of our patches applied to it each time Oracle adds a new tag, or when convenient for us. I.E., when Oracle tags 2.0.60.0, we create 2.0.60.0-opensfs, which contains all of our patches from the 2.0.59.0-opensfs, rebased onto the 2.0.60.0 tag. Eventually, 2.1.0 will be tagged, and we perform a rebase onto it named 2.1.0-opensfs. 2) We create a "opensfs-devel" mailing list, where Lustre developers and testers can communicate and coordinate their efforts. 3) When we discover bugs, we can initially use github's built-in issue tracking, or just use bugzilla.lustre.org, and agree to make all OpenSFS bugs dependants of a master OpenSFS tracking bug. The developers can work out the details amongst themselves on opensfs-devel. We have a volunteer who is willing to be the initial gatekeeper for the OpenSFS 2.1 release if there are no objections, or unless better candidates volunteer. Whoever the gatekeeper is needs to maintain the fairly rigorous processes developed at Oracle to maintain quality and reliability in both development and production branches. Another idea which could be launched quickly is to put out an RFP to do an investigation into Btrfs and its use as a possible backend file system for Lustre. This wouldn't involve any code development. The primary deliverable for this contract would be a report. Both of the above ideas could be executed is the VERY short term and buy ourselves more time to work out the details of an RFP for release development and support. It's the details and the questions they raise which make drafting a support RFP difficult and the reason we were going to have a face-to-face meeting. Questions like the following: * What is the dollar amount OpenSFS expects to spend for such a contract? * What timeframe does the contract cover, one year, two years, or ??? * Where will the money come from to pay for the contract? Dues? Or additional funds from members? We can still get started on a draft, if the board is willing to answer the above questions and any others we have as we proceed. We are also willing to make up our own answers to such questions knowing that the board has the final say. Guidance would be appreciated but we'd also like our ideas given consideration. Regards, Pam ___________________________________ Pam Hamilton Lawrence Livermore National Lab P.O. Box 808, L-556 Livermore, CA 94551-9900 E-Mail: pgh at llnl.gov Phone: 925-423-1332 Fax: 925-423-8719 ___________________________________ ________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Tue Jan 4 18:21:29 2011 From: carrier at cray.com (John Carrier) Date: Tue, 4 Jan 2011 12:21:29 -0600 Subject: [Twg] call for agenda items Message-ID: Hi team, We are scheduled to have our next concall this Thursday at 9:30a PT / 12:30p ET. I submitted our whitepaper draft to the board before the holiday break. At today's community meeting, I was told to move forward with gathering requirements. In the whitepaper, we described breaking up into teams to seek requirements from the various community groups. I would like to move forward with this proposal at this next meeting. So, one agenda item will be to identify the communities and select the teams who will solicit requirements from each group. What else do you think we should try to cover? Please send suggestions to me and Dave. Thanks, --jc From carrier at cray.com Thu Jan 6 17:31:48 2011 From: carrier at cray.com (John Carrier) Date: Thu, 6 Jan 2011 11:31:48 -0600 Subject: [Twg] meeting reminder Message-ID: The concall this morning will use the same dial-in numbers we've used in the past: Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and password are 7012090. Thanks, --jc From dillowda at ornl.gov Thu Jan 6 18:04:54 2011 From: dillowda at ornl.gov (David Dillow) Date: Thu, 06 Jan 2011 13:04:54 -0500 Subject: [Twg] Ideas for consideration In-Reply-To: References: Message-ID: <1294337094.14630.2.camel@lap75545.ornl.gov> TWG members: Please review Pam's memo and comment to the list. We'd like to discuss the ideas on the Jan 13th call. That is expected to be a busy call with other matters, so the more discussion we can shift to the mailing list, the better. Thanks, Dave Dillow On Mon, 2011-01-03 at 17:55 -0500, Hamilton, Pam wrote: > Dear OpenSFS Execs, > > > > The Support Working Group has had discussions regarding the board’s > request to show rapid progress by getting a support RFP and > development RFP out ASAP. We have two other ideas which can show > progress immediately that we’d like the board to consider. > > > > First, we could start with a community-developed OpenSFS Lustre > release as a bootstrapping step. We’d propose that we target a release > based on Oracle's 2.1 release (which is currently in development), for > roughly half way through 2011. Our collaborative effort to test and > stabilize 2.1 would add legitimacy to that release, and encourage the > community to begin migrating to the Lustre 2 branch. Note that a > final OpenSFS release/development/support model can and likely will be > quite different from this “birthing” step, but will hopefully be able > to fully leverage any infrastructure deployed to support it. > > > > We could proceed almost immediately by performing the following steps: > > > > 1) Create a branch named "2.0.59.0-opensfs" on github, which is based > at Oracle's 2.0.59.0 tag. This is a "pre-release" version on the > branch that will be tagged 2.1 by Oracle in the next few months. > > > > We can easily create a new branch with all of our patches applied to > it each time Oracle adds a new tag, or when convenient for us. I.E., > when Oracle tags 2.0.60.0, we create 2.0.60.0-opensfs, which contains > all of our patches from the 2.0.59.0-opensfs, rebased onto the > 2.0.60.0 tag. > > Eventually, 2.1.0 will be tagged, and we perform a rebase onto it > named 2.1.0-opensfs. > > > > 2) We create a "opensfs-devel" mailing list, where Lustre developers > and testers can communicate and coordinate their efforts. > > > > 3) When we discover bugs, we can initially use github's built-in > issue tracking, or just use bugzilla.lustre.org, and agree to make all > OpenSFS bugs dependants of a master OpenSFS tracking bug. The > developers can work out the details amongst themselves on > opensfs-devel. > > > > We have a volunteer who is willing to be the initial gatekeeper for > the OpenSFS 2.1 release if there are no objections, or unless better > candidates volunteer. Whoever the gatekeeper is needs to maintain the > fairly rigorous processes developed at Oracle to maintain quality and > reliability in both development and production branches. > > > > Another idea which could be launched quickly is to put out an RFP to > do an investigation into Btrfs and its use as a possible backend file > system for Lustre. This wouldn’t involve any code development. The > primary deliverable for this contract would be a report. > > > > Both of the above ideas could be executed is the VERY short term and > buy ourselves more time to work out the details of an RFP for release > development and support. It’s the details and the questions they raise > which make drafting a support RFP difficult and the reason we were > going to have a face-to-face meeting. Questions like the following: > > · What is the dollar amount OpenSFS expects to spend for such > a contract? > > · What timeframe does the contract cover, one year, two years, > or ??? > > · Where will the money come from to pay for the contract? > Dues? Or additional funds from members? > > > > We can still get started on a draft, if the board is willing to answer > the above questions and any others we have as we proceed. We are also > willing to make up our own answers to such questions knowing that the > board has the final say. Guidance would be appreciated but we’d also > like our ideas given consideration. > > > > Regards, > > Pam > > > > ___________________________________ > Pam Hamilton > Lawrence Livermore National Lab > P.O. Box 808, L-556 > Livermore, CA 94551-9900 > E-Mail: pgh at llnl.gov > Phone: 925-423-1332 Fax: 925-423-8719 > ___________________________________ > > > > > > > > ______________________________________________________________________ -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From uja at ornl.gov Fri Jan 7 15:06:09 2011 From: uja at ornl.gov (James A Simmons) Date: Fri, 07 Jan 2011 10:06:09 -0500 Subject: [Twg] [Rpwg] Ideas for consideration In-Reply-To: References: Message-ID: <1294412769.21542.21.camel@bohr.ornl.gov> > 1) Create a branch named "2.0.59.0-opensfs" on github, which is based > at Oracle's 2.0.59.0 tag. This is a "pre-release" version on the > branch that will be tagged 2.1 by Oracle in the next few months. Or we could use https://github.com/rread/lustre We should avoid having multiple lustre branches on github. Of course the chaos version is also on github but that is very special branch that has existed for a long time. I have noticed that Oracle is allowing people that left to still check into their source tree. > 2) We create a "opensfs-devel" mailing list, where Lustre developers > and testers can communicate and coordinate their efforts. Yes please do. My experince is that this list usually becomes the main list where everyone post to. We talked about having one main mailing list before. This could easily fill in that gap. > 3) When we discover bugs, we can initially use github's built-in > issue tracking, or just use bugzilla.lustre.org, and agree to make all > OpenSFS bugs dependants of a master OpenSFS tracking bug. The > developers can work out the details amongst themselves on > opensfs-devel. The hope is only one bugzilla will exist for lustre. If the bugzilla goes away at oracle then we use the github issue tracker. If not someone can easily create a OpenSFS ticket at bugzilla.lustre.org. ORNL as well as livermore have their own trackers already there. > We have a volunteer who is willing to be the initial gatekeeper for > the OpenSFS 2.1 release if there are no objections, or unless better > candidates volunteer. Whoever the gatekeeper is needs to maintain the > fairly rigorous processes developed at Oracle to maintain quality and > reliability in both development and production branches. That is good news. If we go with the lustre tree on github that already exit then the gate keeper will have to be giving access to that tree. > Another idea which could be launched quickly is to put out an RFP to > do an investigation into Btrfs and its use as a possible backend file > system for Lustre. This wouldn’t involve any code development. The > primary deliverable for this contract would be a report. I can't believe it's not Btr :-> From dillowda at ornl.gov Fri Jan 7 15:44:23 2011 From: dillowda at ornl.gov (David Dillow) Date: Fri, 07 Jan 2011 10:44:23 -0500 Subject: [Twg] [Rpwg] Ideas for consideration In-Reply-To: <1294412769.21542.21.camel@bohr.ornl.gov> References: <1294412769.21542.21.camel@bohr.ornl.gov> Message-ID: <1294415063.6219.19.camel@lap75545.ornl.gov> On Fri, 2011-01-07 at 10:06 -0500, James A Simmons wrote: > > > 1) Create a branch named "2.0.59.0-opensfs" on github, which is based > > at Oracle's 2.0.59.0 tag. This is a "pre-release" version on the > > branch that will be tagged 2.1 by Oracle in the next few months. > > Or we could use > > https://github.com/rread/lustre > > We should avoid having multiple lustre branches on github. I disagree -- there should be a canonical OpenSFS source, independent of a particular person's tree. The beauty of git is its ability to pull from multiple trees and handle many branches. You have a point considering customer confusion, but I think that is solved by prominent linking to the "blessed" tree -- just as in the kernel, there is one main tree, and people can do their development in their own, possibly public, trees. > > 2) We create a "opensfs-devel" mailing list, where Lustre developers > > and testers can communicate and coordinate their efforts. > > Yes please do. My experince is that this list usually becomes the main > list where everyone post to. We talked about having one main mailing > list before. This could easily fill in that gap. I like the idea of open mailing lists, but I thought the idea was that we'd use lustre-devel as long is it remained viable? Given the rumblings it may make sense to open our own, but I worry about fragmenting the community. Look at OFA's ewg vs linux-rdma vs linux-kernel vs linux-scsi vs netdev. I have to be on three -- four for SRP and IPoIB -- to be reasonably sure I'm not missing relevant discussions. > > 3) When we discover bugs, we can initially use github's built-in > > issue tracking, or just use bugzilla.lustre.org, and agree to make all > > OpenSFS bugs dependants of a master OpenSFS tracking bug. The > > developers can work out the details amongst themselves on > > opensfs-devel. > > The hope is only one bugzilla will exist for lustre. If the bugzilla > goes away at oracle then we use the github issue tracker. If not someone > can easily create a OpenSFS ticket at bugzilla.lustre.org. ORNL as well > as livermore have their own trackers already there. As long as bugzilla.lustre.org remains alive, that is viable. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From morrone2 at llnl.gov Fri Jan 7 19:22:33 2011 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Fri, 07 Jan 2011 11:22:33 -0800 Subject: [Twg] [Rpwg] Ideas for consideration In-Reply-To: <1294415063.6219.19.camel@lap75545.ornl.gov> References: <1294412769.21542.21.camel@bohr.ornl.gov> <1294415063.6219.19.camel@lap75545.ornl.gov> Message-ID: <4D2767F9.2070105@llnl.gov> I'll just prefix this by saying that the original suggestions were devised before Oracle's latest changes. Once the dust settles, we can reevaluate some of the details. On 01/07/2011 07:44 AM, David Dillow wrote: > On Fri, 2011-01-07 at 10:06 -0500, James A Simmons wrote: >> >>> 1) Create a branch named "2.0.59.0-opensfs" on github, which is based >>> at Oracle's 2.0.59.0 tag. This is a "pre-release" version on the >>> branch that will be tagged 2.1 by Oracle in the next few months. >> >> Or we could use >> >> https://github.com/rread/lustre >> >> We should avoid having multiple lustre branches on github. > > I disagree -- there should be a canonical OpenSFS source, independent of > a particular person's tree. The beauty of git is its ability to pull > from multiple trees and handle many branches. > > You have a point considering customer confusion, but I think that is > solved by prominent linking to the "blessed" tree -- just as in the > kernel, there is one main tree, and people can do their development in > their own, possibly public, trees. I am with you on this. The blessed tree should not be under a single user's account. As long as that blessed/primary tree is well publicized, I see no problem with many people having their own clone of lustre on github or anywhere else. In fact, it can be quite useful. Someone developing a new feature can have their commits in their own branch, own repo, but still be public and allow everyone to see what they are doing. >>> 2) We create a "opensfs-devel" mailing list, where Lustre developers >>> and testers can communicate and coordinate their efforts. >> >> Yes please do. My experince is that this list usually becomes the main >> list where everyone post to. We talked about having one main mailing >> list before. This could easily fill in that gap. > > I like the idea of open mailing lists, but I thought the idea was that > we'd use lustre-devel as long is it remained viable? My thinking was that opensfs-devel would be to discuss the specifics of the opensfs release (which at the time, was based off of Oracle's cannonical release), things that the broader community would not care about. I too would prefer to keep general development discussions in one place at lustre-devel. From carrier at cray.com Tue Jan 11 21:10:38 2011 From: carrier at cray.com (John Carrier) Date: Tue, 11 Jan 2011 15:10:38 -0600 Subject: [Twg] FW: Hotel Reservation for Working Group Meeting at Hilton Chicago O'Hare Airport (O'Hare Terminal 2) - Check-in Jan 24, Check-out Jan 26 Message-ID: Here is the hotel information for any of you planning to attend the OpenSFS F2F meeting in Chicago on 1/25 & 1/26. The meeting starts at 8:30am on 1/25 in the hotel. Please let Shay know if you are planning to attend (shay at opensfs.org). --jc Sent: Tuesday, January 11, 2011 12:04 PM Subject: Hotel Reservation for Working Group Meeting at Hilton Chicago O'Hare Airport (O'Hare Terminal 2) - Check-in Jan 24, Check-out Jan 26 Hilton Chicago O'Hare Airport O'Hare International Airport, Chicago, Illinois, United States 60666 Tel: 1-773-686-8000 Fax: 1-773-601-2873 The Hilton Chicago O'Hare Airport hotel is ideally located in the heart of O'Hare International Airport. No need to worry about those early morning flights. This O'Hare airport hotel is located in Terminal 2 and is within walking distance to all O'Hare Airport's domestic terminals. When making reservations be sure to use code OSB: this will ensure that you get our discount rate and reservations are booked under the OpenSFS group block. The number to dial is 877-865-5322. Reservations can also be made by going to Hilton.com and putting OSB in the group/convention box. Group code: OSB Room Rate - $109 single Note: When this group block of rooms was originally set up the Hilton representative typed in "Open Scalable Bio Systems" as the group name and we have been unable to get this changed to "Open Scalable File Systems" even after lots of phone calls. So don't be surprised if the hotel operator/reservations person still has the wrong name. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssseager at gmail.com Wed Jan 12 16:02:47 2011 From: ssseager at gmail.com (Shay Seager) Date: Wed, 12 Jan 2011 08:02:47 -0800 Subject: [Twg] Wiki survey: profiles Message-ID: <561E329F-0C78-47DB-9C9F-476341C64085@gmail.com> Hello Working Groups! Jay and I have been working on Wikis for each working group so that each group will have a forum for communication, document storage, and management of tasks. We would love your feedback as we are creating this Wiki so that we can make sure to meet all of your needs. Because, not all of us have met in person or share contact information with the entire group, it was proposed to have brief profiles about each member under "the team" page on the wiki. This will give the group more familiarity with everyone in the group and contact information for everyone in the group. Because I understand that each person might feel differently about sharing his/her information I created a 1 question survey to see how much information we could post each member. Please respond to this survey and this e-mail with the contact information you feel comfortable disclosing. We will only disclose what you feel comfortable with. We are working on options to make this "only group viewable" so if you would feel comfortable with only that option please specify. Please go to this link for the survey http://www.surveymonkey.com/s/LLNRLZ9 Thank you for your consideration and participation, I know you are all very busy so I really appreciate you taking the time to do this! Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -The most important moment of your life is the one you are living right now - -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Wed Jan 12 19:38:55 2011 From: carrier at cray.com (John Carrier) Date: Wed, 12 Jan 2011 13:38:55 -0600 Subject: [Twg] 2011-01-06 meeting minutes Message-ID: My apologies for the delay posting these minutes. I was traveling all last week and am playing catch-up this week. I'm also including Shay's notes (in MS word .doc), which cover more of the flow of conversation. Please send any corrections or additions to the list. --jc -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 01/06/2011 Concall: start : 9:30a PT, end : 10:00a PT) Next meeting: Thursday, 1/13/2011 @ 9:30a PT/12:30p ET Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and password are 7012090. Attending: Name Organization email ----------------- -------------- ---------------------------- John Carrier Cray carrier at cray.com Justin Miller Indiana Univ. jupmille at indiana.edu Steve Simms Indiana Univ. ssimms at indiana.edu Marc Stearman LLNL marc at llnl.gov Chris Morrone LLNL morrone2 at llnl.gov Shay Seager OpenSFS shay.seager at opensfs.org Dave Dillow ORNL dillowda at ornl.gov Sarp Oral ORNL oralhs at ornl.gov Andreas Dilger Whamcloud adilger at whamcloud.com Eric Barton Whamcloud eeb at whamcloud.com Robert Reed Whamcloud rread at whamcloud.com Agenda * F2F meeting : Chicago: 1/25 8:30am - 1/26 12:00pm * requirements gathering * other business F2F meeting The meeting was originally meant to get the WG chairs together. All WG members are welcome to attend, but you should make a request to the board to make sure space is available before booking travel. At the meeting, we are going to review the list of items the OpenSFS working groups should be working on. The board would like the TWG to present a prioritized list of requirements, which we have gathered from OpenSFS members and other organizations in the Lustre HPC community. We should also expect discussion of which feature(s) the TWG will pursue for development. Requirements gathering Our goal is to reach out to the community to find requirements that we can use to create a long-term Lustre roadmap. Eric's slides are a starting point. The roadmap will lead to features that OpenSFS should consider developing. We need this process to be inclusive, but don't have the time to be exhaustive. We will listen to all suggestions, but need to acknowledge that not all will be feasible or can be met. We will accept suggestions at any time, but this initial search is intended to help set our long-term vision for the Lustre roadmap and identify some short-term projects that we can fund to move Lustre along the path. The following is a list of organizations we identified and the name of the TWG member who volunteered to contact the organization and present what they learn to the TWG. JohnC has the action of sending draft text to the reflector that we can all use to contact these organizations. Please post any additions to the list to the TWG reflector. Organization TWG contact ----------------- ---------------- Fujitsu andreasD NASA andreasD TACC andreasD Terascala daveD Bull ericB CEA ericB EuropeanOFS ericB (via JC) JSC ericB Mellanox ericB HPCFS johnC lustre-community johnC SGI johnC Xyratex johnC ARL marcS AWE marcS PNNL marcS SNL marcS CSCS sarpO TSC sarpO NRL simms TI-TECH ?? schedule 1/13 - accumulate / present requirement lists 1/20 - prioritize gathered requirements for F2F other business DaveD asked that people read Pam Hamilton's email "Ideas for consideration" and provide the requested feedback. Dave will resend the email to the TWG. -------------- next part -------------- A non-text attachment was scrubbed... Name: 1-6-10 TWG Con Call.doc Type: application/msword Size: 31232 bytes Desc: 1-6-10 TWG Con Call.doc URL: From dillowda at ornl.gov Wed Jan 12 22:41:52 2011 From: dillowda at ornl.gov (David Dillow) Date: Wed, 12 Jan 2011 17:41:52 -0500 Subject: [Twg] Sample text for requirements Message-ID: <1294872112.15826.49.camel@lap75545.ornl.gov> Here's some sample text for asking for requirements and development plans: The Technical Working Group of OpenSFS is working on the development roadmap for the organization. We are reaching out to the Lustre community for suggestions and pain points, as while there is a great deal of experience in the TWG, we are a small subset of the whole community and have no monopoly on good ideas. We would also like to know more of the communities plans for development to try to avoid overlap and duplicated effort. You can find some of the ideas discussed for the roadmap on the TWG list archives at -- in particular I would call your attention to Our current plan is to start discussing the requirements during our Jan 13th conference call, with more discussion and prioritization for the Jan 20th call. We hope to present some work-product from those calls to the OpenSFS executives during our Jan 25/26 face-to-face meeting. This will not be the last time we do this exercise -- new requirements will certainly arrive over time -- so there will be continuing opportunities to make your voice heard. -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From dillowda at ornl.gov Thu Jan 13 22:07:50 2011 From: dillowda at ornl.gov (David Dillow) Date: Thu, 13 Jan 2011 17:07:50 -0500 Subject: [Twg] RFP reflector Message-ID: <1294956470.3667.60.camel@lap75545.ornl.gov> To facilitate discussion about upcoming RFPs without generating a conflict of interest, I have asked for a reflector to be set up. This list will be limited to member organizations that will not be bidding on the RFPs. If you would like to take part in the RFP discussion, please add yourself to twg-rfp at http://lists.opensfs.org/listinfo.cgi/twg-rfp-opensfs.org Thanks! -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From dillowda at ornl.gov Thu Jan 13 22:18:45 2011 From: dillowda at ornl.gov (David Dillow) Date: Thu, 13 Jan 2011 17:18:45 -0500 Subject: [Twg] TWG meeting notes for Jan 13, 2011 Message-ID: <1294957125.3667.63.camel@lap75545.ornl.gov> OpenSFS Technical Working Group Meeting minutes for Jan 13, 2011 9:30a PST start of call 10:00a PST vendors drop off; RFP discussion with remaining participants 10:15a PST end of call Agenda Discuss Pam Hamilton's "Ideas for consideration" Requirements gathering update Discuss RFP Attending: Damian Hazen LBL dhazen at lbl.gov Marc Stearman LLNL marc at llnl.gov Chris Morrone LLNL morrone2 at llnl.gov Shay Seager OpenSFS shay at opensfs.org Galen Shipman ORNL gshipman at ornl.gov David Dillow ORNL dillowda at ornl.gov Andreas Dilger Whamcloud adilger at whamcloud.com Robert Reed Whamcloud rread at whamcloud.com Ideas for consideration There was a brief discussion regarding the suggestions put forth by Pam Hamilton from the support working group, with concurrence on exploring the BTRFS investigation. The other items had some brief discussion on the reflector and were not discussed on the call. Requirements gathering Most people were waiting on the template text to make their contact with their list of organizations. They will get their messages out today. Eric Barton, John Carrier, and Steve Simms were not on the call, so no status update on their contacts. Reiterated the importance of the TWG having a list of priorities and potential projects going into the face-to-face meeting in Chicago if we wish to achieve our goals. A high priority for Chicago is to leave with RFPs in hand, ready to put on the street. Would prefer to have text in hand for our top items going into the F2F. Asked that members send their priority lists to the reflector, as we may not get much feed back from our contacts due to short notice. We then had a brief recap of IU and ORNL's priorities, and added a few. Alphabetically: IU -- Improved error reporting ORNL -- Online filesystem integrity checks LBL -- Integrated performance monitoring LLNL -- Completing OSD restructuring/kDMU work LLNL -- Size on MDS Multi -- BTRFS investigation Multi -- Imperative recovery Multi -- Improved admin/mgmt tools Multi -- Improved metadata performance SGI? -- LNET Channel Bonding RFP discussion Vendors were asked to leave so we could discuss items relating to the RFP. Members to send item text to Galen, will send around draft to member organizations _not_ interested in bidding. List is at http://lists.opensfs.org/listinfo.cgi/twg-rfp-opensfs.org -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From dillowda at ornl.gov Wed Jan 19 21:09:53 2011 From: dillowda at ornl.gov (David Dillow) Date: Wed, 19 Jan 2011 16:09:53 -0500 Subject: [Twg] Requirements Message-ID: <1295471393.6105.28.camel@lap75545.ornl.gov> If anyone has requirements for their organization or from their community contacts, please post them to the list. Thank you! -- Dave Dillow National Center for Computational Science Oak Ridge National Laboratory (865) 241-6602 office From adilger at whamcloud.com Wed Jan 19 22:16:29 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 19 Jan 2011 15:16:29 -0700 Subject: [Twg] Requirements In-Reply-To: <1295471393.6105.28.camel@lap75545.ornl.gov> References: <1295471393.6105.28.camel@lap75545.ornl.gov> Message-ID: On 2011-01-19, at 14:09, David Dillow wrote: > If anyone has requirements for their organization or from their > community contacts, please post them to the list. Unfortunately, I haven't gotten any replies to the emails that I sent out. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From carrier at cray.com Thu Jan 20 17:26:56 2011 From: carrier at cray.com (John Carrier) Date: Thu, 20 Jan 2011 11:26:56 -0600 Subject: [Twg] Cray roadmap priorities Message-ID: The following are Cray's roadmap priorities: 1) larger LUNs Needed for Lustre 1.8.x (both RHEL, SLES) as well as 2.x (RHEL-based only). 2) metadata performance General improvements needed for Lustre 1.8.x (RHEL, SLES) as well as Lustre 2.x. Clustered Metadata is an option to improve scalability for Lustre 2.x. 3) end-to-end data integrity Improve the reliability and resiliency of the file system by protecting data from the client to disk. (Lustre 2.x only) 4) online file system check Scrub metadata in the background, capture errors before seen by the application, and reduce recovery time when errors are found. (Lustre 2.x only) 5) interface to HSM Seemlessly move data from Lustre OSTs to archival storage. (Lustre 2.x only) 6) channel bonding Provide redundant, active LNET links between routers and servers to increase LNET reliability and improve performance between nodes. (Lustre 2.x only) From avidwansa at ddn.com Thu Jan 20 18:06:29 2011 From: avidwansa at ddn.com (Atul Vidwansa) Date: Thu, 20 Jan 2011 10:06:29 -0800 Subject: [Twg] Requirements In-Reply-To: References: <1295471393.6105.28.camel@lap75545.ornl.gov> Message-ID: <765768D8E686B84BACC0EEE5FEB80DCE5480F26E99@MAILBOXCLUSTER.datadirect.datadirectnet.com> Here are some requirements I captured during my discussion with customers: 1. Replication on network: Ability to replicate data as per user policies. User should be able to specify how many copied of certain data he wants. Simplest form is to do RAID1 like mirroring along with data striping. 2. Ease of management: A central console like utility that can show status of clients and servers, performance data, availability, ability to drill down from logical to physical components to check health, plugins for integrating with Nagios/Ganglia. Letter versions of such tool cold provice features like storage and user accounting, ability to track filesystem uptime. 3. Streamlined patching mechanism: Lustre currently has ad-hoc patching mechanism with no scope of dependency checking. A mechanism to fetch patches from central repository, check dependency, ability to apply and rollback, easy upgrades, ability to display currently applied patches etc 4. Built in failover: No dependency on external packages like heartbeat/pacemaker, ability to have more than 2 nodes in failover group, quick failover of node, network or services. 5. Better diagnostic messages: Log messages that tell system administrator what is wrong and further course of action. No cryptic messages. 6. Better use of flash devices: Either as intermediate cache or for increasing disk rebuild times. Cheers, _Atul -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of Andreas Dilger Sent: Thursday, 20 January 2011 3:46 AM To: David Dillow Cc: twg at lists.opensfs.org Subject: Re: [Twg] Requirements On 2011-01-19, at 14:09, David Dillow wrote: > If anyone has requirements for their organization or from their > community contacts, please post them to the list. Unfortunately, I haven't gotten any replies to the emails that I sent out. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From ssseager at gmail.com Fri Jan 21 22:16:53 2011 From: ssseager at gmail.com (Shay Seager) Date: Fri, 21 Jan 2011 14:16:53 -0800 Subject: [Twg] wiki - please review Message-ID: <2795FC48-462F-4F6D-A413-35EDA1C613CD@gmail.com> Hello all, The TWG wiki is up and ready to be edited. Ideas for possible use: -message boards -posting and prioritizing priorities -calendar of events The content of the site still needs to be edited so if anyone wants to send me content or work on the site just e-mail me your requests. I will post this site to the website as soon as it's content is ready. https://sites.google.com/site/opensfstechnicalworkinggroup/ Thank you! Shay From carrier at cray.com Wed Jan 26 21:51:37 2011 From: carrier at cray.com (John Carrier) Date: Wed, 26 Jan 2011 15:51:37 -0600 Subject: [Twg] meeting reminder 1/27/2011 Message-ID: Hi all, This is a reminder of our meeting tomorrow 1/27 at 9:30a PT / 12:30 ET. Dial-in numbers are 715-726-4994 or 866-304-8294. The meeting ID and password are 7012090. Dave and I will share what we learned at the F2F meeting this week. We received feedback on the TWG whitepaper and directions on how the TWG should pursue the OpenSFS Lustre roadmap. I will send out notes before the call. Please let us know if you have any other agenda items you would like to discuss. --jc From carrier at cray.com Thu Jan 27 17:24:52 2011 From: carrier at cray.com (John Carrier) Date: Thu, 27 Jan 2011 11:24:52 -0600 Subject: [Twg] Lustre Requirements and Roadmap Message-ID: Below is my summary of the outcome from yesterday's meeting at the OpenSFS F2F. I would like to use this as the basis of our meeting today. --jc Summary of F2F Requirements Discussion (1/26/2011) -------------------------------------------------- In the last few weeks, the TWG members gathered the following requirements from within their organizations and the Lustre community: - Improved error reporting - Online filesystem integrity checks - Integrated performance monitoring - Completing OSD restructuring/kDMU work - Size on MDS - BTRFS investigation - Imperative recovery - Improved admin/mgmt tools - Improved metadata performance - LNET Channel Bonding - end-to-end data integrity - interface to HSM Further discussion within the TWG organized these requirements into two high priority categories: * improve metadata performance * improve scalability and reliability of backend file system At our 1/20 meeting, TWG members discussed several options to meet these goals: * metadata performance - clustered metadata servers (CMD) - network request scheduler (NRS) - RPC aggregation - SMP scaling - subtree lockings - size on MDS * backend storage - OSD restructuring - online fsck - btrfs evaluation Much of the design and implementation for these features was started by Sun/Oracle. At the F2F meeting in Chicago, the OpenSFS board decided that there is an opportunity to acknowledge any technical debt in these initial designs and requested that, instead of an RFI for these features, the TWG specify our long-term requirements for Lustre and then engage the Lustre community to discuss architectures to meet these requirements. Therefore, the TWG will spend the next two meetings refining the requirements for metadata and backend storage. We will then spend the following two weeks leading discussions with the Lustre community to define the architecture and features that will meet the requirements. >From these discussions, the TWG will create a roadmap that OpenSFS can use to direct feature development for Lustre 2.2 and beyond. The following is an incomplete list of requirements to motivate further discussion : * metadata performance GOAL: improve file system scalability and interactive performance requirements: min max - # files in file system 100 billion 1 trillion - # files in directory 50 million 10 billion - file creates / sec 100 thousand 30 thousand (aggregate) (single client) - directory lisings / sec - open files per process - 100 thousand - file system capacity 30 PB 100 PB - # clients 30 thousand ?00 thousand * backend storage GOAL: provide reliable, scalable backing store for Lustre servers requirements: - large LUNs (min 32 TB) - end-to-end data integrity (T10 PI or equivalent) - no performance impact for file system repair - framework to enable alternatives to ldiskfs - direct I/O mode - ?? From avidwansa at ddn.com Thu Jan 27 20:44:10 2011 From: avidwansa at ddn.com (Atul Vidwansa) Date: Thu, 27 Jan 2011 12:44:10 -0800 Subject: [Twg] Lustre Requirements and Roadmap In-Reply-To: References: Message-ID: <765768D8E686B84BACC0EEE5FEB80DCE5481051041@MAILBOXCLUSTER.datadirect.datadirectnet.com> Thanks John for the detailed notes. I just have a comment about large LUN support. We should plan for having support for LUNs much larger than 32TB. Looking at the way disk size is growing, we will soon be struggling if we plan for 32TB only. Thanks, -Atul -----Original Message----- From: twg-bounces at lists.opensfs.org [mailto:twg-bounces at lists.opensfs.org] On Behalf Of John Carrier Sent: Thursday, 27 January 2011 10:55 PM To: twg at lists.opensfs.org Subject: [Twg] Lustre Requirements and Roadmap Below is my summary of the outcome from yesterday's meeting at the OpenSFS F2F. I would like to use this as the basis of our meeting today. --jc Summary of F2F Requirements Discussion (1/26/2011) -------------------------------------------------- In the last few weeks, the TWG members gathered the following requirements from within their organizations and the Lustre community: - Improved error reporting - Online filesystem integrity checks - Integrated performance monitoring - Completing OSD restructuring/kDMU work - Size on MDS - BTRFS investigation - Imperative recovery - Improved admin/mgmt tools - Improved metadata performance - LNET Channel Bonding - end-to-end data integrity - interface to HSM Further discussion within the TWG organized these requirements into two high priority categories: * improve metadata performance * improve scalability and reliability of backend file system At our 1/20 meeting, TWG members discussed several options to meet these goals: * metadata performance - clustered metadata servers (CMD) - network request scheduler (NRS) - RPC aggregation - SMP scaling - subtree lockings - size on MDS * backend storage - OSD restructuring - online fsck - btrfs evaluation Much of the design and implementation for these features was started by Sun/Oracle. At the F2F meeting in Chicago, the OpenSFS board decided that there is an opportunity to acknowledge any technical debt in these initial designs and requested that, instead of an RFI for these features, the TWG specify our long-term requirements for Lustre and then engage the Lustre community to discuss architectures to meet these requirements. Therefore, the TWG will spend the next two meetings refining the requirements for metadata and backend storage. We will then spend the following two weeks leading discussions with the Lustre community to define the architecture and features that will meet the requirements. >From these discussions, the TWG will create a roadmap that OpenSFS can use to direct feature development for Lustre 2.2 and beyond. The following is an incomplete list of requirements to motivate further discussion : * metadata performance GOAL: improve file system scalability and interactive performance requirements: min max - # files in file system 100 billion 1 trillion - # files in directory 50 million 10 billion - file creates / sec 100 thousand 30 thousand (aggregate) (single client) - directory lisings / sec - open files per process - 100 thousand - file system capacity 30 PB 100 PB - # clients 30 thousand ?00 thousand * backend storage GOAL: provide reliable, scalable backing store for Lustre servers requirements: - large LUNs (min 32 TB) - end-to-end data integrity (T10 PI or equivalent) - no performance impact for file system repair - framework to enable alternatives to ldiskfs - direct I/O mode - ?? _______________________________________________ twg mailing list twg at lists.opensfs.org http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org From morrone2 at llnl.gov Thu Jan 27 21:56:34 2011 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Thu, 27 Jan 2011 13:56:34 -0800 Subject: [Twg] Lustre Requirements and Roadmap In-Reply-To: References: Message-ID: <4D41EA12.4030809@llnl.gov> John, Thanks for creating that list! I think that for the metadata performance, we should use labels other than "min" and "max". After all, we would never say "no, we won't accept a filesystem that is capable of more than than this". :) Also, the requirements are only useful if we can give target dates for the requirements. So maybe the "min" column becomes Q3/2012, and the max column Q1/2014. I am just pulling those dates out of thin air, they could be something else. I think we should split the aggregate and single client file creates/sec requirements into two separate requirement lines. For backend storage: I think that we should take out the "framework to enable alternatives to ldiskfs" line. That is a design decision rather than a requirement. I think it is certainly necessary given the requirements, but if we are trying to distinguish between requirements and architectural decisions, that line should be removed. Rather than "no performance impact", say "low performance impact". Zero impact is probably impossible. :) At some point we need to be clear on exactly what quilifies as "low". One part of that might be: - The solution must perform filesystem integrity checking and repair on-line. I.E., There may be no required downtime just for consistence checking and repair. What is the rationale for the "direct I/O mode" requirement? Not that I am opposed to it...but is that really a requirement, or a solution? For end-to-end integrity, I would suggest that T10 PI is not sufficient, since it only concerns itself with the path between the HBA and the backend storage (disks/flash). I think that we need to define where the "ends" are, and they should really extend up into memory on the lustre servers. T10 DIF + Oracle's DIX might be a better example. Examples are not necessarily what we want here. Chris On 01/27/2011 09:24 AM, John Carrier wrote: > The following is an incomplete list of requirements to motivate further > discussion : > > * metadata performance > > GOAL: improve file system scalability and interactive > performance > > requirements: min max > - # files in file system 100 billion 1 trillion > - # files in directory 50 million 10 billion > - file creates / sec 100 thousand 30 thousand > (aggregate) (single client) > - directory lisings / sec > - open files per process - 100 thousand > - file system capacity 30 PB 100 PB > - # clients 30 thousand ?00 thousand > > > * backend storage > > GOAL: provide reliable, scalable backing store for > Lustre servers > > requirements: > - large LUNs (min 32 TB) > - end-to-end data integrity (T10 PI or equivalent) > - no performance impact for file system repair > - framework to enable alternatives to ldiskfs > - direct I/O mode > - ?? > > > > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org > . > From adilger at whamcloud.com Thu Jan 27 22:41:54 2011 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 27 Jan 2011 15:41:54 -0700 Subject: [Twg] Lustre Requirements and Roadmap In-Reply-To: <4D41EA12.4030809@llnl.gov> References: <4D41EA12.4030809@llnl.gov> Message-ID: <5D70CFA1-EC75-42E2-B161-B7F20C9BE197@whamcloud.com> On 2011-01-27, at 14:56, Christopher J. Morrone wrote: > I think that for the metadata performance, we should use labels other than "min" and "max". After all, we would never say "no, we won't accept a filesystem that is capable of more than than this". :) > > Also, the requirements are only useful if we can give target dates for the requirements. So maybe the "min" column becomes Q3/2012, and the max column Q1/2014. I am just pulling those dates out of thin air, they could be something else. The other thing that is always difficult about specifying performance requirements for Lustre, is that these numbers cannot be specified in the absence of some idea about the system on which the tests are expected to run. > I think we should split the aggregate and single client file creates/sec requirements into two separate requirement lines. > > For backend storage: > > I think that we should take out the "framework to enable alternatives to ldiskfs" line. That is a design decision rather than a requirement. I think it is certainly necessary given the requirements, but if we are trying to distinguish between requirements and architectural decisions, that line should be removed. > > Rather than "no performance impact", say "low performance impact". Zero impact is probably impossible. :) At some point we need to be clear on exactly what quilifies as "low". I totally agree with this, and have never liked the "no performance impact" term. It essentially means that you are throwing away performance in the non-degraded case, just so that the degraded case does not appear to have any performance impact. > One part of that might be: > > - The solution must perform filesystem integrity checking and repair > on-line. > > I.E., There may be no required downtime just for consistence checking and repair. > > What is the rationale for the "direct I/O mode" requirement? Not that I am opposed to it...but is that really a requirement, or a solution? > > For end-to-end integrity, I would suggest that T10 PI is not sufficient, since it only concerns itself with the path between the HBA and the backend storage (disks/flash). That is not necessarily true. T10 PI could likely be integrated into the Lustre integrity checking, in the same manner that was proposed for the ZFS checksums in the HPCS design, so that it goes all the way to the memory of the client. One major issue that has been observed with T10 PI in Linux (even for local filesystems) is that the VM/VFS allow data blocks to be modified while IO is in flight (in particular for MMAP files), and that can cause spurious data integrity failures. Lustre works around this today in its network checksum implementation, but the only sure-fire way to handle it is to either copy the data before it is sent over the network, or to block writes to pages that are currently in RPCs, both of which can have significant performance impact. Fortunately, this is not a common use case for HPC applications. > I think that we need to define where the "ends" are, and they should really extend up into memory on the lustre servers. > On 01/27/2011 09:24 AM, John Carrier wrote: > >> The following is an incomplete list of requirements to motivate further >> discussion : >> >> * metadata performance >> >> GOAL: improve file system scalability and interactive >> performance >> >> requirements: min max >> - # files in file system 100 billion 1 trillion >> - # files in directory 50 million 10 billion >> - file creates / sec 100 thousand 30 thousand >> (aggregate) (single client) >> - directory lisings / sec >> - open files per process - 100 thousand >> - file system capacity 30 PB 100 PB >> - # clients 30 thousand ?00 thousand >> >> >> * backend storage >> >> GOAL: provide reliable, scalable backing store for >> Lustre servers >> >> requirements: >> - large LUNs (min 32 TB) >> - end-to-end data integrity (T10 PI or equivalent) >> - no performance impact for file system repair >> - framework to enable alternatives to ldiskfs >> - direct I/O mode >> - ?? >> >> >> >> >> _______________________________________________ >> twg mailing list >> twg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org >> . >> > > _______________________________________________ > twg mailing list > twg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/twg-opensfs.org Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. From morrone2 at llnl.gov Fri Jan 28 01:46:13 2011 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Thu, 27 Jan 2011 17:46:13 -0800 Subject: [Twg] Lustre Requirements and Roadmap In-Reply-To: <5D70CFA1-EC75-42E2-B161-B7F20C9BE197@whamcloud.com> References: <4D41EA12.4030809@llnl.gov> <5D70CFA1-EC75-42E2-B161-B7F20C9BE197@whamcloud.com> Message-ID: <4D421FE5.2030704@llnl.gov> On 01/27/2011 02:41 PM, Andreas Dilger wrote: > On 2011-01-27, at 14:56, Christopher J. Morrone wrote: >> For end-to-end integrity, I would suggest that T10 PI is not sufficient, since it only concerns itself with the path between the HBA and the backend storage (disks/flash). > > That is not necessarily true. T10 PI could likely be integrated into the Lustre integrity checking, in the same manner that was proposed for the ZFS checksums in the HPCS design, so that it goes all the way to the memory of the client. Perhaps I was nit-picking on that one. You can certainly do that with DIX+T10 PI. But I think that technically T10 PI only covers HBA to storage, and DIX (created by Oracle) defines the interfaces between an "application" an the HBA. I retract that complaint. Chris From Nathan_Rutman at xyratex.com Sat Jan 29 04:09:25 2011 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Fri, 28 Jan 2011 20:09:25 -0800 Subject: [Twg] Lustre Requirements and Roadmap In-Reply-To: <5D70CFA1-EC75-42E2-B161-B7F20C9BE197@whamcloud.com> References: <4D41EA12.4030809@llnl.gov> <5D70CFA1-EC75-42E2-B161-B7F20C9BE197@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92101D11F8B@XYUS-EX22.xyus.xyratex.com> On Jan 27, 2011, at 2:41 PM, Andreas Dilger wrote: > On 2011-01-27, at 14:56, Christopher J. Morrone wrote: >> I think that for the metadata performance, we should use labels other than "min" and "max". After all, we would never say "no, we won't accept a filesystem that is capable of more than than this". :) >> >> Also, the requirements are only useful if we can give target dates for the requirements. So maybe the "min" column becomes Q3/2012, and the max column Q1/2014. I am just pulling those dates out of thin air, they could be something else. > > The other thing that is always difficult about specifying performance requirements for Lustre, is that these numbers cannot be specified in the absence of some idea about the system on which the tests are expected to run. I think this is key. 30k creates/sec on a 5400 rpm spindle, or on a SSD on some heavy hardware? It probably makes sense to be pretty flexible here, but we should set some limits (i.e. probably not any custom silicon). Maybe do that by setting a price cap on the hardware? ______________________________________________________________________ This email may contain privileged or confidential information, which should only be used for the purpose for which it was sent by Xyratex. No further rights or licenses are granted to use such information. If you are not the intended recipient of this message, please notify the sender by return and delete it. You may not use, copy, disclose or rely on the information contained in it. Internet email is susceptible to data corruption, interception and unauthorised amendment for which Xyratex does not accept liability. While we have taken reasonable precautions to ensure that this email is free of viruses, Xyratex does not accept liability for the presence of any computer viruses in this email, nor for any losses caused as a result of viruses. Xyratex Technology Limited (03134912), Registered in England & Wales, Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA. The Xyratex group of companies also includes, Xyratex Ltd, registered in Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia) Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in The People's Republic of China and Xyratex Japan Limited registered in Japan. ______________________________________________________________________