From hamilton5 at llnl.gov Wed Jun 1 23:17:44 2011 From: hamilton5 at llnl.gov (Hamilton, Pam) Date: Wed, 1 Jun 2011 16:17:44 -0700 Subject: [cdwg] OpenSFS Community Development WG call Message-ID: When: Occurs every 2 weeks on Wednesday effective 6/8/2011 from 9:00 AM to 10:00 AM (GMT-08:00) Pacific Time (US & Canada). Where: Call-in: 866-914-3976 (925-424-8105) Passcode: 534986# Note: The GMT offset above does not reflect daylight saving time adjustments. *~*~*~*~*~*~*~*~*~* Hi all, This is the bi-weekly conference call of the OpenSFS Community Development Working Group. This working group was formed to facilitate a discussion about a community development model for Lustre, taking into consideration the needs/concerns of Lustre vendors. For this to be successful, it's important to have vendor participation. Regards, Pam -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/calendar Size: 2325 bytes Desc: not available URL: From hamilton5 at llnl.gov Wed Jun 1 23:36:51 2011 From: hamilton5 at llnl.gov (Hamilton, Pam) Date: Wed, 1 Jun 2011 16:36:51 -0700 Subject: [cdwg] OpenSFS Community Development WG news Message-ID: Hi all, I've sent a meeting proposal for a bi-weekly OpenSFS Community Development WG conference call. Attached are the notes from our first call held on 5/25. Thanks go to Shay for taking notes. Please send along any updates you would like to see in them because taking verbatim notes on conference calls can be challenging. Here is (my) short summary of our discussion with regard to an interim solution for the Lustre infrastructure while we work on a development model for the long term. _______________________________________ Path Forward In an initial discussion, it was acknowledged that any sort of community development model would not be ready in time for Lustre v2.2 and possibly not even v2.3. A proposal was put forth for an interim solution which would allow time for a long term model to be thoroughly vetted with the Lustre community and vendors. Interim Solution A singular concern within the community is that the Lustre infrastructure (e.g code respository, bug tracking/history, etc) not be solely held by one vendor. This concern stems from not wanting to repeat what happened when Oracle decided to drop its support of Lustre. The interim solution proposed is to have OpenSFS host an independent mirroring of the Lustre infrastructure and archive it so if the worst case ever did happen Lustre can still move forward without an interruption of service. With the mirroring in place, the community can take the time necessary to develop a model for the long term which is sustainable and takes into consideration the needs/concerns of the Lustre vendors. We next need to identify a host site for the mirror. Comments/suggestions are welcome. Please plan to join the next call on Wednesday, June 8, at 9am Pacific (Call-in: 866-914-3976 (925-424-8105) Passcode: 534986#). Regards, Pam ___________________________________ Pam Hamilton Lawrence Livermore National Lab P.O. Box 808, L-556 Livermore, CA  94551-9900 E-Mail:  pgh at llnl.gov Phone:  925-423-1332          Fax:  925-423-8719 -------------- next part -------------- A non-text attachment was scrubbed... Name: CDWG5-25-11.doc Type: application/msword Size: 31744 bytes Desc: CDWG5-25-11.doc URL: From carrier at cray.com Wed Jun 8 16:23:34 2011 From: carrier at cray.com (John Carrier) Date: Wed, 8 Jun 2011 11:23:34 -0500 Subject: [cdwg] FW: [lustre-devel] Orion Landing Plan/Process Message-ID: In case any of you missed this on lustre-devel. -----Original Message----- From: lustre-devel-bounces at lists.opensfs.org [mailto:lustre-devel-bounces at lists.opensfs.org] On Behalf Of Andreas Dilger Sent: Tuesday, June 07, 2011 12:03 PM To: lustre-devel Mailing List; lustre-devel at lists.opensfs.org Subject: [lustre-devel] Orion Landing Plan/Process This (long) email is of primary interest to developers and testers of Lustre, and is focussed on how to integrate the substantial amount of development that has been done on the Orion development branch into the mainline Lustre release branch in a stable manner, without seriously disrupting other ongoing development projects. Introduction ============ The Lustre 2.0 release introduced a new Object Storage Device (OSD) interface, which improves the abstraction of the Lustre IO operations from the underlying filesystem implementation. Moving to the new OSD interface allows Lustre to start using more advanced back-end filesystems like ZFS, and Btrfs in the future. For Lustre 2.0, the MDS stack was largely moved over to use the new OSD interface, but some parts of the MDS code (llogs for distributed recovery MDS->OST RPCs, etc) are still using the older "obd/lvfs/fsfilt" MDS->interface, which has left unwanted complexity in the code. As well, none of the code for the OSS or MGS have been moved to use this new OSD interface, leaving a large amount of duplicated code within Lustre, and sometimes confusion for developers about which IO methods were in use for a particular operation. Moving everything over to use the OSD interface will allow this duplicated code to be removed, and facilitates development projects like Distributed Namespace (formerly CMD), unified targets (i.e. small files on the MDT), and others in the future. The Orion project at Whamcloud, in conjunction with LLNL, is focussed on completing the restructuring of the Lustre code base to use the OSD interface. LLNL will begin using the Orion branch for their Sequoia system (https://asc.llnl.gov/computing_resources/sequoia/) with ZFS (http://zfsonlinux.org/lustre.html) as the back-end filesystem. It is expected that the Orion project will take about a year to complete. Ongoing Development =================== The existing Orion codebase represents a significant amount of development that has already been done to update the Lustre server code to use the OSD API. Work is ongoing to complete the transition of the OSS, MGS, and recovery code to use the OSD interface. The large amount of change in this branch presents a serious obstacle to integration of this code into the mainline Lustre codebase. Directly landing all of the branch to master would present a significant risk of destabilizing the master branch. Even with significant pre-landing testing on the "orion" branch, it will still only be a fraction of the different real-world load and environment combinations that are being tested by different members of the community. As well, debugging any problems that appear after a large single landing would be very difficult and time consuming. Proposed Landing Process ======================== What we propose is to split the current changes in the Orion branch into a series of smaller commits to the "master" branch over several months, each of which is only changing a specific part of the code. All of the commits will provide stand-alone functionality, that will be pre-tested in isolation before landing to meet the quality standards of the master branch. The benefits of making a series of independent commits spread over several months are manyfold: - each commit can be tested separately, both in advance of landing, and after integration, to isolate defects to the specific areas of the code that have been changed - testing can be more extensive and focussed on the code being changed - defects in smaller changes are easier to find and fix during pre-landing testing and are easier to isolate after landing - in the unlikely case of serious defects appearing after landing, the offending patch(es) can be backed out without forcing all of the unrelated changes from orion to be backed out as well - the commits will be isolated to a specific area of code or API, and will be grouped by logical change, rather than the more unordered sequence of changes and bug fixes from the ongoing development - spacing major changes over a longer time period it will give other developers (both inside and outside Whamcloud) more time to become aware of the changes in the Orion branch, and adapt their projects to use the changes being made - landing parts of the orion branch early means less code is developed that is in conflict with these changes, and less work will need in both the orion and other development branches to merge those changes - smaller commits can be inspected more easily by developers, hopefully getting more eyes on the changes being made, and finding bugs earlier As isolated changes are being extracted from the orion development branch, they will be inspected, tested to the standard of the master branch, and then landed to the master branch. The orion branch will then be rebased against the updated master branch, and the landed changes will be removed from the outstanding changes on orion. This process is shown in the attached diagram, and can also be used for larger features unrelated to the orion branch. This development and code contribution model has served the Linux kernel community very well to manage integration of large features. At this stage, this email is focussed on raising awareness of these plans within the Lustre community. A separate email detailing the plans for landing specific changes will be sent at a later date. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. -------------- next part -------------- A non-text attachment was scrubbed... Name: Diagram 1.0.png Type: image/png Size: 68288 bytes Desc: Diagram 1.0.png URL: From noreply at googlegroups.com Thu Jun 9 17:33:13 2011 From: noreply at googlegroups.com (noreply at googlegroups.com) Date: Thu, 09 Jun 2011 17:33:13 +0000 Subject: [cdwg] Google Groups: You've been invited to CDWG Message-ID: <20110609173723.20A8B9FC09B@zapata.dreamhost.com> Shay ssseager at gmail.com has invited you to join the CDWG group with this message: Hello CDWG, Here is the google groups page for this group. Please post documents here for review. Weekly captures of e-mail reflector conversations will also be posted here. If you have any questions/comments/concerns you can contact shay at opensfs. org. Thank you for your participation, Shay Here is the group's description: Community Development Working Group ---------------------- Google Groups Information ---------------------- You can accept this invitation by clicking the following URL: http://groups.google.com/group/communitydevelopmentworkinggroup/sub?s=il7c8xQAAADIAuTc5C7Ogx0wJ7HxmDzKat0s4rx0b7wtyGYM6xnx0w&hl=en --------------------- If This Message Is Unwanted --------------------- If you feel that this message is abuse, please inform the Google Groups staff by using the URL below. http://groups.google.com/groups/abuse?invite=YQAAAAP-Wh1rAAAA1-idBk4AAABWR1AKm89syq0FYFsssgWkSX8as3M&hl=en From hamilton5 at llnl.gov Mon Jun 20 15:39:19 2011 From: hamilton5 at llnl.gov (Hamilton, Pam) Date: Mon, 20 Jun 2011 08:39:19 -0700 Subject: [cdwg] OpenSFS Community Development Working Group Notes for 6/8/11 Message-ID: Hi all, Attached are the notes from the last Community Development Working Group conference call held on June 8th. Our next call is this Wednesday, June 22nd, at 9am Pacific. Visit the OpenSFS web site at http://www.opensfs.org/?page_id=89 for information about the call. Below are the Action Items from the last call. Stay tuned for the "straw-man" definition of what it takes to support a canonical release. Regards, Pam ACTION ITEMS ● IU is going to host the Lustre mirror ● Pam Hamilton is going to take the idea of what resources are required to support a canonical release and create a “straw-man” definition for the community. This definition will be the basis for a discussion about creating a sustainable community development model. ● The group will plan and host a “deep-dive” developers session in mid-September 2011. Event will be hosted at a Hub airport. Those not able to attend can participate on-line through a podcast and call-in number. The pod-cast will be recorded for future viewing. ● The group needs to decide if they want to define proper release collateral. ● Kevin Canady will send out some internal documents from Xyratex to the mailing list and post on the google groups page. ● Shay Seager will create a google groups page for the CDWG. (DONE – Thanks, Shay!!) ● The group is going to carry on conversation in the e-mail reflector. Shay will capture and put the e-mail conversation on the google groups page. Documents for group review will be on the google groups page. Some discussion will be on the google groups page, however, the group still needs to define a communication tool to reach developers. ● Neutral repository group will merge with this group. ___________________________________ Pam Hamilton Lawrence Livermore National Lab P.O. Box 808, L-556 Livermore, CA 94551-9900 E-Mail: pgh at llnl.gov Phone: 925-423-1332 Fax: 925-423-8719 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: CDWG6-8-11_pam.doc Type: application/msword Size: 42496 bytes Desc: CDWG6-8-11_pam.doc URL: From hamilton5 at llnl.gov Tue Jun 21 01:04:22 2011 From: hamilton5 at llnl.gov (Hamilton, Pam) Date: Mon, 20 Jun 2011 18:04:22 -0700 Subject: [cdwg] Community Development Model straw-man Message-ID: Hi all, Attached is the beginning of a white paper for the CDWG. Some of it is just rehash but the latter half which is included below should fuel our next round of discussions. Please respond to the mail list with thoughts or be prepared to discuss them on Wednesday's call at 9am Pacific (Call-in: 866-914-3976 (925-424-8105) Passcode: 534986#). Regards, Pam Community Development Model Canonical Release Support Before a discussion can take place regarding a community development model, we must define what resources are involved with supporting a canonical release. Based on information from former Oracle staff that supported Lustre, the breakdown of resources necessary to manage the Lustre infrastructure and prepare canonical Lustre releases can be defined as follows: * Gatekeeping (1 - 2 FTEs) * Testers (3 - 4 FTEs) * Release Manager (1 FTE) * Admin support (.5 FTE) * Documentation (.25 FTE) * Dedicated testing equipment (~150K / year) - Tech refresh, power, space, cooling Total costs can approach $2M US/year . How do we pay for this? The Lustre community has to decide how to pay for an activity which in the recent past has been subsidized by vendors such as Oracle. Some options might include: * Support contracts with a vendor * Overhead cost on integrated appliances * Volunteer resources * Community organization dues The Community Development Working Group must discuss these and possibly other options before it can reasonably state OpenSFS is in a position to produce any canonical Lustre release. ___________________________________ Pam Hamilton Lawrence Livermore National Lab P.O. Box 808, L-556 Livermore, CA 94551-9900 E-Mail: pgh at llnl.gov Phone: 925-423-1332 Fax: 925-423-8719 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenSFS CDWG white paper_ver1.docx Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document Size: 21276 bytes Desc: OpenSFS CDWG white paper_ver1.docx URL: