From morrone2 at llnl.gov Wed Sep 4 04:53:16 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 03 Sep 2013 21:53:16 -0700 Subject: [cdwg] pending review list In-Reply-To: References: Message-ID: <5226BCBC.9000106@llnl.gov> On 08/29/2013 06:10 AM, Denis Kondratenko wrote: > Hi All, [cut] > What conclusions should be done, numbers like: > 31, 26, 150, 180, 120, 30, 70, 120, 112 > indicate issues. > > Problem stages are: no update from engineer, no review. I think we could have all agreed to that even without specific examples. :) But I think I should clarify that "no update from engineer" means "no update from the patch submitter". [cut] > So I think you probably understood the way. We need stat numbers and not > individual check for every ticket. I am not sure that I understand your point about needing more statistics. I think the software developers all knew the major slow spots in the patch submission process without seeing these selected patch examples. I think we are all in agreement: patch submission is sometimes a long process. But it is not clear to me how additional statistics can solve, or really even help, with that problem. I think what your numbers show is that a patch submitter must remain an active participant in the process. If a submitter does not participate actively, the process can, and frequently will, take longer than is necessary. > We might be need some automatic notification for stages longer than 1-2 > weeks (what ever we will define). I think that such a notification would probably be more annoying than useful. If I want to know the state of my patches, I can pull up gerrit at any time and learn their current status. I also get notification email any time my patches are reviewed, someone comments on the patch, someone comments in jira, etc. In my opinion there are plenty of existing notifications, and I personally don't really want any additional notifications unless they provide new information. Further, how could we possibly pick a number that makes sense? Patches sitting in the review stage may do so for a whole host of reasons, many that are perfectly reasonable. For instance, a new feature that is submitted after feature freeze is fairly likely to sit unreviewed until master opens again for feature landings. Patch submissions also need to be triaged and prioritized, and sometimes less important patches will need to wait until higher priority patches have been addressed. A tool cannot tell the difference between a patch submitter working tirelessly for a week to find an better solution, and a patch submitter ignoring the patch for a week. No simple time limit or accounting scheme is going to account for the reasonable delays. And I would rather not go down the route of adding a great deal of complicated machinery to account for every possible reasonable delay. After all, we are primarily here to develop Lustre software, not spend our time developing patch management software. :) Please don't take that to the extreme and think I am against all tool improvement. I am heartily in favor of some improvements that were suggested in a previous thread. I just want to spend our effort where it is likely to make the most impact. The two problem stages that you point out really don't seem much different than any other open source project of reasonable size. Folks always need to prioritize the incoming work and take into account the stability of the project when considering landings. Of course, there is one major difference from other open source projects, and that is the Lustre Development Community Tree Maintenance contract that OpenSFS has with Intel. This contract means that we are guaranteed to have a say in the prioritizing of patch work. If a developer feels that a patch is not prioritized high enough and moving fast enough towards landing, they may bring the patch to the attention of the CDWG and OpenSFS's Technical Representative on the contract (Chris Morrone). This is another example of a way that the patch submitter needs to stay actively engaged in the process. The CDWG and the Technical Representative are not likely to be much help with a patch problem if they are not made aware of the problem. Please don't ask me to monitor all patches in gerrit from all organizations. I simply don't have the time to do that, regardless of how good the tools are. I need the individual software developers to make a reasonable attempt to first resolve the delays amongst themselves through normal conversation, be it in jira or in gerrit. Then, if in their reasoned judgment a patch's priority needs to be raised and worked on sooner, they can bring the patch to the attention of the CDWG and myself. > Numbers will not say us all trues about review - they will indicate > problem reviews and problem stages. They will also reveal a large number of false positives. Many patches sit for good reason. How do we avoid wasting time on the false positives? > That what I want to discuss on CDWG every week - what problem reviews we > have, what problem stages now. I am certainly in favor of discussing process improvement on a regular basis! But keep in mind that our goal here is not to land patches as fast as possible. Our goal is to strike a reasonable balance in Lustre between stability, performance, and new features. All patches fall somewhere different on that continuum, and there are necessarily many judgment calls along the way to landing. That will mean that some patches get attention within hours or days, some will not get attention for weeks or months. And that is OK. That is just the reality of an active, vibrant open source development project. > Please review and provide your feedback. > > Thanks, > Denis From vitaly_fertman at xyratex.com Tue Sep 10 14:03:54 2013 From: vitaly_fertman at xyratex.com (Vitaly Fertman) Date: Tue, 10 Sep 2013 18:03:54 +0400 Subject: [cdwg] pending review list In-Reply-To: References: <5226BCBC.9000106@llnl.gov> Message-ID: <87502D07-C082-43F1-8DDA-A6315BB8298C@xyratex.com> Hi Chris, Here are some comments on your email. Although it is really important that developers monitor their own tickets perhaps there are some simple ways to make this easier. Have a read and we can discuss on Wednesday. > I think that such a notification would probably be more annoying than useful. If I want to know the state of my patches, I can pull up gerrit at any time and learn their current status. I also get notification email any time my patches are reviewed, someone comments on the patch, someone comments in jira, etc. In my opinion there are plenty of existing notifications, and I personally don't really want any additional notifications unless they provide new information. 1. no useless notifications. I agree it is useless to send out an automatically generated list of tickets, which may have some garbage or already discussed tickets. however, we want to minimize a time of getting such a list of tickets in hand. automatization will give us a "full" list of ticket - to be looked at by Intel to see if there is anything new to take care of; - to be looked at by Xyratex to see if the progress is adequate; after such digesting, both parts can do their own work and identify the problem tickets which in their turn are ready to be sent to a common list and discussed at CDWG. > Further, how could we possibly pick a number that makes sense? Patches sitting in the review stage may do so for a whole host of reasons, many that are perfectly reasonable. For instance, a new feature that is submitted after feature freeze is fairly likely to sit unreviewed until master opens again for feature landings. Patch submissions also need to be triaged and prioritized, and sometimes less important patches will need to wait until higher priority patches have been addressed. 2. prioritization. I agree the prioritization is needed, and this is what is currently missed, it is absolutely not clear why some our patches are inspected quickly, some are hanging for months. We want a process which would include prioritization step - once done, we can disagree with the priority and quickly rase it on CDWG meeting, or we would know the inspection will not be done soon and will not waste CDWG time on it. Once such priority is set, it also could be used for further automatization so that the final pending list would be more correct. > Please don't ask me to monitor all patches in gerrit from all organizations. I simply don't have the time to do that, regardless of how good the tools are. I need the individual software developers to make a reasonable attempt to first resolve the delays amongst themselves through normal conversation, be it in jira or in gerrit. Then, if in their reasoned judgment a patch's priority needs to be raised and worked on sooner, they can bring the patch to the attention of the CDWG and myself. 3. process individual developers will not do it so regularly as you want. people may react on some event (email, etc), they are mostly concentrated on their current task, it is easy to forget to check the state of some patches, moreover do it regularly. they neither attend CDWG regularly. a company representative attends, and it would be good to minimize a time for him of making a list of pending tickets, checking their states regularly before meetings. the proper way is to create a proper process so that each ticket could be viewed at any time in a common list of patches waiting for review, and its position in its list would be clear to everyone (depending on priority/other labels). such a basis will let developers from different companies to resolves issues "amongst themselves through normal conversation" as you want. imho, CDWG should take care about such a process, the more productive it will be, less interrupts for CDWG people. process is supposed to cover the following steps: a. once patch is submitted, it should be prioritized (by Intel?), other needed labels are also set. b. some auto-tool makes a list of tickets (no test/review failures) sorted by priority/delay time/etc. c. list is looked through by Intel and Submitter Companies, problem places are identified by both parties: - wrong priority; - large delay not matching the set priority; - re-priritization is needed; - etc; d. company representatives rase problem tickets on CDWG; not a final process of course and any suggestions to improve it are welcome. -- Vitaly From morrone2 at llnl.gov Tue Sep 10 22:30:29 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 10 Sep 2013 15:30:29 -0700 Subject: [cdwg] OpenSFS CDWG Call Message-ID: <522F9D85.2040803@llnl.gov> Hi all, This is a reminder that we will have a Community Development Working Group (CDWG) call on Wednesday, September 10, at 9:00am PDT. Agenda: * Review action items * Lustre 2.5.0 status update * Lustre 2.6.0 planning * Discuss patch landing process Call-in: 866-914-3976 (925-424-8105) Passcode: 534986# Regards, Chris From peter.a.jones at intel.com Fri Sep 13 20:43:14 2013 From: peter.a.jones at intel.com (Jones, Peter A) Date: Fri, 13 Sep 2013 20:43:14 +0000 Subject: [cdwg] Lustre 2.5 update - September 13th 2013 Message-ID: Hi there Here is an update on the Lustre 2.5 release. Landings ======== -A number of landings made - see http://git.whamcloud.com/?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master Testing ======= - Testing on the 2.4.92 in progress Blockers ======== -Full list available at https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVersion%20%3D%20%22Lustre%202.5.0%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC -If there are any issues not presently marked as blockers that you believe should be, please let me know Other ===== -We are trialling using the Fix In Release field for the 2.5 release to track work hoped to be included in the release; this means that a fuller picture of the progress on the release can be viewed at https://jira.hpdd.intel.com/browse/LU/fixforversion/10295#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel -Feature Freeze is now in effect and 2.5 has entered its stabilization period. We encourage community members to test out the latest code on master and to use JIRA (https://jira.hpdd.intel.com) to report any bugs encountered -Note that in order to adhere to our end of October release target date, we will need to enter code freeze in the next 2-3 weeks; this will mean that only changes to address blocker issues will be accepted for landing for 2.5.0 and lower priority issues will need to be deferred until 2.5.1 -You can also keep up to date with matters relating to the 2.5 release on the CDWG wiki - http://wiki.opensfs.org/Lustre_2.5.0 Thanks Peter From peter.a.jones at intel.com Fri Sep 13 23:15:57 2013 From: peter.a.jones at intel.com (Jones, Peter A) Date: Fri, 13 Sep 2013 23:15:57 +0000 Subject: [cdwg] Lustre 2.4.1 released Message-ID: We are pleased to announce that the Lustre 2.4.1 Release has been declared GA and is available for download . You can also grab the source from git The release highlights are LU-3415: It is now possible to upgrade from 1.8.x to 2.4.0 with quotas enabled without hitting an LBUG LU-3420: OI scrub will now be engaged after a file level backup running with DNE LU-3421: A case where OSTs might erroneously report ENOSPC has been identified and fixed LU-3305: A metadata performance drop running with quotas enabled has been addressed Fuller details can be found in the change log The following are known issues in the Lustre 2.4.1 Release: LU-3027: A race condition can result in mis-reporting of file size LU-3430 : A rare race can result in an LBUG on an OSS while running the orr policy for NRS LU-3544: Re-exporting Lustre over NFS from SLES11 SP2 has known issues LU-3934: Files created under Lustre 1.8.x could be inaccessible for a filesystem first upgraded to 2.1.x and then subsequently upgraded to 2.4.1 with a client remaining mounted. The workaround it to manually trigger an OI scrub after the upgrade. Work is in progress for these issues. Please log any issues found in the issue tracking system From morrone2 at llnl.gov Wed Sep 25 02:16:13 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Tue, 24 Sep 2013 19:16:13 -0700 Subject: [cdwg] OpenSFS CDWG Call Message-ID: <5242476D.7000805@llnl.gov> Hi all, This is a reminder that we will have a Community Development Working Group (CDWG) call on Wednesday, September 25, at 9:00am PDT. Agenda: * Review action items * Describe monthly BoD/WG Leads meeting * Lustre 2.5.0 status update * Lustre 2.6.0 planning Call-in: 866-914-3976 (925-424-8105) Passcode: 534986# Regards, Chris From spitzcor at cray.com Thu Sep 26 17:32:40 2013 From: spitzcor at cray.com (Cory Spitz) Date: Thu, 26 Sep 2013 17:32:40 +0000 Subject: [cdwg] 2.5 blockers list Message-ID: Hello, It seems that it is about time to create a 2.5 blockers list at jira.hpdd.intel.com. I didn't see one yet though. Should it be done now or later? Thanks, -Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From morrone2 at llnl.gov Thu Sep 26 17:51:41 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Thu, 26 Sep 2013 10:51:41 -0700 Subject: [cdwg] 2.5 blockers list In-Reply-To: References: Message-ID: <5244742D.3030209@llnl.gov> Since we are using the Fix Version/s field to mark tickets that are targeted for resolution in a particular release, you can now use normal Jira mechanisms to get that list. We have a link to the 2.5.0 Unresolved Issues here: http://wiki.opensfs.org/Lustre_2.5.0 Or you can always find it through Jira, through a number of paths. Here is one: Select Projects->Lustre(LU) from the top tab. Select "Versions" from the left side. Click on "Lustre 2.5.0" Select "Issues" from the left Or here is a direct link to that search pattern: https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVersion%20%3D%20%22Lustre%202.5.0%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC Chris On 09/26/2013 10:32 AM, Cory Spitz wrote: > Hello, > > It seems that it is about time to create a 2.5 blockers list at > jira.hpdd.intel.com. I didn't see one yet though. Should it be done > now or later? > > Thanks, > -Cory > > > _______________________________________________ > cdwg mailing list > cdwg at lists.opensfs.org > http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org > From spitzcor at cray.com Thu Sep 26 17:57:16 2013 From: spitzcor at cray.com (Cory Spitz) Date: Thu, 26 Sep 2013 17:57:16 +0000 Subject: [cdwg] 2.5 blockers list In-Reply-To: <5244742D.3030209@llnl.gov> Message-ID: If we create a filter, than users can subscribe as well so I'd still like to see the blocker list created and managed regardless of search options. Are we going to micro-manage the Fix Version/s field in the tickets though? If we decide a bug isn't a blocker for 2.5.0, someone will reset the Fix Version/s field to 2.5.1 right away? We should be careful that the 2.5 blocker list and a list formed from tickets with Fix Version/s == 2.5 are exact duplicate lists. -Cory On 9/26/13 12:51 PM, "Christopher J. Morrone" wrote: >Since we are using the Fix Version/s field to mark tickets that are >targeted for resolution in a particular release, you can now use normal >Jira mechanisms to get that list. > >We have a link to the 2.5.0 Unresolved Issues here: > > http://wiki.opensfs.org/Lustre_2.5.0 > >Or you can always find it through Jira, through a number of paths. Here >is one: > > Select Projects->Lustre(LU) from the top tab. > Select "Versions" from the left side. > Click on "Lustre 2.5.0" > Select "Issues" from the left > >Or here is a direct link to that search pattern: > > >https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVers >ion%20%3D%20%22Lustre%202.5.0%22%20AND%20resolution%20%3D%20Unresolved%20A >ND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC > >Chris > >On 09/26/2013 10:32 AM, Cory Spitz wrote: >> Hello, >> >> It seems that it is about time to create a 2.5 blockers list at >> jira.hpdd.intel.com. I didn't see one yet though. Should it be done >> now or later? >> >> Thanks, >> -Cory >> >> >> _______________________________________________ >> cdwg mailing list >> cdwg at lists.opensfs.org >> http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org >> > >_______________________________________________ >cdwg mailing list >cdwg at lists.opensfs.org >http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org From peter.a.jones at intel.com Thu Sep 26 18:58:06 2013 From: peter.a.jones at intel.com (Jones, Peter A) Date: Thu, 26 Sep 2013 18:58:06 +0000 Subject: [cdwg] 2.5 blockers list In-Reply-To: Message-ID: Cory We discussed using the FixVersion to track targeted content for the 2.5 release some months back so I have been using the links from the project dashboard in my regular updates on the release. I have shared a filter that jumps to the 2.5 blockers so that you can subscribe to it - https://jira.hpdd.intel.com/issues/?filter=10661 . Yes, both lists will be in sync Regards Peter On 9/26/13 10:57 AM, "Cory Spitz" wrote: >If we create a filter, than users can subscribe as well so I'd still like >to see the blocker list created and managed regardless of search options. > >Are we going to micro-manage the Fix Version/s field in the tickets >though? If we decide a bug isn't a blocker for 2.5.0, someone will reset >the Fix Version/s field to 2.5.1 right away? We should be careful that >the 2.5 blocker list and a list formed from tickets with Fix Version/s == >2.5 are exact duplicate lists. > > >-Cory > >On 9/26/13 12:51 PM, "Christopher J. Morrone" wrote: > >>Since we are using the Fix Version/s field to mark tickets that are >>targeted for resolution in a particular release, you can now use normal >>Jira mechanisms to get that list. >> >>We have a link to the 2.5.0 Unresolved Issues here: >> >> http://wiki.opensfs.org/Lustre_2.5.0 >> >>Or you can always find it through Jira, through a number of paths. Here >>is one: >> >> Select Projects->Lustre(LU) from the top tab. >> Select "Versions" from the left side. >> Click on "Lustre 2.5.0" >> Select "Issues" from the left >> >>Or here is a direct link to that search pattern: >> >> >>https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVer >>s >>ion%20%3D%20%22Lustre%202.5.0%22%20AND%20resolution%20%3D%20Unresolved%20 >>A >>ND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC >> >>Chris >> >>On 09/26/2013 10:32 AM, Cory Spitz wrote: >>> Hello, >>> >>> It seems that it is about time to create a 2.5 blockers list at >>> jira.hpdd.intel.com. I didn't see one yet though. Should it be done >>> now or later? >>> >>> Thanks, >>> -Cory >>> >>> >>> _______________________________________________ >>> cdwg mailing list >>> cdwg at lists.opensfs.org >>> http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org >>> >> >>_______________________________________________ >>cdwg mailing list >>cdwg at lists.opensfs.org >>http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org > >_______________________________________________ >cdwg mailing list >cdwg at lists.opensfs.org >http://lists.opensfs.org/listinfo.cgi/cdwg-opensfs.org From morrone2 at llnl.gov Thu Sep 26 21:04:20 2013 From: morrone2 at llnl.gov (Christopher J. Morrone) Date: Thu, 26 Sep 2013 14:04:20 -0700 Subject: [cdwg] 2.5 blockers list In-Reply-To: References: Message-ID: <5244A154.1060205@llnl.gov> On 09/26/2013 10:57 AM, Cory Spitz wrote: > If we create a filter, than users can subscribe as well so I'd still like > to see the blocker list created and managed regardless of search options. > > Are we going to micro-manage the Fix Version/s field in the tickets > though? If we decide a bug isn't a blocker for 2.5.0, someone will reset > the Fix Version/s field to 2.5.1 right away? We should be careful that > the 2.5 blocker list and a list formed from tickets with Fix Version/s == > 2.5 are exact duplicate lists. I would just call that managing the tickets, not micromanaging (with all its negative connotations). And it kind of depends on what you mean by "blocker list". In one sense, "blockers" are the tickets that have a priority setting of Blocker, and that is probably what Intel folks think of when you say "blocker list". But I am trying to work to change that perception, and get people to realize than even tickets with a priority lower than "Blocker" sometimes needs to be scoped, worked, and resolved before the release can happen. In other words, in a broader sense _all_ of the unresolved tickets lodged against 2.5.0 need to be addressed in some way before the release. I think that it is important to include tickets with all priorities in the normal search results of unfinished work for a release. If we continually show folks only a list of Blocker priority tickets, they will only be seeing the tip of the iceberg. My belief is that this leaves too many people ignorant of the vast amount of work that we continue to leave unfinished in every release. When people underestimate the unfinished work, I feel that has a negative impact on our planning for Lustre in general. Folks continually fight attempts to address the technical debt in lustre (horrible API, unintelligble and undocumented code, barely useable command line tools, horrible configuration system, etc.) When folks aren't aware of the backlog of work, we see too much funding going to fancy new features. Those feature introduce new bugs on top of the large pile of existing bugs, and Lustre gets worse over time rather than better. I'm not advocating that we stop feature development; we always need to strike a balance between stability and new features. But in my opinion speed and stability are on the decline, and we need to tip the balance back a bit in in the other directions. So in summary, I strongly believe that the "2.5.0 Unresolved Issues" search on the http://wiki.opensfs.org/Lustre_2.5.0 page is the correct one to be watching on a regular basis. I have made a publicly accessible filter named 2.5.0 Unresolved Issues in jira, for those who don't want to hit the "Save as" button on their own. :) If that list is not zero, we don't make the release. Chris From peter.a.jones at intel.com Fri Sep 27 22:35:10 2013 From: peter.a.jones at intel.com (Jones, Peter A) Date: Fri, 27 Sep 2013 22:35:10 +0000 Subject: [cdwg] Lustre 2.5 update - September 27th 2013 Message-ID: Hi there Here is an update on the Lustre 2.5 release. Landings ======== -A number of landings made - see http://git.whamcloud.com/?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master Testing ======= -Testing on the 2.4.92 tag completed; testing on the 2.4.93 tag in progress -The HSM Test plan (see https://jira.hpdd.intel.com/browse/LU-3608) has been executed on the OpenSFS test cluster Blockers ======== -Full list available at https://jira.hpdd.intel.com/issues/?jql=project%20%3D%20LU%20AND%20fixVersion%20%3D%20%22Lustre%202.5.0%22%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC -If there are any issues not presently marked as blockers that you believe should be, please let me know Other ===== -We are trialling using the Fix In Release field for the 2.5 release to track work hoped to be included in the release; this means that a fuller picture of the progress on the release can be viewed at https://jira.hpdd.intel.com/browse/LU/fixforversion/10295#selectedTab=com.atlassian.jira.plugin.system.project%3Aversion-issues-panel -Feature Freeze is now in effect and 2.5 has entered its stabilization period. We encourage community members to test out the latest code on master and to use JIRA (https://jira.hpdd.intel.com) to report any bugs encountered -Note that in order to adhere to our end of October release target date, we will need to enter code freeze early next week; this will mean that only changes to address blocker issues will be accepted for landing for 2.5.0 and lower priority issues will need to be deferred until 2.5.1 -You can also keep up to date with matters relating to the 2.5 release on the CDWG wiki - http://wiki.opensfs.org/Lustre_2.5.0 Thanks Peter