From shay at opensfs.org Thu Mar 1 17:29:53 2012 From: shay at opensfs.org (Shay Seager) Date: Thu, 1 Mar 2012 09:29:53 -0800 Subject: [Twg] LUG Program Announced Message-ID: Hello Lustre Community! The LUG committee is happy to announce that we have just published the LUG Program. Please follow this link http://www.opensfs.org/lug/program to see what all the excitement is about. Don't forget to register by March 9th to ensure early bird pricing and hotel room discounts! Register here: http://www.opensfs.org/lug/registration. We look forward to seeing you at LUG! - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu Mar 8 14:32:01 2012 From: carrier at cray.com (John Carrier) Date: Thu, 8 Mar 2012 14:32:01 +0000 Subject: [Twg] TWG meeting minutes for 2012-03-01 Message-ID: Attached are the meeting minutes from last week's TWG call. We spent this meeting introducing features that the TWG should consider as we update our requirements and roadmaps document. We will continue this discussion this week (3/8). Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 Please post corrections/questions/comments to discuss at lists.opensfs.org My apologies for getting these out so late. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 2012-03-01 Concall: start : 9:30a PT, adjourn : 10:44a PT Next meeting: Thursday, 03/08/2011 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Attending : Name Organization email ----------------- -------------- ---------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Gene Oleynik FNAL oleynik at fnal.gov Joshua Walgenbach IU jjw at indiana.edu Chris Morrone LLNL morrone2 at llnl.gov Norm Morse OpenSFS normanrmorse at gmail.com Shay Seager OpenSFS shay at opensfs.org Dave Dillow ORNL dillowda at ornl.gov Andreas Dilger Whamcloud adilger at whamcloud.com Agenda - Requirements & Roadmap * update community requirements * identify features in need of development * funneling projects outside OpenSFS into CDWG We started discussing the list of requirements the TWG gathered a year ago (http://goo.gl/cZSWG) [The topic list is copied at the end of these minutes]. We quickly focused on the new items below. = single client performance improvements DaveD said that single client gets CPU bound which limits bandwidth to 500 MB/s. AndreasD said the new multi-thread RPC code, which has been funded by ORNL and LLNL, improves single threaded reader/writer performance. The new code should land in Lustre 2.2. ChrisM said Livermore is seeing 2x improvements with the new code (1000 MB/s using 8 threads). The CPU is not limited and the bottleneck beyond GB/s is elsewhere. ORNL's point is still valid: there is a bottleneck limiting single client performance. Though CPU affinity for the MDS may propagate to the clients, the client needs to improve SMP scalability and efficiency. = improved MDS rates JohnC asked if the debug feature Whamcloud uses to increase the MDS rate for testing could be used to improve performance by increasing the number of MDS RPCs in flight. Andreas said the debug symbol he mentioned last week is useful for testing but breaks recovery. Nonetheless, there is a need to increase the number of file-modifying RPCs in flight for DNE, which will must support multiple threads on a single client and many RPCs in flight from one MDS to another. This feature will appear as a debug patch in 2.2. If testing with the patch works, then there is work to deliver the feature to the client. = online data migration Andreas suggested that this feature was needed to empty OSTs, rebalance space between OSTs and to rebalance between OST pools. The first phase is tied to the HSM since a migration tool would need layout locks, a policy engine and the IOCTL to tell MDS to change the layout and flush the client cache. Apparently, the layout locks in the current implementation of the HSM are sufficient for quiescent files, but not for active files. Hence, online data migration will require fixing layout locks in the HSM first. DiegoM said that the HSM is fully operational, but not robust. He said Bull seems to be the only one interested in testing and the code needs more groups using it to make progress. OpenSFS could ask CEA how OpenSFS could help. AlexK said FNAL was very interested in data migration. He is interested in inactive files. Andreas said lfs_migrate already does this and would need a layout lock patch. The problem is that the script is active all the time and today has a performance impact. Fixing the layout locks is a first step to migration, replication, and other enterprise features. For example, a policy manager could move copies to other sites after the I/O quiesces using a similar mechanism to migration. John asked if the layout locks and migration could be used for snapshots. Andreas thought this needs to be supported in the backend file system. ldiskfs can't do it, but ZFS can. ChrisM said LLNL is not yet looking at this ZFS feature in their work. = improved security Alex said his site needs kerberos and x509. The open science grid needs x509 for data transfers. JoshW described the work IU plans to do to improve security for the LustreWAN project. He agreed to talk to Alex on how x509 could be done within the new infrastructure. = dynamic LNET configuration Andreas introduced the idea of providing dynamic LNET configuration so that changes to the LNET configuration would not require stopping the file system for the changes to be propagated. The group enthusiastically supported any attempt to remove LNET's current dependence on modprobe. The configuration also has to allow flexibility in some of the LNET parameters. Cory mentioned that tuning peer credites on o2iblnd has to agree across all LNET tiers. He has found that the only way to fix the mismatch is to shutdown all clusters. ORNL currently uses scripts to modify lNET configuration by hand. Dave said he thinks this effort would have to fix ptlrpc to remove stale entries. He thinks this is a long-term solution, but it wouldn't really help him at ORNL. Andreas said the effort for dynamic LNET config isn't gigantic, but it would touch lots of code (LNET, ptlrpc) At this point, we had run over our scheduled meeting time. The following items were presented without much discussion: = IPv6 This item was on our list last year. Andreas said adding support for IPv6 will change the network protocol so the impact is more involved than just changing internal data structures. He suggested a phased approach that could make the feature available in the code base, but would not be used until a site upgraded to IPv6. Andreas said the lack of IPv6 will hit WAN users first. = Network Request Scheduler (NRS) The core infrastructure of NRS has been prototyped and Xyratex has done some work recently. What is needed to complete the project and have it land in the canonical tree? = complex file layouts Andreas suggested adding support to allow different file layouts for different file types (eg HDF5). A simple example would let a file start with one stripe definition then move to different striping after the file grew. Dave ended the meeting by asking the TWG to consider projects that affected Lustre's long term roadmap. Should OpenSFS fund research projects? ------------------------- OpenSFS Requirements (3/22/2011) * Performance Requirements - Metadata server performance [FUNDED] - Metadata server scalability [FUNDED] - Single file performance - Quality of service - Locality and scalability - LNET channel bonding * Foundational Requirements - Support for alternate backend file systems [FUNDED] - Backend storage investigation - Scalable fault management * Manageability and Administrative Requirements - Better support for newer kernels - Improved configuration of Lustre - Allowing for controlled partial-system maintenance - Balancing storage use - Adaptive storage layout - Arbitrary OST assignment - Better userspace tools - File system consistency checks [FUNDED] - Snapshots - User Identity Mapping [FUNDED] * Application Interface Requirements - Improved storage semantics/interfaces - Better user tool API * Other Requirements - Varying page-sizes - Mixed endian support - IPv6 From carrier at cray.com Thu Mar 8 14:58:41 2012 From: carrier at cray.com (John Carrier) Date: Thu, 8 Mar 2012 14:58:41 +0000 Subject: [Twg] TWG meeting reminder 3/8/2012 Message-ID: The TWG will continue the discussion of roadmap features for future OpenSFS funding during today's TWG call: Date/Time : 03/08/2011 @ 9:30a PDT/12:30p EDT Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjw at iu.edu Thu Mar 8 18:04:00 2012 From: jjw at iu.edu (Joshua Walgenbach) Date: Thu, 08 Mar 2012 13:04:00 -0500 Subject: [Twg] OpenSFS Test Cluster Configuration Message-ID: <4F58F490.5040600@iu.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 All, Here is all the deliverable items for the OpenSFS test cluster for a discussion of concerns. Thanks, - -Josh - ---------------------------------------------------------------- 6 X Test Client Cluster (Client Nodes; 2U per 4 nodes, blade type) FT-E5620SXi/2UQ, Intel Single Xeon E5620 2.4GHz 2U-Quad Nodes Including: * (1) SM 2U 6026TT-HiBQRF 2U Quad Nodes * (4) sets of 1 x Intel E5620 2.4GHz Westmere (Total 4 Cores) * (4) sets of 1 x Copper Base CP0217 CPU Cooler 1U with Heat Pipe * (4) sets of 32GB DDRIII 1333MHz ECC/REG - (4x8GB Modules Installed) * (4) sets of On Board Dual 10/100/1000T Ports * (4) sets of On Board VGA * (4) sets of On Board IPMI 2.0 Via 3rd. Lan * (4) sets of 1 x 500GB SATA Enterprises 24x7 * (4) sets of 3 x Hot Swap Drive Bays for SATA/SAS * (4) sets of PCi-e Slots 8X * (4) sets of QDR 40GB QSFP to QSFP iB Cables * (4) sets of Mellanox QDR 40GB QSFP Port - Embedded * (1) Platinum level 1400W redundant Power * (1) UL (NRTL) Certified Power Supply/System * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 4 X Test Server Nodes (MDS server nodes) FT-E5620DXi/2U, Intel Single Xeon E5620 2.4GHz 2U Including: * (1) SM X8DTH-iF Motherboard * (2) Intel E5620 2.4GHz Westmere (Total 8 Cores) * (2) Copper Base CP0217 CPU Cooler 1U with Heat Pipe * (1) 64GB DDRIII 1333MHz ECC/REG - (8x8GB Modules Installed) * (1) On Board Dual 10/100/1000T Ports * (1) On Board VGA * (1) On Board IPMI 2.0 Via 3rd. Lan * (1) 500GB SATA Enterprises 24x7 * (1) 40GB SSD OCZ SATA * (8) Hot Swap Drive Bays for SATA/SAS * (6) PCi-e Slots 8X * (1) QDR 40GB QSFP to QSFP iB Cables * (1) Mellanox QDR 40GB QSFP Single Port * (2) LCI 9200-8E 2xSAS External Ports to Storage * (4) SAS Cables for Storage * (1) Platinum level 720W redundant Power * (1) Set of Screw Free Drop-in Rails * (1) UL (NRTL) Certified Power Supply/System * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 4 X Shared Storage Devices for MDS (1 per MDS server node, and by Pair) Including: * (1) Infortrend DSS12SR2240MB-0A30 SAS-to-SAS/SATA 2U/12-bay Redundant RAID, Data Service Option, 4 x SAS-6G ports, SAS-6G Exp, ASIC667, 4GB, BBU * (2) Intel 710 Enterprises 100GB SSD * (10) Toshiba 2TB SAS II 6GB/s 24x7 Enterprises * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 4 X OSS server ndoes FT-E5620DXi/2U, Intel Single Xeon E5620 2.4GHz 2U Including: * (1) SM X8DTH-iF Motherboard * (2) Intel E5620 2.4GHz Westmere (Total 8 Cores) * (2) Copper Base CP0217 CPU Cooler 1U with Heat Pipe * (1) 64GB DDRIII 1333MHz ECC/REG - (8x8GB Modules Installed) * (1) On Board Dual 10/100/1000T Ports * (1) On Board VGA * (1) On Board IPMI 2.0 Via 3rd. Lan * (1) 500GB SATA Enterprises 24x7 * (1) 40GB SSD OCZ SATA * (8) Hot Swap Drive Bays for SATA/SAS * (6) PCi-e Slots 8X * (1) QDR 40GB QSFP to QSFP iB Cables * (1) Mellanox QDR 40GB QSFP Single Port * (2) LCI 9200-8E 2xSAS External Ports to Storage * (4) SAS Cables for Storage * (1) Platinum level 720W redundant Power * (1) Set of Screw Free Drop-in Rails * (1) UL (NRTL) Certified Power Supply/System * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 4 X Shared Storage Devices for MDS (1 per OSS server node) Including: Including: * (1) Infortrend DSS12SR2240MB-0A30 SAS-to-SAS/SATA 2U/12-bay Redundant RAID, Data Service Option, 4 x SAS-6G ports, SAS-6G Exp, ASIC667, 4GB, BBU * (1) DSS12SJ2000R-0A32 SAS-to-SAS/SATA 2U/12-bay Redundant JBOD for EonStor DS, 4 x SAS-6G ports. * (4) Intel 710 Enterprises 100GB SSD * (20) Toshiba 2TB SAS II 6GB/s 24x7 Enterprises * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 1 X Head Node FT-E5620SXi/1U, Intel Single Xeon E5620 2.4GHz 1U Including: * (1) Intel E5620 2.4GHz Westmere (Total 4 Cores) * (1) Copper Base CP0217 CPU Cooler 1U with Heat Pipe * (1) 24GB DDRIII 1333MHz ECC - (6x4GB Modules Installed) * (1) On Board Dual 10/100/1000T Ports * (1) On Board VGA * (1) On Board IPMI 2.0 Via 3rd. Lan * (1) 500GB SATA Enterprises 24x7 * (2) Hot Swap Drive Bays for SATA/SAS * (1) PCi-e Slots 8X * (1) Platinum level 450W redundant Power * (1) Set of Screw Free Drop-in Rails * (1) UL (NRTL) Certified Power Supply/System * 3 years Warranty Parts and Labor Fully Warranty - ---------------------------------------------------------------- 1 X Avocent 6048AC, 48 Ports Console 1U Server with 48 DB9 to RJ45 Adapter and Cat5e Cables - ---------------------------------------------------------------- 2 X NetGear 48 Ports 10/100/1000T Gige Switch with 48 Cat5e cables - ---------------------------------------------------------------- 1 X InfiniScale IV QDR InfiniBand Switch, 1U, 36 QSFP ports, 1 Power Supply, Managed (PPC405EXR), PSU side to connector side airflow, Standard depth, Rail Kit - ---------------------------------------------------------------- 1 X 1U 17" LCD Rack Mount with KB and Mouse PS/2 or USB- connect to Head node - ---------------------------------------------------------------- 2 X APC NetShelter SX AR3347 Enclosure Rack Cabinet NETSHELTER SX 48U 750X1200MM NETWORKING ENCLOSURE W/ SIDES BLACK - ---------------------------------------------------------------- 8 X PDU: CS-24VD-L30M; Smart CDU, 0U, 24 x IEC C13 Outlets, 3 x LED amp meters, 208V 3-Phase Delta input w/ NEMA L15-30P connector & cord, 10ft (3m), (2) Temperature/Humidity Sensor Capable, with retractable fuses -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9Y9JAACgkQcqyJPuRTYp/IaACfXxu/DTke4dW4vaWL0Zhk6A+l 5HIAnicphwoOAAlKhi0MyfUCq97O77jL =uyvN -----END PGP SIGNATURE----- From aik at fnal.gov Thu Mar 8 20:12:37 2012 From: aik at fnal.gov (Alex Kulyavtsev) Date: Thu, 8 Mar 2012 14:12:37 -0600 Subject: [Twg] FDMI Re: [Discuss] TWG meeting reminder 3/8/2012 In-Reply-To: References: Message-ID: http://www.xyratex.com/pdfs/whitepapers/Xyratex_white_paper_Lustre_Architecture_Priorities_Overview_1-0.pdf > 4.1 Replication Migration page 12: > The APIs used by the replication, backup, and file system > infrastructure are part of the Xyratex File Data Management > Interface (FDMI). Is there a specification or description of FDMI? Alex. -------------- next part -------------- An HTML attachment was scrubbed... URL: From adilger at whamcloud.com Thu Mar 15 00:31:29 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Wed, 14 Mar 2012 18:31:29 -0600 Subject: [Twg] Lustre and cross-platform portability Message-ID: <5609A4E0-2356-400F-9D53-F71B7007338C@whamcloud.com> Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From carrier at cray.com Thu Mar 15 07:04:31 2012 From: carrier at cray.com (John Carrier) Date: Thu, 15 Mar 2012 07:04:31 +0000 Subject: [Twg] TWG meeting minutes for 2012-03-08 Message-ID: Attached are the meeting minutes from last week's TWG call. This meeting continued the discussion of features that the TWG should consider as we update our requirements and roadmaps document. We will continue this discussion this week (3/15). Date/Time : 03/15/2011 @ 9:30a PDT/12:30p EDT Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 Please post corrections/questions/comments to discuss at lists.opensfs.org --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu Mar 15 12:14:47 2012 From: carrier at cray.com (John Carrier) Date: Thu, 15 Mar 2012 12:14:47 +0000 Subject: [Twg] TWG meeting minutes for 2012-03-08 In-Reply-To: References: Message-ID: Here are the advertised minutes. From: discuss-bounces at lists.opensfs.org [mailto:discuss-bounces at lists.opensfs.org] On Behalf Of John Carrier Sent: Thursday, March 15, 2012 12:05 AM To: twg at lists.opensfs.org; discuss at lists.opensfs.org Subject: [Discuss] TWG meeting minutes for 2012-03-08 Attached are the meeting minutes from last week's TWG call. This meeting continued the discussion of features that the TWG should consider as we update our requirements and roadmaps document. We will continue this discussion this week (3/15). Date/Time : 03/15/2011 @ 9:30a PDT/12:30p EDT Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 Please post corrections/questions/comments to discuss at lists.opensfs.org --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 2012-03-08 Concall: start : 9:30a PT, adjourn : 10:25a PT Next meeting: Thursday, 03/15/2011 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Attending : Name Organization email ----------------- -------------- ---------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Joshua Walgenbach IU jjw at indiana.edu Richard Vanderbilt NetApp Richard.Vanderbilt at netapp.com Dave Dillow ORNL dillowda at ornl.gov Evan Felix PNL Evan.Felix at pnl.gov Nathan Rutman Xyratex nathan_rutman at xyratex.com Agenda - update on OSFS test cluster - continue requirements & roadmap discussion Test cluster at IU The OSFS board has approved purchase of small test cluster for lustre development at IU to include 24 client nodes, 4 MDS, 4 OSS and storage. IU will host the system for OpenSFS. The cluster will be available to the OpenSFS Community for Lustre testing without citizenship restrictions. Initial use will be for testing the DNE feature (multiple MDTs, failover). Dave asked the TWG for feedback on how to manage allocations and usage of this community resource. This will be a topic for discussion at the TWG meeting at LUG. Josh will send the system specs to the TWG reflector. Roadmap / Requirements = version compatability ? John asked about tools to verify client/server compatability. A more complete compatability matrix could help, but Cray was thinking more about a utility that system vendors could use to verify server functionality since Lustre will soon be deployed on a range of server appliances. This would be a tool used to test compatibility without having to deploy a system with storage. Lustre already has automatic version or feature control since the client and server negotiate which features each client can use. Can look at client or server proc files to see what was negotiated. Hence, most felt this feature wasn't required. = HSM Diego talked to CEA. HSM patches are waiting for integration. CEA is waiting to complete the inspection for inclusion in 2.2 and 2.3. It was reported that Johann Lombardi is the inspector at whamcloud for the HSM patches. Many will land in 2.3, with full HSM support in 2.4. There is concern that HSM isn't getting sufficient attention to complete integration. John is to ask the CDWG on behalf of the TWG about the priority of HSM for upcoming releases. Diego said the community can get involved by testing CEA's patches. = NRS Nathan said Xyratex is actively working on NRS and will present their work with different policies at LUG. After Xyratex proves no regressions during large scale testing they will land the infrastructure for a release. As a result, there is nothing for the TWG to do to support NRS. = replication Alex asked about the API's that Xyratex described in their migration whitepaper. Nathan explained that the software will move a lustre file system, but it is not a snapshot. Xyratex's tool doesn't walk the namespace, instead using the changelog to move the file system. Xyratex has two tools: a migrator and repliator. The migrator makes a one time move from 1.8.x to 2.x MDS by adding true FIDS to the MDS. The replicator is a brute force copy to duplicate an entire file system. = benchmarking working group John introduced Richard Vanderbilt as the co-chair of the new benchmarking working group. Sarp Oral is the other co-chair. The new WG will focus on identifying I/O workloads and tools to emulate them. The benchmarking working group will have a F2F the Sunday before LUG. (The TWG and CDWG will have F2F the Wednesday after LUG.) [meeting ended 10:25a PT] From adilger at whamcloud.com Thu Mar 15 18:38:59 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 15 Mar 2012 12:38:59 -0600 Subject: [Twg] Lustre and cross-platform portability In-Reply-To: References: Message-ID: <7F9C8D28-8F0C-4B06-841D-8CE5E947ED64@whamcloud.com> On 2012-03-15, at 8:22 AM, Tyler Hawes wrote: > Having a native Windows & Mac client is by far our company's #1 most important feature for the future. We have been seriously considering how we could help make this happen, including putting funds and/or developers toward the cause. Tyler, thanks for your feedback. While at Oracle, there was serious effort put toward having a Windows Native Client. Unfortunately, this died along with other Oracle Lustre development projects, and the proprietary code has been sitting unused inside Oracle since then. Similarly, the MacOS native client project started a couple of years ago, but as yet none of the code has been released. While some work was done for a Solaris server, I don't see much hope in that making progress at all. In both cases, I would be supportive of these efforts if there was a likelihood of them bearing fruit any time in the future, but so far I haven't seen any progress in that direction. If Oracle could somehow release the WNC code and/or NRL can overcome the government roadblocks in their path for releasing the MacOS client (MLC?), it would definitely change the direction of my thinking. As it stands now, we have portability code for Windows/Mac without any ability to even build the code, let alone test it, so it is just a burden for Linux. If there was real value being derived from that code (e.g. actual users), then it might be an acceptable burden. > I know we're in the minority on this, but I believe it's because we are not using Lustre for HPC. We use it for post-production of TV/film. There are a few others companies in our industry who have started doing this as well. My point is that, while Lustre currently is focused on the HPC crowd who seem not to care about Windows/Mac, Lustre's maturity is giving it the potential to grow into other uses besides HPC. I wouldn't call them general use, but other high-performance uses. In our industry, where there are a lot of Windows & Mac workstations that we want to connect to the Lustre storage, the Linux-only client is a major obstacle to that. I'm sure there are other industries that would benefit from this. Yes, the TV/film industry was one of the prime motivators for WNC. There have always been a small number of users from this market, and growth in this industry (and others) is definitely welcomed. > If the community wants to keep Lustre strictly HPC focused and discourage other industries from joining in, then abandoning the bridge (albeit the half-built bridge that it is) to Linux/Windows is a good way to do that. If, on the other hand, there is a desire to get some other industries involved, perhaps with more resources and contribution coming from them, then I think it's important to build on the work that has been done. In that regard, upstream Linux kernel inclusion seems like a very low priority to me. I think even for upstream kernel submission, if it were clear that the layering in Lustre was for a valid purpose (i.e. existing Mac/Win clients) then it might be accepted. As it stands now, we can't honestly make that argument to the kernel maintainers. > 2012/3/15 > ---------- Forwarded message ---------- > From: Andreas Dilger > To: twg at lists.opensfs.org > Cc: wc-discuss , lustre-discuss discuss , Lustre Devel > Date: Wed, 14 Mar 2012 18:31:29 -0600 > Subject: [Lustre-discuss] Lustre and cross-platform portability > Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. > > As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). > > I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. > > I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? > > Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. > > The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. > > > Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > > > > Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From kenh at cmf.nrl.navy.mil Thu Mar 15 18:45:38 2012 From: kenh at cmf.nrl.navy.mil (Ken Hornstein) Date: Thu, 15 Mar 2012 14:45:38 -0400 Subject: [Twg] Lustre and cross-platform portability In-Reply-To: <5609A4E0-2356-400F-9D53-F71B7007338C@whamcloud.com> Message-ID: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> >I have no information that the WinNT project will ever be released >by Oracle, and as yet there has not been any code released from the >MacOS port, so the libcfs portability layer is potentially exacting >a high cost in code maintenance and complexity (CLIO being a prime >example) for no apparent benefit. Similarly, the liblustre client needs >a portability layer for userspace, and suffers from the same apparent >lack of interest or users. In terms of the MacOS X port, I don't think that's quite fair ... the code I did is available and anyone can download it. It was functional in a very basic way but needed some additonal love. Okay, I haven't rolled that stuff into the Whamcloud release ... what happened there was when there was all the uncertainty with Oracle & Lustre development I lost momentum and got caught up in other things. I've talked with the guys at Whamcloud about bringing at least the portability changes over, and that's all been on me; it's certainly on my list to work on. I can say that at least for MacOS X, there has been interest; I can't speak for the amount of interest, and there's a bit of a chicken and egg problem ... people don't plan their Lustre use around MacOS X clients because there isn't one that works well, and people don't put work into it because there isn't people who plan their Lustre use around it. >I'd like to get some feedback from the Lustre community about removing >the libcfs abstraction entirely, or possibly restructuring it to look >like the Linux kernel API, and having the other platforms code against >it as a Linux portability layer, like ZFS on Linux uses the Solaris >Portability Layer (SPL) to avoid changing the core ZFS code. A related >topic is whether it would be better to replace all cfs_* functions with >standard Linux kernel functions en-masse, or migrate away from cfs_* >functions slowly? The only thing I can think of is that if this is done, then officially Lustre is going to be a Linux-only filesystem. I understand there are real costs to maintaining the cfs layer, and I can't speak to whether or not the loss would be worth the gains. --Ken From adilger at whamcloud.com Thu Mar 15 19:39:43 2012 From: adilger at whamcloud.com (Andreas Dilger) Date: Thu, 15 Mar 2012 13:39:43 -0600 Subject: [Twg] Lustre and cross-platform portability In-Reply-To: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> Message-ID: On 2012-03-15, at 12:45 PM, Ken Hornstein wrote: >> I have no information that the WinNT project will ever be released >> by Oracle, and as yet there has not been any code released from the >> MacOS port, so the libcfs portability layer is potentially exacting >> a high cost in code maintenance and complexity (CLIO being a prime >> example) for no apparent benefit. Similarly, the liblustre client >> needs a portability layer for userspace, and suffers from the same >> apparent lack of interest or users. > > In terms of the MacOS X port, I don't think that's quite fair ... > the code I did is available and anyone can download it. Ken, my apologies for this misstatement. I guess that my faulty memory is to blame for the fact that I didn't recall the MacOS code was made publicly available for download. > It was > functional in a very basic way but needed some additonal love. > Okay, I haven't rolled that stuff into the Whamcloud release ... I don't think I've ever seen patches sent from you to either Oracle or Whamcloud, and unfortunately nobody on our side has had the bandwidth or user demand/funding to be pulling such changes either. > what happened there was when there was all the uncertainty with > Oracle & Lustre development I lost momentum and got caught up in > other things. I've talked with the guys at Whamcloud about bringing > at least the portability changes over, and that's all been on me; > it's certainly on my list to work on. > > I can say that at least for MacOS X, there has been interest; I can't > speak for the amount of interest, and there's a bit of a chicken and > egg problem ... people don't plan their Lustre use around MacOS X > clients because there isn't one that works well, and people don't put > work into it because there isn't people who plan their Lustre use > around it. Another alternative that has been proposed is FUSE for MacOS/Solaris. I'm not sure if FUSE exists for Windows or not. Of course, this is only useful for casual access, and not for high performance usage. >> I'd like to get some feedback from the Lustre community about removing >> the libcfs abstraction entirely, or possibly restructuring it to look >> like the Linux kernel API, and having the other platforms code against >> it as a Linux portability layer, like ZFS on Linux uses the Solaris >> Portability Layer (SPL) to avoid changing the core ZFS code. A related >> topic is whether it would be better to replace all cfs_* functions with >> standard Linux kernel functions en-masse, or migrate away from cfs_* >> functions slowly? > > The only thing I can think of is that if this is done, then officially > Lustre is going to be a Linux-only filesystem. I understand there are > real costs to maintaining the cfs layer, and I can't speak to whether or > not the loss would be worth the gains. This isn't strictly correct. It would be possible to change the libcfs portability layer to export the same API as the Linux kernel to MacOS and Windows. This would simplify getting the client into the Linux kernel, but still allow a native client on MacOS. I'm fine with keeping the abstraction layer if there is legitimate use for it. Having the MacOS client in the Lustre tree in a state where it can at least be built would allow us to make changes to the code (e.g. function renaming) with some chance of it not being totally broken immediately. Cheers, Andreas -- Andreas Dilger Whamcloud, Inc. Principal Lustre Engineer http://www.whamcloud.com/ From jjw at iu.edu Thu Mar 15 19:51:45 2012 From: jjw at iu.edu (Joshua Walgenbach) Date: Thu, 15 Mar 2012 15:51:45 -0400 Subject: [Twg] [Lustre-devel] Lustre and cross-platform portability In-Reply-To: References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> Message-ID: <4F624851.6030707@iu.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/15/2012 03:39 PM, Andreas Dilger wrote: > Another alternative that has been proposed is FUSE for > MacOS/Solaris. I'm not sure if FUSE exists for Windows or not. Of > course, this is only useful for casual access, and not for high > performance usage. > I think the FUSE for Windows project is defunct, but there is a user space filesystem project for Windows called Dokan (http://dokan-dev.net/en/) that had a release a little more than a year ago. For my part, a Lustre client on Windows or OS X would be used mostly for visualization of data, rather than being computed against so a slower user space implementation would be more than sufficient. There are a few applications that are using an SMB exported Lustre filesystem for data collection, but those applications are similarly low bandwidth. It would be nice to remove the SMB server in the middle. - -Josh - -- Joshua Walgenbach, High Performance File Systems, Indiana University -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9iSFEACgkQcqyJPuRTYp8dRgCePbaKA2P1Km80KBJM/YG6Nky1 8pgAn3RQW1i5omkhJfFZOG+0yhtXKJFB =08vB -----END PGP SIGNATURE----- From Nathan_Rutman at xyratex.com Thu Mar 15 22:53:49 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Thu, 15 Mar 2012 15:53:49 -0700 Subject: [Twg] [wc-discuss] Re: Lustre and cross-platform portability In-Reply-To: <7F9C8D28-8F0C-4B06-841D-8CE5E947ED64@whamcloud.com> References: <7F9C8D28-8F0C-4B06-841D-8CE5E947ED64@whamcloud.com> Message-ID: <73AED5C780AE05478241DB067651A92102F108AC@XYUS-EX22.xyus.xyratex.com> We had a thought that a FUSE-based client based on liblustre might make sense to avoid the NFS/Samba re-export problems (scalability, coherency), at the potential price of some performance. You mentioned on the phone this morning that someone had already done something like this in the past? If we revive this work, presumably we can drop the Mac native client. On the other hand, I agree that the baggage is large, and it will take much more work to see this through, and without a real champion and/or a real community need, I understand the desire to drop it. On Mar 15, 2012, at 11:38 AM, Andreas Dilger wrote: > On 2012-03-15, at 8:22 AM, Tyler Hawes wrote: >> Having a native Windows & Mac client is by far our company's #1 most important feature for the future. We have been seriously considering how we could help make this happen, including putting funds and/or developers toward the cause. > > Tyler, thanks for your feedback. > > While at Oracle, there was serious effort put toward having a Windows Native Client. Unfortunately, this died along with other Oracle Lustre development projects, and the proprietary code has been sitting unused inside Oracle since then. > > Similarly, the MacOS native client project started a couple of years ago, but as yet none of the code has been released. While some work was done for a Solaris server, I don't see much hope in that making progress at all. > > In both cases, I would be supportive of these efforts if there was a likelihood of them bearing fruit any time in the future, but so far I haven't seen any progress in that direction. If Oracle could somehow release the WNC code and/or NRL can overcome the government roadblocks in their path for releasing the MacOS client (MLC?), it would definitely change the direction of my thinking. > > As it stands now, we have portability code for Windows/Mac without any ability to even build the code, let alone test it, so it is just a burden for Linux. If there was real value being derived from that code (e.g. actual users), then it might be an acceptable burden. > >> I know we're in the minority on this, but I believe it's because we are not using Lustre for HPC. We use it for post-production of TV/film. There are a few others companies in our industry who have started doing this as well. My point is that, while Lustre currently is focused on the HPC crowd who seem not to care about Windows/Mac, Lustre's maturity is giving it the potential to grow into other uses besides HPC. I wouldn't call them general use, but other high-performance uses. In our industry, where there are a lot of Windows & Mac workstations that we want to connect to the Lustre storage, the Linux-only client is a major obstacle to that. I'm sure there are other industries that would benefit from this. > > Yes, the TV/film industry was one of the prime motivators for WNC. There have always been a small number of users from this market, and growth in this industry (and others) is definitely welcomed. > >> If the community wants to keep Lustre strictly HPC focused and discourage other industries from joining in, then abandoning the bridge (albeit the half-built bridge that it is) to Linux/Windows is a good way to do that. If, on the other hand, there is a desire to get some other industries involved, perhaps with more resources and contribution coming from them, then I think it's important to build on the work that has been done. In that regard, upstream Linux kernel inclusion seems like a very low priority to me. > > I think even for upstream kernel submission, if it were clear that the layering in Lustre was for a valid purpose (i.e. existing Mac/Win clients) then it might be accepted. As it stands now, we can't honestly make that argument to the kernel maintainers. > >> 2012/3/15 >> ---------- Forwarded message ---------- >> From: Andreas Dilger >> To: twg at lists.opensfs.org >> Cc: wc-discuss , lustre-discuss discuss , Lustre Devel >> Date: Wed, 14 Mar 2012 18:31:29 -0600 >> Subject: [Lustre-discuss] Lustre and cross-platform portability >> Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. >> >> As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). >> >> I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. >> >> I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? >> >> Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. >> >> The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. >> >> >> Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) >> >> Cheers, Andreas >> -- >> Andreas Dilger Whamcloud, Inc. >> Principal Lustre Engineer http://www.whamcloud.com/ >> >> >> >> >> >> >> > > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > From alexey_lyashkov at xyratex.com Fri Mar 16 06:45:48 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Fri, 16 Mar 2012 10:45:48 +0400 Subject: [Twg] [wc-discuss] Lustre and cross-platform portability In-Reply-To: <73AED5C780AE05478241DB067651A92102F108AC@XYUS-EX22.xyus.xyratex.com> References: <7F9C8D28-8F0C-4B06-841D-8CE5E947ED64@whamcloud.com> <73AED5C780AE05478241DB067651A92102F108AC@XYUS-EX22.xyus.xyratex.com> Message-ID: <4DEC0B58-6E61-4B52-AA9E-54B242031A0D@xyratex.com> If cray don't wan't to use a liblustre single thread client - liblustre is good start to create a multi platform client. As i know it's work now on Mac and FreeBSD, as Windows have a FUSE port - it's should be work in Windows also. On Mar 16, 2012, at 02:53, Nathan Rutman wrote: > We had a thought that a FUSE-based client based on liblustre might make sense to avoid the NFS/Samba re-export problems (scalability, coherency), at the potential price of some performance. You mentioned on the phone this morning that someone had already done something like this in the past? If we revive this work, presumably we can drop the Mac native client. > On the other hand, I agree that the baggage is large, and it will take much more work to see this through, and without a real champion and/or a real community need, I understand the desire to drop it. > > On Mar 15, 2012, at 11:38 AM, Andreas Dilger wrote: > >> On 2012-03-15, at 8:22 AM, Tyler Hawes wrote: >>> Having a native Windows & Mac client is by far our company's #1 most important feature for the future. We have been seriously considering how we could help make this happen, including putting funds and/or developers toward the cause. >> >> Tyler, thanks for your feedback. >> >> While at Oracle, there was serious effort put toward having a Windows Native Client. Unfortunately, this died along with other Oracle Lustre development projects, and the proprietary code has been sitting unused inside Oracle since then. >> >> Similarly, the MacOS native client project started a couple of years ago, but as yet none of the code has been released. While some work was done for a Solaris server, I don't see much hope in that making progress at all. >> >> In both cases, I would be supportive of these efforts if there was a likelihood of them bearing fruit any time in the future, but so far I haven't seen any progress in that direction. If Oracle could somehow release the WNC code and/or NRL can overcome the government roadblocks in their path for releasing the MacOS client (MLC?), it would definitely change the direction of my thinking. >> >> As it stands now, we have portability code for Windows/Mac without any ability to even build the code, let alone test it, so it is just a burden for Linux. If there was real value being derived from that code (e.g. actual users), then it might be an acceptable burden. >> >>> I know we're in the minority on this, but I believe it's because we are not using Lustre for HPC. We use it for post-production of TV/film. There are a few others companies in our industry who have started doing this as well. My point is that, while Lustre currently is focused on the HPC crowd who seem not to care about Windows/Mac, Lustre's maturity is giving it the potential to grow into other uses besides HPC. I wouldn't call them general use, but other high-performance uses. In our industry, where there are a lot of Windows & Mac workstations that we want to connect to the Lustre storage, the Linux-only client is a major obstacle to that. I'm sure there are other industries that would benefit from this. >> >> Yes, the TV/film industry was one of the prime motivators for WNC. There have always been a small number of users from this market, and growth in this industry (and others) is definitely welcomed. >> >>> If the community wants to keep Lustre strictly HPC focused and discourage other industries from joining in, then abandoning the bridge (albeit the half-built bridge that it is) to Linux/Windows is a good way to do that. If, on the other hand, there is a desire to get some other industries involved, perhaps with more resources and contribution coming from them, then I think it's important to build on the work that has been done. In that regard, upstream Linux kernel inclusion seems like a very low priority to me. >> >> I think even for upstream kernel submission, if it were clear that the layering in Lustre was for a valid purpose (i.e. existing Mac/Win clients) then it might be accepted. As it stands now, we can't honestly make that argument to the kernel maintainers. >> >>> 2012/3/15 >>> ---------- Forwarded message ---------- >>> From: Andreas Dilger >>> To: twg at lists.opensfs.org >>> Cc: wc-discuss , lustre-discuss discuss , Lustre Devel >>> Date: Wed, 14 Mar 2012 18:31:29 -0600 >>> Subject: [Lustre-discuss] Lustre and cross-platform portability >>> Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. >>> >>> As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). >>> >>> I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. >>> >>> I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? >>> >>> Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. >>> >>> The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. >>> >>> >>> Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger Whamcloud, Inc. >>> Principal Lustre Engineer http://www.whamcloud.com/ >>> >>> >>> >>> >>> >>> >>> >> >> >> Cheers, Andreas >> -- >> Andreas Dilger Whamcloud, Inc. >> Principal Lustre Engineer http://www.whamcloud.com/ >> >> >> >> From greg.matthews at diamond.ac.uk Fri Mar 16 10:11:23 2012 From: greg.matthews at diamond.ac.uk (Gregory Matthews) Date: Fri, 16 Mar 2012 10:11:23 +0000 Subject: [Twg] [wc-discuss] Re: [Lustre-devel] Lustre and cross-platform portability In-Reply-To: <4F624851.6030707@iu.edu> References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> <4F624851.6030707@iu.edu> Message-ID: <4F6311CB.8040506@diamond.ac.uk> On 15/03/12 19:51, Joshua Walgenbach wrote: > For my part, a Lustre client on Windows or OS X would be used mostly > for visualization of data, rather than being computed against so a > slower user space implementation would be more than sufficient. > There are a few applications that are using an SMB exported Lustre > filesystem for data collection, but those applications are similarly > low bandwidth. It would be nice to remove the SMB server in the > middle. why bother having a windows client if you lose the performance? We have windows based detectors and proprietary windows based analysis software that would definitely benefit from higher performance access to lustre file systems but replacing existing CIFS servers for no gain seems a bit pointless. I understand the problems that a native MS client would have, without buy-in from Microsoft it would be a nightmare to keep the MS tree and the Linux tree in sync. One possible future workaround proposed by Whamcloud at last years LUG is pNFS. GREG > > - -Josh -- Greg Matthews 01235 778658 Senior Computer Systems Administrator Diamond Light Source, Oxfordshire, UK -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom From alexey_lyashkov at xyratex.com Fri Mar 16 10:35:42 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Fri, 16 Mar 2012 14:35:42 +0400 Subject: [Twg] [wc-discuss] Re: [Lustre-devel] Lustre and cross-platform portability In-Reply-To: <4F6311CB.8040506@diamond.ac.uk> References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> <4F624851.6030707@iu.edu> <4F6311CB.8040506@diamond.ac.uk> Message-ID: Windows don't have a good OFED port so we anyway will lost a performance. but using a full client will able to use a striping information and accept a data from a several OST's with comparing a get a single data stream from a SMB. Also fuse client will able to run on any OS have a FUSE porting that is any BSD, OpenSolaris, MacOS, in additional to the windows. That is easy way to maintain a single client for many OS. On Mar 16, 2012, at 14:11, Gregory Matthews wrote: > On 15/03/12 19:51, Joshua Walgenbach wrote: >> For my part, a Lustre client on Windows or OS X would be used mostly >> for visualization of data, rather than being computed against so a >> slower user space implementation would be more than sufficient. >> There are a few applications that are using an SMB exported Lustre >> filesystem for data collection, but those applications are similarly >> low bandwidth. It would be nice to remove the SMB server in the >> middle. > > why bother having a windows client if you lose the performance? We have > windows based detectors and proprietary windows based analysis software > that would definitely benefit from higher performance access to lustre > file systems but replacing existing CIFS servers for no gain seems a bit > pointless. > > I understand the problems that a native MS client would have, without > buy-in from Microsoft it would be a nightmare to keep the MS tree and > the Linux tree in sync. One possible future workaround proposed by > Whamcloud at last years LUG is pNFS. > > GREG > >> >> - -Josh > > > -- > Greg Matthews 01235 778658 > Senior Computer Systems Administrator > Diamond Light Source, Oxfordshire, UK > > -- > This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > > > From spitzcor at cray.com Fri Mar 16 14:17:05 2012 From: spitzcor at cray.com (Cory Spitz) Date: Fri, 16 Mar 2012 09:17:05 -0500 Subject: [Twg] [wc-discuss] Lustre and cross-platform portability In-Reply-To: <4DEC0B58-6E61-4B52-AA9E-54B242031A0D@xyratex.com> References: <7F9C8D28-8F0C-4B06-841D-8CE5E947ED64@whamcloud.com> <73AED5C780AE05478241DB067651A92102F108AC@XYUS-EX22.xyus.xyratex.com> <4DEC0B58-6E61-4B52-AA9E-54B242031A0D@xyratex.com> Message-ID: <4F634B61.9080306@cray.com> Cray no longer deploys liblustre. Therefore, we probably take a 'don't care' position in the discussion about the future of liblustre or libcfs. Thanks, -Cory On 03/16/2012 01:45 AM, Alexey Lyashkov wrote: > If cray don't wan't to use a liblustre single thread client - liblustre is good start to create a multi platform client. > As i know it's work now on Mac and FreeBSD, as Windows have a FUSE port - it's should be work in Windows also. > > > On Mar 16, 2012, at 02:53, Nathan Rutman wrote: > >> We had a thought that a FUSE-based client based on liblustre might make sense to avoid the NFS/Samba re-export problems (scalability, coherency), at the potential price of some performance. You mentioned on the phone this morning that someone had already done something like this in the past? If we revive this work, presumably we can drop the Mac native client. >> On the other hand, I agree that the baggage is large, and it will take much more work to see this through, and without a real champion and/or a real community need, I understand the desire to drop it. >> >> On Mar 15, 2012, at 11:38 AM, Andreas Dilger wrote: >> >>> On 2012-03-15, at 8:22 AM, Tyler Hawes wrote: >>>> Having a native Windows & Mac client is by far our company's #1 most important feature for the future. We have been seriously considering how we could help make this happen, including putting funds and/or developers toward the cause. >>> >>> Tyler, thanks for your feedback. >>> >>> While at Oracle, there was serious effort put toward having a Windows Native Client. Unfortunately, this died along with other Oracle Lustre development projects, and the proprietary code has been sitting unused inside Oracle since then. >>> >>> Similarly, the MacOS native client project started a couple of years ago, but as yet none of the code has been released. While some work was done for a Solaris server, I don't see much hope in that making progress at all. >>> >>> In both cases, I would be supportive of these efforts if there was a likelihood of them bearing fruit any time in the future, but so far I haven't seen any progress in that direction. If Oracle could somehow release the WNC code and/or NRL can overcome the government roadblocks in their path for releasing the MacOS client (MLC?), it would definitely change the direction of my thinking. >>> >>> As it stands now, we have portability code for Windows/Mac without any ability to even build the code, let alone test it, so it is just a burden for Linux. If there was real value being derived from that code (e.g. actual users), then it might be an acceptable burden. >>> >>>> I know we're in the minority on this, but I believe it's because we are not using Lustre for HPC. We use it for post-production of TV/film. There are a few others companies in our industry who have started doing this as well. My point is that, while Lustre currently is focused on the HPC crowd who seem not to care about Windows/Mac, Lustre's maturity is giving it the potential to grow into other uses besides HPC. I wouldn't call them general use, but other high-performance uses. In our industry, where there are a lot of Windows & Mac workstations that we want to connect to the Lustre storage, the Linux-only client is a major obstacle to that. I'm sure there are other industries that would benefit from this. >>> >>> Yes, the TV/film industry was one of the prime motivators for WNC. There have always been a small number of users from this market, and growth in this industry (and others) is definitely welcomed. >>> >>>> If the community wants to keep Lustre strictly HPC focused and discourage other industries from joining in, then abandoning the bridge (albeit the half-built bridge that it is) to Linux/Windows is a good way to do that. If, on the other hand, there is a desire to get some other industries involved, perhaps with more resources and contribution coming from them, then I think it's important to build on the work that has been done. In that regard, upstream Linux kernel inclusion seems like a very low priority to me. >>> >>> I think even for upstream kernel submission, if it were clear that the layering in Lustre was for a valid purpose (i.e. existing Mac/Win clients) then it might be accepted. As it stands now, we can't honestly make that argument to the kernel maintainers. >>> >>>> 2012/3/15 >>>> ---------- Forwarded message ---------- >>>> From: Andreas Dilger >>>> To: twg at lists.opensfs.org >>>> Cc: wc-discuss , lustre-discuss discuss , Lustre Devel >>>> Date: Wed, 14 Mar 2012 18:31:29 -0600 >>>> Subject: [Lustre-discuss] Lustre and cross-platform portability >>>> Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. >>>> >>>> As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). >>>> >>>> I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. >>>> >>>> I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? >>>> >>>> Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. >>>> >>>> The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. >>>> >>>> >>>> Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) >>>> >>>> Cheers, Andreas >>>> -- >>>> Andreas Dilger Whamcloud, Inc. >>>> Principal Lustre Engineer http://www.whamcloud.com/ >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> >>> Cheers, Andreas >>> -- >>> Andreas Dilger Whamcloud, Inc. >>> Principal Lustre Engineer http://www.whamcloud.com/ >>> >>> >>> >>> > From kenh at cmf.nrl.navy.mil Fri Mar 16 14:38:27 2012 From: kenh at cmf.nrl.navy.mil (Ken Hornstein) Date: Fri, 16 Mar 2012 10:38:27 -0400 Subject: [Twg] Lustre and cross-platform portability In-Reply-To: Message-ID: <201203161438.q2GEcSVL005257@hedwig.cmf.nrl.navy.mil> >Ken, my apologies for this misstatement. I guess that my faulty memory >is to blame for the fact that I didn't recall the MacOS code was made >publicly available for download. No problem. Back when I gave the talk at LUG the source wasn't available yet due to issues here, but we got that worked out and I was pushing my changes to a publically available Oracle git repo. I did send out email to everyone about that, but I'm sure it was easy to miss. >I don't think I've ever seen patches sent from you to either Oracle or >Whamcloud, and unfortunately nobody on our side has had the bandwidth or >user demand/funding to be pulling such changes either. Well, I did actually submit patches to Oracle to start the process of working out at least the portability issues, but I believe that was when Oracle started to implode the Lustre group so things sort of stalled. I'll take 75% of the blame for that if we assign 25% to Larry Ellison :-) >This isn't strictly correct. It would be possible to change the libcfs >portability layer to export the same API as the Linux kernel to MacOS >and Windows. This would simplify getting the client into the Linux >kernel, but still allow a native client on MacOS. Well ... that shifts the burden to cross-platform people basically having to re-implment the Linux kernel. For some things, that's possible without too much pain. For other things, it's not. --Ken From kenh at cmf.nrl.navy.mil Fri Mar 16 14:46:22 2012 From: kenh at cmf.nrl.navy.mil (Ken Hornstein) Date: Fri, 16 Mar 2012 10:46:22 -0400 Subject: [Twg] [wc-discuss] Re: [Lustre-devel] Lustre and cross-platform portability In-Reply-To: Message-ID: <201203161446.q2GEkNMO005460@hedwig.cmf.nrl.navy.mil> >Also fuse client will able to run on any OS have a FUSE porting that is >any BSD, OpenSolaris, MacOS, in additional to the windows. That is >easy way to maintain a single client for many OS. It is, unfortunately, not quite that simple. I can't claim to be a FUSE expert, but I've been paying attention to it on other platforms. From what I can tell, FUSE works great on Linux, but on other platforms the support is iffy. Also, it's not quite implemented the same on other operating systems as it is on Linux, making porting a Linux FUSE module to other platforms not trivial; from what I've seen, this is due to the Linux filesystem interface versus the vnode interface used by every Unix except Linux (and this is part of what makes Lustre hard to port). I guess what I'm saying is that don't fall into the underwear gnomes trap of thinking: 1) Get liblustre working with FUSE 2) ??? 3) Lustre client everywhere! It might make it easier, but I doubt it will make it easy. --Ken From Allen.Todd at sig.com Fri Mar 16 15:06:04 2012 From: Allen.Todd at sig.com (Todd, Allen) Date: Fri, 16 Mar 2012 15:06:04 +0000 Subject: [Twg] [wc-discuss] Re: [Lustre-devel] Lustre and cross-platform portability In-Reply-To: <4F6311CB.8040506@diamond.ac.uk> References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> <4F624851.6030707@iu.edu> <4F6311CB.8040506@diamond.ac.uk> Message-ID: <097F86FD722A2445B248BC51CC5DB88D1702BC@xchmbbal504.ds.susq.com> On Friday, March 16, 2012 6:11 AM, Gregory Matthews wrote: > why bother having a windows client if you lose the performance? We have > windows based detectors and proprietary windows based analysis software > that would definitely benefit from higher performance access to lustre > file systems but replacing existing CIFS servers for no gain seems a bit > pointless. >From the perspective of my firm, the benefit of a windows lustre client (native or fuse-based), even one that performs at 10% or 20% of the linux client, is the improved scalability that it offers. Our current solution uses 4 samba gateways per hundred windows servers to achieve acceptable bandwidth, but that bandwidth can be cut to an unacceptable trickle if a large wave of native linux clients simultaneously accesses the filesystem. We have repeatedly asked our Microsoft HPC contacts to intervene to either fund a native windows client or to get oracle to release the existing one, since scalable storage is a big hole in the Microsoft HPC toolkit. Obviously, nothing has come out of that. It seems Microsoft is banking on pNFS, eventually working in this space. Allen Todd ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. From lee at sandia.gov Fri Mar 16 16:03:36 2012 From: lee at sandia.gov (Ward, Lee) Date: Fri, 16 Mar 2012 16:03:36 +0000 Subject: [Twg] [EXTERNAL] [Lustre-devel] Lustre and cross-platform portability In-Reply-To: <5609A4E0-2356-400F-9D53-F71B7007338C@whamcloud.com> References: <5609A4E0-2356-400F-9D53-F71B7007338C@whamcloud.com> Message-ID: On Mar 14, 2012, at 6:31 PM, Andreas Dilger wrote: > Whamcloud and EMC are jointly investigating how to be able to contribute the Lustre client code into the upstream Linux kernel. > > As a prerequisite to this, EMC is working to clean up the Lustre client code to better match the kernel coding style, and one of the anticipated major obstacles to upstream kernel submission is the heavy use of code abstraction via libcfs for portability to other operating systems (most notably MacOS and WinNT, but also for liblustre, and potentially *BSD). > > I have no information that the WinNT project will ever be released by Oracle, and as yet there has not been any code released from the MacOS port, so the libcfs portability layer is potentially exacting a high cost in code maintenance and complexity (CLIO being a prime example) for no apparent benefit. Similarly, the liblustre client needs a portability layer for userspace, and suffers from the same apparent lack of interest or users. > > I'd like to get some feedback from the Lustre community about removing the libcfs abstraction entirely, or possibly restructuring it to look like the Linux kernel API, and having the other platforms code against it as a Linux portability layer, like ZFS on Linux uses the Solaris Portability Layer (SPL) to avoid changing the core ZFS code. A related topic is whether it would be better to replace all cfs_* functions with standard Linux kernel functions en-masse, or migrate away from cfs_* functions slowly? > > Also, we're planning on deprecating the liblustre client code, due to lack of interest/usage. The current code is in disrepair, and we've been keeping it around for years without any benefit, and while I was one of the strongest advocates for keeping it in our back pocket in case of future needs, I don't see that materializing in the future. Relatively true I think, unfortunately. In the past, we provided direct support of Lustre for processes running on light-weight kernels via liblustre. I am aware that DOE/NNSA (more than just Sandia) believes light-weight kernels are the future but it seems that it may be quite a while, yet, before we would be forced into that choice. I don't see NNSA having some sort of heart-attack over your obfuscating liblustre then. --Lee > > The liblustre code would be left in the tree for now, in case someone from the community is interested to get it working and maintain it, and it may be updated on a best effort basis. If nobody steps forward to do this work, the liblustre code would be deleted from the development branch in a year or so. > > > Unfortunately, after starting this thread, I may not be able to reply to questions in a timely manner due to vacation. I look forward to a thread that concludes with unanimous agreement from all parties. :-) > > Cheers, Andreas > -- > Andreas Dilger Whamcloud, Inc. > Principal Lustre Engineer http://www.whamcloud.com/ > > > > > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel > From alexey_lyashkov at xyratex.com Sat Mar 17 10:42:48 2012 From: alexey_lyashkov at xyratex.com (Alexey Lyashkov) Date: Sat, 17 Mar 2012 14:42:48 +0400 Subject: [Twg] [wc-discuss] [Lustre-devel] Lustre and cross-platform portability In-Reply-To: <201203161446.q2GEkNMO005460@hedwig.cmf.nrl.navy.mil> References: <201203161446.q2GEkNMO005460@hedwig.cmf.nrl.navy.mil> Message-ID: Trust me. I (with two friends) write a FUSE client for a FreeBSD in past. Most issues related to a porting libcfs/libsysio to FreeBSD, but anyway patch isn''t huge ~200kb. I have plan to send that patch in near time, as 2.3 landing window open. On Mar 16, 2012, at 18:46, Ken Hornstein wrote: >> Also fuse client will able to run on any OS have a FUSE porting that is >> any BSD, OpenSolaris, MacOS, in additional to the windows. That is >> easy way to maintain a single client for many OS. > > It is, unfortunately, not quite that simple. > > I can't claim to be a FUSE expert, but I've been paying attention > to it on other platforms. From what I can tell, FUSE works great > on Linux, but on other platforms the support is iffy. Also, it's > not quite implemented the same on other operating systems as it is > on Linux, making porting a Linux FUSE module to other platforms not > trivial; from what I've seen, this is due to the Linux filesystem > interface versus the vnode interface used by every Unix except Linux > (and this is part of what makes Lustre hard to port). > > I guess what I'm saying is that don't fall into the underwear gnomes > trap of thinking: > > 1) Get liblustre working with FUSE > 2) ??? > 3) Lustre client everywhere! > > It might make it easier, but I doubt it will make it easy. > > --Ken From carrier at cray.com Mon Mar 19 05:54:12 2012 From: carrier at cray.com (John Carrier) Date: Mon, 19 Mar 2012 05:54:12 +0000 Subject: [Twg] TWG meeting minutes for 2012-03-15 Message-ID: Attached are the meeting minutes from last week's TWG call. This meeting continued the discussion of features that the TWG should consider as we update our requirements and roadmaps document. The following is a list of features we have discussed this year: - single client performance improvements - migration (OST rebalancing, dynamic layout, async mirroring, etc) - small I/O performance improvements - dynamic LNET configuration - POSIX extensions : file sets - POSIX extensions : scalable opens - health networks In addition we reviewed the impact of EMC's efforts to move the Lustre client into the Linux kernel. The community needs to come to a consensus on the importance of this effort versus maintaining OS interoperability without forking the code. We will continue this discussion this week: Date/Time : 03/22/2012 @ 9:30a PDT/12:30p EDT Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 If there are no new features to discuss, we need to begin converting features to requirements in preparation for making a recommendation to the OpenSFS board. Note that minutes from past meetings are available in the archives of both the twg and discuss lists at http://lists.opensfs.org. Please post corrections/questions/comments to discuss at lists.opensfs.org. --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- OpenSFS Technical Working Group Meeting minutes : 2012-03-15 Concall: start : 9:30a PT, adjourn : 10:45a PT Next meeting: Thursday, 03/22/2012 @ 9:30a PDT/12:30p EDT Dial-in numbers are 1-715-726-4994 or 1-866-304-8294 Meeting ID and password are 72090 Attending: Name Organization email ------------------- ------------- --------------------------------- Diego Moreno Bull Diego.Moreno-Lazaro at bull.net Cory Spitz Cray spitzcor at cray.com John Carrier Cray carrier at cray.com Alex Kulyavtsev FNAL aik at fnal.gov Joshua Walgenbach IU jjw at indiana.edu Norm Morse OpenSFS normanrmorse at gmail.com Dave Dillow ORNL dillowda at ornl.gov Eric Barton Whamcloud eeb at whamcloud.com Andreas Dilger Whamcloud adilger at whamcloud.com Nathan Rutman Xyratex nathan_rutman at xyratex.com Agenda: continue requirements discussion for 2013 roadmap Notes outline corrections requirements - single client performance improvements - migration / replication - dynamic LNET configuration - POSIX extensions : file sets - POSIX extensions : scalable opens - health networks EMC to submit client upstream to kernel.org corrections: Dave was asked to make the following correction: the test cluster that OpenSFS is placing at IU will be used for verifying the functionality of SMP affinity and the online lfsck not just the distributed namespace code. Requirements: Last week we made progress identifying the features and requirements that have been in the works for some time (eg NRS, HSM) do not need any OpenSFS funding to land. Xyratex is moving NRS forward and has started large scale testing with different policies. CEA has HSM in hand with landings planned for 2.2, 2.3, and 2.4 New features we have discussed in the last few weeks that remain on the list are: - single client performance improvements - migration / replication - dynamic LNET configuration * single client performance improvements Whamcloud has an experimental option to increase the number of RPCs from a single client that needs funding to complete testing and integration. * migration and replication We spent a good bit of time defining terms: Replication is the process of copying data from one file system to another. One-way replication is possible today with Lustre rsync. There is no coherency between the remote copy back to the original, but remote replication is sufficient as nearline backup for disaster recovery. The current rsync walks the namespace to look for changes that require files to be copied to the remote destination. Xyratex has developed a tool that does the copy more intelligently using the change logs to copy only files that have changed. Migration moves file objects within the same file system. It relies on the layout lock feature added to Lustre for the HSM interface to ensure that object migration maintains file coherency. The current layout lock implementation will require work to manage files that are not at rest. There are several use cases described that are driving the need for dynamic layous and file migration: - OST rebalancing move objects between OSTs to more evenly distribute free space among the OSTs; distribute objects to new OSTs added to expand the file system. - OST emptying move objects off of an OST before it is replaced or removed from the file system - dynamic layouts allow file layouts to change as the file grows or access patterns change - async mirroring create multiple copies of a file within the same file system namespace (ala Hadoop). This use case requires addition consistency checks to enable real-time mirroring. There were several architectural changes mentioned during this discussion. - client aware layout changes currently the client is unware of layout changes and relies on cascading timeouts to acknowledge layout changes. need to make the client aware of the changes as they happen. - persistent storage pools Andreas calls this making storage pools first class citizens. The current OST pool definition is an ephemeral name for a collection of OSTs. The pool needs to be peristent to enable, for example, quotas by pool. - unified storage target Stop thinkg of storage as MDTs or OSTs. Intead think of storage in terms of performance for different workloads. Improve small I/O performance by using storage tuned for this access pattern. Then restripe the file to different storage as the file gets larger. (Eric mentioned the "giga+" concept, but I'm not sure how Giga+ applies.) - dynamic layout for subset of a file CEA will restore a part of a file, rather than the whole, from an archive to speed access to the critical data. Migration will involve a number of inter-related features. Instead of stating specific features, we should identify the user requirements. For example, a unified storage target might be the feature tracked during development, but it could just as well be motivated by improving small file performance as facilitating dynamic layouts for migration. * dynamic LNET configuration The goal is to make LNET configuration more like IP configuration using admin commands to add/remove interfaces dynamically. Current routes are static and requires LNET (ie the file system) to restart to capture configuration changes. New features mentioned during this meeting: * POSIX extensions : file sets Alex mentioned the need for file sets to ease migration of files from tape to disk. GPFS has this today. Eric implied that this is a new data structure, not just a new tag on the file namespace. If done right, the set will appear in the directory and quotas can be allocated on the set not just the component files. Andreas thought changing a file set via single command, rather than traversing the directory tree could lead to snapshots. * POSIX extensions : scalable opens Andreas said the kernel just got an "open by handle" system call. It now needs to get wired into Lustre. * health networks We discussed this feature last year as a means of providing lower latency fault detection and improved error handling. It is a scalability feature that by providing an active, deterministic mechanism for communicating system status will avoid the sequence of cascading timeouts that limits Lustre at scale. Cray has found that the current timeout mechanism using pings is very difficult to tune at scale. Other Business : EMC to put client into upstream kernel Andreas sent an email to the lustre-discuss and twg reflectors describing EMC's efforts to move the Lustre client upstream into the Linux kernel in order to reduce Lustre's barrier to entry. We had a lively discussion about the impact this change would have on keeping the client interoperable with other operating systems. Putting the client into the kernel drew analogies with OFED. It is significant to note that the windows OFA stack is completely different from the Linux OFA stack (in this respect, OFED has two separate stacks). Lustre currently has a single client and a portability layer to map the client to different operating systems. Removing Lustre's portability layer (libcfs) will negatively impact the Lustre client on MacOS. The windows native client would also be affected, but that code is tied up at Oracle. Unless the Lustre is being actively developed on those platforms, it is hard to convince the Linux kernel to accept portability layers. We discussed using liblustre on FUSE to create a compatbility layer for the MAC and Windows clients, but performance under FUSE is not optimal. This would cripple Lustre on these OSs. Including the client in the kernel will reduce the barrier of entry to use Lustre. It will also clean up the client code and remove accumulated cruft. On the other hand, in addition to the loss of OS interoperability, such a change will increase the support window for any given Lustre version since a given kernel version is available on the market longer. Whamcloud is seeking community feedback on what parts of the client code people are using. Can EMC pull-out old code (such as liblustre) to make a linux client acceptable to the kernel? The goal will be to keep the upstream client in sync with the upstream tree. Once the code lands, however, it is not clear who will be the upstream maintainer. It is a concern that changes made for a kernel landing won't get pushed back to the Lustre tree. [meeting adjourned : 10:45a PT] From aik at fnal.gov Mon Mar 19 22:41:18 2012 From: aik at fnal.gov (Alex Kulyavtsev) Date: Mon, 19 Mar 2012 17:41:18 -0500 Subject: [Twg] TWG meeting minutes for 2012-03-15 In-Reply-To: References: Message-ID: > * migration and replication > We spent a good bit of time defining terms: > > Replication is the process of copying data from one file > system to another. > > Migration moves file objects within the same file system. I would suggest we do not limit ourselves using term "replication" only for remote replication, i.e. "replication" vs. "migration" as in remote vs. local. Term "replication" is used both for remote replication (e.g. Grid 'Replica Catalog') and for local (e.g. replication in HDFS). It will be difficult to substitute term "File Replica." "Replicaton" for creating exact copy of the data, local or remote. "Migration" for relocation of the data where the original is removed. "[continuous] remote replication" or "filesystem replication" for existing Xyratex implementation where use of simple "replication" can cause ambiguity. I think in most cases the meaning 'remote' or 'local' replication is clear from context. Alex. From carrier at cray.com Wed Mar 21 15:58:32 2012 From: carrier at cray.com (John Carrier) Date: Wed, 21 Mar 2012 15:58:32 +0000 Subject: [Twg] no TWG meeting 3/22 Message-ID: Dave and I are both in all day meetings tomorrow and will not be able to run our scheduled twg call. Our next meeting will be Thursday, 29 March. In the meantime, please review the notes from our last few meetings. There you will notice that there are a number of features shared among the possible migration/replication use cases. After last week's meeting, Eric suggested that all of these (dynamic layouts, standard targets, OST rebalancing, active mirroring, small i/o performance, pool quotas, etc.) could fit under the heading of "storage pool management". This would allow development of the needed foundational work that all of these features share. What do you think? Please send your comments to discuss at lists.opensfs.org. Thanks, --jc -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nathan_Rutman at xyratex.com Wed Mar 21 18:29:19 2012 From: Nathan_Rutman at xyratex.com (Nathan Rutman) Date: Wed, 21 Mar 2012 11:29:19 -0700 Subject: [Twg] [Lustre-devel] [wc-discuss] Re: Lustre and cross-platform portability In-Reply-To: <097F86FD722A2445B248BC51CC5DB88D1702BC@xchmbbal504.ds.susq.com> References: <201203151845.q2FIjcXV029439@hedwig.cmf.nrl.navy.mil> <4F624851.6030707@iu.edu> <4F6311CB.8040506@diamond.ac.uk> <097F86FD722A2445B248BC51CC5DB88D1702BC@xchmbbal504.ds.susq.com> Message-ID: <73AED5C780AE05478241DB067651A92102F108DF@XYUS-EX22.xyus.xyratex.com> On Mar 16, 2012, at 8:06 AM, Todd, Allen wrote: > > On Friday, March 16, 2012 6:11 AM, Gregory Matthews wrote: >> why bother having a windows client if you lose the performance? We have >> windows based detectors and proprietary windows based analysis software >> that would definitely benefit from higher performance access to lustre >> file systems but replacing existing CIFS servers for no gain seems a bit >> pointless. 1. Re-exporting Lustre via CIFS or NFS isn't scalable to very large numbers of Windows clients 2. Re-exporting Lustre via CIFS or NFS can have coherency problems when multiple re-exporters are involved 3. Wide-striped file access would likely have greater performance on a native client than via a re-exported single pipe, for some value of "wide". > >> From the perspective of my firm, the benefit of a windows lustre client (native or fuse-based), even one that performs at 10% or 20% of the linux client, is the improved scalability that it offers. Our current solution uses 4 samba gateways per hundred windows servers to achieve acceptable bandwidth, but that bandwidth can be cut to an unacceptable trickle if a large wave of native linux clients simultaneously accesses the filesystem. > > We have repeatedly asked our Microsoft HPC contacts to intervene to either fund a native windows client or to get oracle to release the existing one, since scalable storage is a big hole in the Microsoft HPC toolkit. Obviously, nothing has come out of that. It seems Microsoft is banking on pNFS, eventually working in this space. > > Allen Todd > > > > > ________________________________ > > IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. > _______________________________________________ > Lustre-devel mailing list > Lustre-devel at lists.lustre.org > http://lists.lustre.org/mailman/listinfo/lustre-devel From shay at opensfs.org Tue Mar 27 19:02:50 2012 From: shay at opensfs.org (Shay Seager) Date: Tue, 27 Mar 2012 09:02:50 -1000 Subject: [Twg] LUG is just around the corner, are you ready? Message-ID: Hello All, LUG 2012 is less than a month away; April 23rd-April 25th. We recommend you fly in Sunday April 22nd in the morning because we have some great events for you on Sunday: April 22, 2012 2:00-5:00pm the Benchmarking Work Group will be holding our first face to face meeting. The “Benchmarking Work Group” will focus on identifying and defining I/O workloads for the various Lustre markets, and then define IO generation benchmarking tools that best emulates those workloads. You are invited to help contribute to this important initiative. Pre LUG 2012 Party Sponsored by DataDirect Networks and Whamcloud The annual Lustre User Group is being held in Austin, TX, this year, with all the great Lustre developers, users and vendors in attendance. If you’re coming, you know some of the best connections are made outside of the talks. DDN and Whamcloud are providing beers & wine to help get it started. Come to Zax Restaurant & Bar, right next to the Hyatt Regency where LUG will be held, to kick off the week in a relaxed atmosphere. No hype, no pressure, just lots of local beers, wine and Lustre superstars. We’re closing the restaurant down, it’s just us from 7pm – 10pm. Hope to see you there! Location: Zax Restaurant & Bar 312 Barton Springs Rd, Austin, TX 78704 (across from the Hyatt Regency) Date/Time: Sun, April 22, 7-10pm RSVP: Are you kidding? Just come on by. On Wednesday we will also have 3 events hosted by OpenSFS, everyone is encouraged to attend, we recommend that you set your flights for Wednesday evening or Thursday morning (note, there are government rate rooms on Wednesday night, so book them now before they are all gone): 1pm – 2pm OpenSFS General Meeting 2pm-3pm OpenSFS Technical Working Group Meeting 3pm-4pm OpenSFS Community Development Working Group Meeting Check out our website for the whole program http://www.opensfs.org/lug/program If you have not registered your ticket, your hotel room, or your shuttle follow this link http://www.opensfs.org/lug/registration for all the resources you need to get discounted room rates and shuttle services. At LUG we will have 2 break out rooms that will be available 24-7 throughout the event and are available for private meetings and break-out sessions. Each room holds 20-30 people. If you would like to reserve room space, please reply to this e-mail (Shay Seager) with your preferred time/date. We want to make sure that this is the best LUG you have experienced, so please reply to this e-mail (to Shay Seager) to let us know if you have any dietary or special needs. (note - our menu already accommodates special diets such as vegan, vegetarian, paleo, and atkins) We also want to thank our generous sponsors: Bright Computing, Bull, Cray, DDN, Dell, EMC, Intel, Net App, RAID Inc., SGI, Terascala, Whamcloud, Xyratex. Best, Shay - - - Shay Seager Open SFS Secretary www.opensfs.org (925) 290-7641 shay at opensfs.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From carrier at cray.com Thu Mar 29 00:59:07 2012 From: carrier at cray.com (John Carrier) Date: Thu, 29 Mar 2012 00:59:07 +0000 Subject: [Twg] OpenSFS TWG call 3/29/2012 Message-ID: We have 4 more weeks before LUG. On tomorrow's call, I would like to start creating a priority list of requirements to guide the next development cycle for discussion at our F2F meeting on 4/25 in Austin. I've summarized the current feature discussion into the following requirements: - improve single client performance - improve storage pool management to address : OST rebalancing dynamic layout async mirroring, etc - dynamic LNET configuration - extend POSIX for Lustre access patterns to address : file sets scalable opens - reduce Lustre's dependence on timeouts to address : health networks We need to create statements around these requirements for the OpenSFS board. Are there other features or requirements to discuss? For example, how do we resolve the discussion about the Lustre client? thanks --jc Date/Time : 03/29/2012 @ 9:30a PDT/12:30p EDT Dial: 1-715-726-4994, 1-866-304-8294 ID & Password: 72090 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Roman_Grigoryev at xyratex.com Thu Mar 29 13:07:28 2012 From: Roman_Grigoryev at xyratex.com (Roman Grigoryev) Date: Thu, 29 Mar 2012 17:07:28 +0400 Subject: [Twg] your opinion about testing improvements Message-ID: <4F745E90.2000001@xyratex.com> Hi all, I'm working in testing team in Xyratex. After some time which I(and our team) have spent on testing and automation activities I would like try to ask community opinion about two problems and suggestion in testing. This not closed list of testing problems but I think it is good enough for start discussion. Problem 1 Currently Lustre and test are living in one code space and build in one time,and often have specific dependencies between test and code. This situation directly affect 1) interoperability testing between different version. (because testing is started from client which have different test framework then server and client remotely execute test framework as their own. just copying tests for equalization could not work with big difference between versions) 2) it is not simple execute(especially in automation) testing for test. F.e. a bug is fixed, the test on it added. Executing the test on an old revision(probably on a previous release) should show failed test result. But with big difference between versions where fixed and where execute test-framework can fail to start. Possible solution: split Lustre and lustre tests in code and build levels. It means that lustre and tests will not be connected on code revision, only by logic, f.e. via keywords. Also should be added in same time an abstraction level in test framework which allows to execute lustre utils from different version of lustre. Problem 2 (to avoid term problems, I call there: sanity = test suite, 130 = test, 130c and 130a = test cases) Different test cases, ended with letter(f.e. 130c), have an different idea of dependencies. Some test cases have dependences to previous test cases, some don't have. All they now can be executed with "ONLY" parameter and all they have separated item in result yaml file as just separated tests( which doesn't have test cases ended with letter, f.e. sanity 129). Also, tests which have testcases and don't have their own body can be execute with ONLY parameter( but doesn't have their special result). So, logically, all test which can be execute via using ONLY must be not depended to other tests. But we have test which depended. Moreover, some developers prefer to consider testcases as step of full one test. What is entities which I call "testcases" and "test" from your point of view? Answer of this question affect automated test execution and test development, and maybe ask some test-framework changes. -- Thanks, Roman