Authors
Updated
10 Mar 2023Form Number
LP1698PDF size
21 pages, 408 KBAbstract
The WEKA Data Platform is a high-performance storage solution designed to handle the demands of data-intensive applications in all HPC and AI workloads. This document provides best practices for the use of the WEKA Data Platform combined with Lenovo’s industry-leading ThinkSystem servers in HPC and AI.
Change History
Changes in the March 10, 2023 update:
- Added tables of part numbers - WEKA software licenses section
Introduction
The technological advances of today have created a unique set of challenges for data-intensive applications, such as those used for artificial intelligence (AI), machine learning (ML), deep learning (DL) and high-performance computing (HPC) within, for example, financial analytics, genomics and life sciences. These complex applications require maximum IO performance, but legacy storage solutions were not built to handle the scale of these workloads.
The WEKA® Data Platform is uniquely built to solve the storage challenges of leading-edge applications. WEKA eliminates the complexity and compromises associated with legacy storage (DAS, NAS, SAN) while still providing the enterprise features and benefits of traditional storage solutions, all at a fraction of the cost. WEKA is designed to meet the stringent storage demands of data-intensive workloads and accelerates the process of obtaining insight from mountains of data.
To meet the requirements of these workloads, Lenovo and WEKA have engineered a joint solution that provides maximum acceleration and reduced AI training times. This solution consists of the Lenovo ThinkSystem SR630 V2 server platform, WEKA software, and NVIDIA networking. This combined solution is ideal for data-intensive applications, delivering over 10x more performance than all-flash scale-out NAS and 3x more than locally attached NVMe SSDs. It delivers faster performance than a local file system and can push multiple high-performance network links to their limits with up to 40GB/s. With the Lenovo EveryScale Design Architecture for WEKA performance scales linearly as the infrastructure grows, allowing for the most efficient utilization of expensive compute resources.
Furthermore, the Lenovo and WEKA solution supports organizations to leverage the benefits of a hybrid cloud model, allowing them to take advantage of on-demand public compute resources for cloud-bursting, remote backup, and disaster recovery.
The benefits of this high-performance storage solution extend beyond specific performance and capacity requirements. This modern infrastructure offers IT optimization, data center agility, and data transformation for AI and analytics, resulting in faster time to value and insight. Furthermore, the solution can help organizations gain a competitive advantage, deliver customer value, and grow their businesses.
Use cases
WEKA solves the common storage challenges by eliminating the chokepoints that impact application performance. It is well-suited for demanding environments that need shareable storage with low latency, high-performance, and cloud scalability.
Example use cases include:
- Artificial Intelligence (AI) and Machine Learning (ML), including AIOps and MLOps
- Life sciences including genomics, Cryo-EM, pharmacometrics (NONMEM, PsN)
- Financial trading, including backtesting, time-series analysis, and risk management
- Engineering DevOps
- Electronic Design and Automation (EDA) Manufacturing EDA simulation and verification, software builds, CFD, and thermal imaging
- Media and Entertainment Nonlinear editing, VFX rendering, transcoding, and content delivery
- High-Performance Computing (HPC) GPU pipeline acceleration
- Academic/Government Climate change simulation, computational physics, earthquake studies, space research, and intelligence
Software platform
WEKA is an advanced software solution that combines existing technologies with cutting-edge engineering innovations to deliver a powerful, simplified solution that would have traditionally required multiple storage systems. By leveraging NVMe flash, the approach offers unparalleled performance for a range of workloads, from large files to random and metadata-heavy operations. Furthermore, it is designed to run on commodity server infrastructure, eliminating the need for specialized hardware.
WEKA Software
The distributed parallel file system also includes integrated tiering that expands the namespace to and from hard disk drive (HDD) based object storage, making data access and management easy. The intuitive graphical user interface allows administrators to quickly and easily manage exabytes of data without specialized storage training. With WEKA, businesses can enjoy high performance and optimized storage capabilities with ease.
WEKA provides a rich enterprise feature set, including local snapshots and remote snapshots to the cloud, clones, automated tiering, cloud-bursting, dynamic cluster rebalancing, private cloud multi-tenancy, backup, encryption, authentication, key management, user groups, quotas with advisory, soft and hard parameters and much more.
WEKA’s unique snap-to-object feature allows users to easily create a replica of the production data and instantly push it to any S3 object store — on-premises or in the cloud — enabling snapshot-based replication. The cloud copy can be used for workload migration to another application cluster or to provide a fast recovery point objective (RPO) service guarantee.
Figure 11: WEKA combines NVMe flash with cloud object storage in a single global namespace
The WEKA Data Platform’s unique architecture, as shown in Figure 1, is radically different from legacy storage systems, appliances, and hypervisor-based software-defined storage solutions because it overcomes not only traditional storage scaling and file sharing limitations but also allows parallel file access via multiple protocols.
Multi-Protocol access
Clients with the appropriate credentials and privileges can create, modify, and read data using any of the following protocols:
- POSIX
- NVIDIA® GPUDirect® Storage (GDS)4
- NFS (Network File System) v3 and v4.1
- SMB (Server Message Block) v2 and v3
- S3 (Simple Storage Service)
NOTE: Many non-traditional applications and data systems can take advantage of the POSIX capabilities that WEKA provides as it appears as a local mount. One example of this is HDFS (Hadoop Distributed File System); WEKA’s POSIX connector can directly mount to Hadoop nodes to provide very high performance.
Data written to the file system from one protocol can be read via another one, so the data is fully shareable among applications.
Server platform
The building block of Lenovo’s WEKA storage node, the Lenovo ThinkSystem SR630 V2, has been designed to take advantage of the features of the 3rd generation Intel Xeon Scalable processors, support for 3200 MHz memory and PCIe Gen 4.0 support. The server also offers onboard NVMe PCIe ports that allow direct connections to 10x NVMe SSDs, which results in faster access to store and access data.
The Lenovo EveryScale WEKA Storage server
The following minimum entry-level building block for an initial storage deployment gives you the ability to scale out to much larger capacities and performance levels by adding more servers:
- 6x Lenovo ThinkSystem SR630 V2 servers
- 128 TB to 614 TB of usable storage capacity
The 128 TB usable capacity option provides a great entry point with the ability to scale for small environments. The 614 TB option is perfect for those who need a capacity-optimized storage solution with the best cost per GB. This assumes 4+2P RAID code.
Where a larger cluster is expected, the following minimum entry-level building block is recommended and allows use of 8+2P RAID code which has reduced overhead when scaling to larger-scale clusters:
- 8x Lenovo ThinkSystem SR630 V2 servers
- 204 TB to 982 TB of usable storage capacity
See the table below for more details on the Lenovo EveryScale WEKA Storage Node configuration options.
To provide high-performance capability, WEKA uses DPDK, the high-performance network adapters options are based on WEKA support for the device and compatibility in Lenovo ThinkSystem servers. To provide the highest performance of WEKA, the HDR or 200GbE adapter should be selected in combination with an appropriate high-performance network infrastructure. WEKA currently uses at most 2 RDMA enabled adapters for WEKA traffic, however when an object backend is used, a third adapter can be used that is dedicated to traffic to the object storage system, similarly the third adapter may be used for multi-protocol clients accessing the WEKA storage. In addition to the high-performance networking options, a choice of LOM adapters is available for the operating system network, the selection of this adapter should be made based on the customer preference – both copper and SFP+ based LOM adapters are available.
The performance of WEKA is dependent on the performance of the NVMe drives installed within the system. When configuring a Lenovo EveryScale WEKA storage node, only high-performance NVMe drives are offered. Options for different levels of drive endurance based on customer workload and requirements – both 1DWPD and 3DWPD options are available.
The CPU configuration is based on the best balance of cores to drives in the systems along with high single thread performance to provide optimal metadata performance, with additional cores dedicated to multi-protocol access in the systems. When configuring the solution, it is essential that WEKA containers are distributed over the CPUs to take most advantage of the cores and architecture of the system.
Two memory configurations are qualified. For many clusters the default of 256GB should be sufficient. Where large numbers of small files or where a large capacity object tier is expected and highest metadata performance is also important the larger memory option can support that.
WEKA delivers a differentiated solution that goes beyond the current market and performance standards for storage. The partnership with Lenovo delivers the best solutions for your IT and business challenges.
Figure 12. WEKA node - ThinkSystem SR630 V2
For more information regarding the Lenovo ThinkSystem SR630 V2 in general, see the SR630 V2 product guide.
The following table lists the detailed configuration of the Lenovo EveryScale WEKA Storage Node.
Network platform
Performance of the WEKA storage system depends on the network infrastructure in place. Highest performance will be seen where HDR or 200GbE networking is used. 100GbE networking is possible, however performance will be limited by the network infrastructure. When designing a network for use with WEKA, it is important to ensure that sufficient bandwidth is provided both between WEKA servers and WEKA client systems. When network congestion is present on the network between clients and servers, performance of the WEKA storage will likely be impacted. Lenovo supplies and supports a number of high-performance network switches and recommends these are used when deploying a WEKA solution based on Lenovo ThinkSystem servers.
When installed into a rack, to reduce hot air reflow, the use of the air duct is strongly recommended. All switches are Port-Side Exhaust (PSE) and should be mounted with the ports at the rear of the rack. The Lenovo ThinkSystem servers are equipped with rear-mounted PCIe devices which means that back-of-rack cabling is possible to reduce the need for front-to-back network cabling.
InfiniBand-based networking
The InfiniBand switches listed are managed switches are ideal for small-scale or new InfiniBand networks with no external subnet management software required. Where the storage is integrated as part of an existing InfiniBand network or an external InfiniBand subnet management software is deployed, the choice of an unmanaged switch is sufficient.
Refer to the LeSI Product Guide, Table 14, where unmanaged or oPSE air flow are required. Table 6 provides details of suitable high-speed network cables.
Ethernet-based networking
Refer to the LeSI Product Guide, Table 6 where oPSE air flow is required. Tables 10-12 provide details of suitable high-speed network cables and optics.
WEKA software licenses
Topics in this section:
- License types
- License quantity
- Calculating the XPS Net Capacity
- Part numbers: WEKA Extreme Performance and Scale (XPS) Edition
- Part numbers: WEKA Data Tiering Option-DTO
- Part numbers: WEKA Data Protection Option-DPO
- Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)
- Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)
- Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)
License types
WEKA has tiered subscription-based licensing which is licensed per TB per year. Licenses for WEKA are not perpetual. Discounts are provided based on the tiering level and the number of advance years of purchase.
There are three types of licenses which can be applied to a WEKA cluster:
- WEKA Extreme Performance and Scale (XPS) Edition
XPS is the core foundation license required for a WEKA deployment which covers licensing of the flash tier embedded within the WEKA servers.
- WEKA Data Tiering Option-DTO
WEKA DTO is an additional add-on license on top of XPS. This license is necessary when data is tiered to an object backend behind the XPS layer.
- WEKA Data Protection Option-DPO
WEKA DPO is an additional add-on license on top of XPS, this covers data protection functionality of WEKA including snapshot to object, incremental snapshot sync and secure backup features. DPO also includes the tiering functions of DTO.
Where a system is currently licensed with XPS+DTO and features of DPO are required, contact your local Lenovo sales team to discuss trade-up options.
XPS licenses are calculated using the Net Capacity of the cluster (usable capacity) and varies depending on the number of servers, RAID encoding, hot spares, and capacity of drives, further details and example calculations are provided below. DTO and DPO licenses are calculated based on the usable capacity of the object store presented to be used by WEKA.
License quantity
To calculate the number of XPS licenses required:
- Determine the Net Capacity of the WEKA cluster in TB (see below)
- Use the Net Capacity of the cluster to determine the tiering level
- Determine how long the license is required for to select the correct part number
Discounted part numbers are available for qualified educational (EDU) customers. Check with your Lenovo sales organization to determine if you are eligible.
Monthly license options are provided to support co-terming when expanding the size of a cluster. For example, to add an additional node providing an additional 100TB of capacity, where the cluster has 2 years and 3 months of support remaining.
Calculating the XPS Net Capacity
The capacity required is the Net Capacity of the cluster. To calculate the XPS net capacity of a WEKA Storage cluster, use the following calculation:
Where the File System Overhead is a static value of 0.9. The calculation should then be rounded UP to the nearest whole number to give the SSD Net Capacity license requirement.
When deploying a WEKA Storage cluster using Lenovo ThinkSystem servers, the number of failure domains is typically the number of servers.
For example, when using 6 servers, each with 10x 3.2TB NVMe drives with 4+2P RAID code:
Net Capacity | = | Raw Capacity: 6 servers * 10 drives * 3.2TB | * | 6 servers (Failure Domains) – 1 (Hot Spares) | * | 4 (Stripe Width) | * | 0.9 (File System Overhead) |
6 servers (Failure Domains) | 4 (Stripe Width) + 2 (Protection) | |||||||
96TB | = | 192TB | * | 5 | * | 4 | * | 0.9 |
6 | 6 |
This is exactly 96TB Net Capacity.
Or when using 10 servers, each with 10x 7.68TB NVMe drives with 8+2P RAID code:
Net Capacity | = | Raw Capacity: 10 servers * 10 drives * 7.68TB | * | 10 servers (Failure Domains) – 1 (Hot Spares) | * | 8 (Stripe Width) | * | 0.9 (File System Overhead) |
10 servers (Failure Domains) | 8 (Stripe Width) + 2 (Protection) | |||||||
497.6TB | = | 768TB | * | 9 | * | 8 | * | 0.9 |
10 | 10 |
Round UP to the next integer value – 498TB to calculate license Net Capacity requirement.
For more information on calculating Net Capacity, refer to https://docs.weka.io/overview/ssd-capacity-management
Part numbers: WEKA Extreme Performance and Scale (XPS) Edition
Part numbers: WEKA Data Tiering Option-DTO
Part numbers: WEKA Data Protection Option-DPO
Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)
Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)
Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)
WEKA Installation & Deployment
The Lenovo EveryScale WEKA Storage Solution can be purchased as part of a Lenovo EveryScale cluster which includes options for factory integrated into racks. Systems can also be purchased and delivered for integration into a customer rack. The system can also be purchased as a standalone Lenovo EveryScale WEKA Storage Solution to complement an existing HPC cluster.
When deploying a WEKA storage system for the first time, planning of the hardware and software configuration is needed to ensure maximum performance can be obtained from the system. It is strongly recommended to take installation services to ensure successful integration of the system.
Lenovo professional services can provide full onsite installation services for hardware and software of the Lenovo EveryScale WEKA storage nodes, this includes installation of racks or equipment into customer racks, cabling, operating system deployment and installation of Weka software, this can also extend to a fully managed service. In addition to this, WEKA also provide remote installation services for the Weka software which is available as an option.
To determine the level of deployment services required, contact your local Lenovo sales team to discuss the scope of work required.
Installation Service
Operating System Software
Lenovo recommends the use of Red Hat Enterprise Linux (RHEL) as the operating system to use on WEKA storage servers. When coupled with a premium subscription, the software includes Long Term Support (LTS) updates to the operating system helping to ensure security updates are available for the operating system without the need for regular minor version upgrades. The following table outlines the options for Lenovo ThinkSystem servers. One subscription is needed per storage server.
Why Lenovo
Lenovo is a leading provider of data center infrastructure solutions and a premier WEKA partner. The Lenovo EveryScale WEKA solution is co-designed with WEKA and Lenovo will work with you and WEKA to identify, design, install and support the solution that best ensures your organization's needs.
Lenovo complements a portfolio of leading x86 infrastructure with a full range of storage, software, and comprehensive services that provides excellent performance, reliability, and security for your IT environment from the edge to the cloud.
Integrated Software & Hardware
Lenovo has worked with WEKA to provide a performance engineered solution aimed at extracting maximum performance out of Lenovo ThinkSystem servers which extends to integrating WEKA into an HPC scheduled system.
Lenovo EveryScale systems provide component level validation of the hardware and when shipped as part of a Lenovo EveryScale cluster simplify deployment and help drive reduction in time to results.
Optimized to deliver storage performance for the most demanding IO workloads, Lenovo EveryScale WEKA Storage provides an end-to-end collaboration that transforms access to data providing cost-effective scalability and performance.
Enterprise Level Support
Lenovo EveryScale WEKA Storage Solutions are based on Lenovo Certified Systems, tested, and validated to provide the necessary performance, manageability, security, and scalability. Comprehensive enterprise-grade support for Lenovo EveryScale WEKA solutions will be provided by Lenovo and WEKA software support will be provided directly by WekaIO.
For details about Lenovo support, see the Lenovo ISG Support Plan.
Seller training courses
The following sales training courses are offered for employees and partners (login required). Courses are listed in date order.
-
Partner Technical Webinar - Deployment Ready Solutions
2024-02-13 | 60 minutes | Employees and Partners
DetailsPartner Technical Webinar - Deployment Ready Solutionsin this 60-minute replay, Dave Brown and Joe Murphy reviewed the Deployment Ready Solutions feature of DCSC (https://dcsc.lenovo.com/#/) . Joe showed the DE Fibre Ready Solutions under Storage. Under the Deployment Ready Solutions, Joe showed WEKA Solutions and Dave showed the VEEAM Solutions.
Published: 2024-02-13
Length: 60 minutes
Employee link: Grow@Lenovo
Partner link: Lenovo Partner Learning
Course code: 020224 -
ThinkSystem High Performance File Offering with WEKA
2023-10-25 | 25 minutes | Employees and Partners
DetailsThinkSystem High Performance File Offering with WEKAThis course introduces you the file system solution with WEKA. By the end of this course, you will be able to: describe what is WEKA, identify the high-performance workloads where WEKA can improve the performance, review the file storage, identify the customer’s concerns, and describe the process to sell & win.
Published: 2023-10-25
Length: 25 minutes
Employee link: Grow@Lenovo
Partner link: Lenovo Partner Learning
Course code: DSOLO203 -
Benefits and the positioning of Lenovo ThinkSystem file offerings with WEKA, DSS, and DM+ONTAP
2023-05-08 | 10 minutes | Employees and Partners
DetailsBenefits and the positioning of Lenovo ThinkSystem file offerings with WEKA, DSS, and DM+ONTAPCourse objectives:
Published: 2023-05-08
1. Learn about file storage
2. Understand the different types of storage
3. Recognize the Lenovo data management portfolio
Length: 10 minutes
Employee link: Grow@Lenovo
Partner link: Lenovo Partner Learning
Course code: DSOLS207 -
Partner Technical Webinar - WEKA Data Platform
2023-03-28 | 60 minutes | Employees and Partners
DetailsPartner Technical Webinar - WEKA Data PlatformIn this 60-minute replay, Ed Stickney, WEKA Director of Strategic Partnerships and Chak Singamsetti, WEKA Senior Sales Engineered introduced WEKA and the WEKA Data Platform. They presented who is WEKA, the key use cases, the architecture, the competitors and what WEKA looks like in DCSC.
Published: 2023-03-28
Length: 60 minutes
Employee link: Grow@Lenovo
Partner link: Lenovo Partner Learning
Course code: 032423 -
ThinkSystem High Performance File offering with WEKA Preview
2023-02-07 | 3 minutes | Employees and Partners
DetailsThinkSystem High Performance File offering with WEKA PreviewThis preview introduces the ThinkSystem High Performance File offering with WEKA. Lenovo and WEKA are announcing a strategic partnership to deliver enterprise class solutions to accelerate HPC and AI workloads. Lenovo ThinkSystem Software Defined Storage (SDS) Ready Nodes with WEKA Data Platform software deliver a high performance solution that supports every step of the organizations’ data lifecycle – from ingest and pre-processing, to analyzing, storing, and archiving.
Published: 2023-02-07
Note: This course is presented as audio only. There are no slides or video.
Length: 3 minutes
Employee link: Grow@Lenovo
Partner link: Lenovo Partner Learning
Course code: SXXW2515a -
Technical Champions Webinar: WEKA - Delivering the Data Platform for the Modern Workload
2023-01-26 | 93 minutes | Employees Only
DetailsTechnical Champions Webinar: WEKA - Delivering the Data Platform for the Modern WorkloadHave you heard about WEKA lately and are wondering what exactly “WEKA” is? Maybe your customer’s data needs (both performance and capacity) are spiraling out of control, as is the complexity of managing multiple data platforms?
Published: 2023-01-26
Join Ed Stickney and Chak Singamsetti as they clear up the confusion and provide the training and tools you’ll need to become a WEKA-selling powerhouse. Ed will cover an overview of WEKA, sales positioning, licensing, and more. Chak will take over in the second half for a deeper dive into WEKA’s technological differentiation and how to build Lenovo-WEKA solutions.
In addition to being a value differentiator and a new market space for Lenovo, this is an EXCITING offering that will provide you with a new and different conversation with your customers. Please join us for what will certainly be an informative and profitable enablement investment in your future.
Length: 93 minutes
Employee link: Grow@Lenovo
Course code: DSOLS206
For More Information
To learn more about the Lenovo EveryScale WEKA Storage Solution, contact your Lenovo Sales Representative or Business Partner.
Products related to this document are the following:
- Lenovo Scalable Infrastructure (LeSI) Product Guide
- Lenovo ThinkSystem SR630 V2 Product Guide
- Lenovo Distributed Storage Solution for IBM Spectrum Scale (DSS-G) Product Guide
- Lenovo High Performance Computing home page
- Lenovo TruScale Infrastructure Services home page
More information on WEKA and related product families
Trademarks
Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.
The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
ThinkSystem®
The following terms are trademarks of other companies:
Intel® and Xeon® are trademarks of Intel Corporation or its subsidiaries.
Linux® is the trademark of Linus Torvalds in the U.S. and other countries.
Other company, product, or service names may be trademarks or service marks of others.
Configure and Buy
Full Change History
Changes in the March 10, 2023 update:
- Added tables of part numbers - WEKA software licenses section
First published: February 9, 2023
Course Detail
Employees Only Content
The content in this document with a is only visible to employees who are logged in. Logon using your Lenovo ITcode and password via Lenovo single-signon (SSO).
The owner of the document has determined that this content is classified as Lenovo Internal and should not be normally be made available to people who are not employees or contractors. This includes partners, customers, and competitors. The reasons may vary and you should reach out to the authors of the document for clarification, if needed. Be cautious about sharing this content with others as it may contain sensitive information.
Any visitor to the Lenovo Press web site who is not logged on will not be able to see this employee-only content. This content is excluded from search engine indexes and will not appear in any search results.
For all users, including logged-in employees, this employee-only content does not appear in the PDF version of this document.
This functionality is cookie based. The web site will normally remember your login state between browser sessions, however, if you clear cookies at the end of a session or work in an Incognito/Private browser window, then you will need to log in each time.
If you have any questions about this feature of the Lenovo Press web, please email David Watts at dwatts@lenovo.com.