Lenovo High Performance File System Solution with WEKA Data Platform

Big data has created new opportunities for innovation but also unforeseen problems for data-intensive applications, particularly in artificial intelligence (AI), machine learning (ML), deep learning (DL), financial analytics, genomics, life sciences, and high-performance computing (HPC). These compute and data-intensive applications demand maximum performance even while their datasets scale to extreme capacities. Legacy storage and data management solutions were designed to solve yesterday’s problems and cannot take on the storage challenges that organizations running leading-edge applications are facing now or will in the future. These demanding new workloads can place significant stress on legacy solutions, which were not originally designed to handle these new challenges.

The Lenovo High Performance File System Storage Solution with the WEKA® Data Platform, built on Lenovo ThinkSystem SR630 V2, SR635, SR650 V2, and SR655 servers, is a modern data infrastructure solution that is uniquely built to solve big problems that previously had no solution. It eliminates the complexity and compromises associated with legacy data storage and offers unmatched performance at ANY scale while still providing all the necessary enterprise features and benefits at a fraction of the cost. By meeting the stringent storage demands of key industry workloads, the Lenovo and WEKA solution accelerates the process of obtaining insight from mountains of data and enhances the promise of discovering breakthrough innovation and helping solve humanity’s greatest challenges.

Figure 1. WEKA Unified Data Platform

The workloads that address these challenges require a modern infrastructure with extreme performance in compute, storage, and networking resources. The engineered solution from Lenovo and WEKA addresses these needs and helps you gain the insights needed to gain a competitive advantage, deliver customer value, and grow your business.

The Lenovo ThinkSystem server platforms and the WEKA Data Platform work together to maximize acceleration, reduce training times, and deliver unmatched performance and reliability at scale. This solution is ideal for all of today’s data-intensive applications. This storage solution has proven scalable performance, delivering over 10x more performance than all-flash scale-out NAS and 3x more than locally attached NVMe SSDs. Together, Lenovo and WEKA support high-performance workloads in AI, finance, genomics, and technical computing. Because these workloads consist of billions of files, small and large, the storage system must deliver high throughput, low latency, and excellent metadata performance at extreme scale to keep these high-performance applications saturated with data. This is unattainable by legacy storage systems, but it is made possible with WEKA’s modern NVMe‐optimized, distributed, and parallel file system running on reliable and performance optimized Lenovo servers.

This software defined storage solution leverages WEKA software to support a hybrid cloud model, allowing enterprises to leverage on-demand public compute resources for cloud-bursting, remote backup, and disaster recovery.

Figure 2. WEKA Data Platform for AI/ML and HPC

With Lenovo ThinkSystem servers and WEKA Data Platform software, you can take advantage of a plug‐and‐play solution that helps extract greater value from data. The solution allows you to start small and flexibly scale to meet your performance and capacity requirements.

The Lenovo ThinkSystem servers offer uncompromising performance, scalability, reliability, and density in your choice of a 1U or 2U form factors. At the core of the WEKA Data Platform is WekaFS, a POSIX-compliant file system that distributes both data and metadata evenly across the entire storage cluster to ensure massively parallel access. This enables the WEKA platform to reduce time to innovation by delivering more data to the applications that need it, faster than any other available solution.  With a single namespace that offers on-premises storage and cloud connectivity, the software delivers simplified storage management and data protection. ThinkSystem adapters provide Ethernet and InfiniBand connectivity to deliver up to 200 Gb/sec bandwidth in a single network port, the lowest available latency, and application hardware offloads, satisfying even the most demanding application requirements.

This high-performance storage solution not only addresses your specific needs based on workload types and performance and capacity requirements, but goes beyond that to provide greater overall benefits for your business:

• Optimization of your IT environments with a modern storage infrastructure and simplified storage management
• Datacenter agility with faster data access and effective resource utilization
• Data transformation for AI, analytics, and technical computing with faster time to value and insight

Lenovo recommends the following minimum entry-level building block for your initial storage deployment, giving you the ability to scale out to greater capacities and performance levels by adding more servers.

6x Lenovo ThinkSystem SR630 V2 or SR635 servers, each configured with the following:

• 256GB or 512GB of memory
• 2x Mellanox ConnectX-6 network adapters
• 4-10x PCIe 4.0 NVMe SSD drives

Lenovo and WEKA deliver a differentiated solution that goes beyond the current market and performance standards for storage and deliver the best solutions for your IT and business challenges.

The following table lists the minimum recommended configuration for the Lenovo HPFS Solution.

Notes:

1. 4+2 stripe size with one configured hot spare. Configurations may vary based upon tiering customer network configuration.  Please speak with your Lenovo representative for solution guidance appropriate to your environment.
2. N+4 is available only for a cluster with 14 or more servers
3. Performance estimates are for a 6-server cluster on a 200Gb network

Configuration details are subject to change without notice. Additional capacity, drive, and server configurations are available. Please call your Lenovo representative for more information.

The following table lists the available Ready Nodes.

Topics in this section:

• License types
• License quantity
• Calculating the XPS Net Capacity
• Part numbers: WEKA Extreme Performance and Scale (XPS) Edition
• Part numbers: WEKA Data Tiering Option-DTO
• Part numbers: WEKA Data Protection Option-DPO
• Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)
• Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)
• Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)

License types

WEKA has tiered subscription-based licensing which is licensed per TB per year. Licenses for WEKA are not perpetual. Discounts are provided based on the tiering level and the number of advance years of purchase.

There are three types of licenses which can be applied to a WEKA cluster:

• WEKA Extreme Performance and Scale (XPS) Edition

  XPS is the core foundation license required for a WEKA deployment which covers licensing of the flash tier embedded within the WEKA servers.

• WEKA Data Tiering Option-DTO

  WEKA DTO is an additional add-on license on top of XPS. This license is necessary when data is tiered to an object backend behind the XPS layer.

• WEKA Data Protection Option-DPO

  WEKA DPO is an additional add-on license on top of XPS, this covers data protection functionality of WEKA including snapshot to object, incremental snapshot sync and secure backup features. DPO also includes the tiering functions of DTO.

Where a system is currently licensed with XPS+DTO and features of DPO are required, contact your local Lenovo sales team to discuss trade-up options.

XPS licenses are calculated using the Net Capacity of the cluster (usable capacity) and varies depending on the number of servers, RAID encoding, hot spares, and capacity of drives, further details and example calculations are provided below. DTO and DPO licenses are calculated based on the usable capacity of the object store presented to be used by WEKA.

License quantity

To calculate the number of XPS licenses required:

1. Determine the Net Capacity of the WEKA cluster in TB (see below)
2. Use the Net Capacity of the cluster to determine the tiering level
3. Determine how long the license is required for to select the correct part number

Discounted part numbers are available for qualified educational (EDU) customers. Check with your Lenovo sales organization to determine if you are eligible.

Monthly license options are provided to support co-terming when expanding the size of a cluster. For example, to add an additional node providing an additional 100TB of capacity, where the cluster has 2 years and 3 months of support remaining.

Calculating the XPS Net Capacity

The capacity required is the Net Capacity of the cluster. To calculate the XPS net capacity of a WEKA Storage cluster, use the following calculation:

Where the File System Overhead is a static value of 0.9. The calculation should then be rounded UP to the nearest whole number to give the SSD Net Capacity license requirement.

When deploying a WEKA Storage cluster using Lenovo ThinkSystem servers, the number of failure domains is typically the number of servers.

For example, when using 6 servers, each with 10x 3.2TB NVMe drives with 4+2P RAID code:

This is exactly 96TB Net Capacity.

Or when using 10 servers, each with 10x 7.68TB NVMe drives with 8+2P RAID code:

Round UP to the next integer value – 498TB to calculate license Net Capacity requirement.

For more information on calculating Net Capacity, refer to https://docs.weka.io/overview/ssd-capacity-management

Part numbers: WEKA Extreme Performance and Scale (XPS) Edition

Part numbers: WEKA Data Tiering Option-DTO

Part numbers: WEKA Data Protection Option-DPO

Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)

Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)

Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)

The following table lists the available installation/professional services.

Lenovo recommends the use of Red Hat Enterprise Linux (RHEL) as the operating system to use on WEKA storage servers. When coupled with a premium subscription, the software includes Long Term Support (LTS) updates to the operating system helping to ensure security updates are available for the operating system without the need for regular minor version upgrades. The following table outlines the options for Lenovo ThinkSystem servers. One subscription is needed per storage server.

The following sales training courses are offered for employees and partners (login required). Courses are listed in date order.

1. Partner Technical Webinar - Deployment Ready Solutions
   2024-02-13 | 60 minutes | Employees and Partners
   Details
   Partner Technical Webinar - Deployment Ready Solutions
   
   

   in this 60-minute replay, Dave Brown and Joe Murphy reviewed the Deployment Ready Solutions feature of DCSC (https://dcsc.lenovo.com/#/) . Joe showed the DE Fibre Ready Solutions under Storage. Under the Deployment Ready Solutions, Joe showed WEKA Solutions and Dave showed the VEEAM Solutions.

   Published: 2024-02-13
   Length: 60 minutes
   Employee link: Grow@Lenovo
   Partner link: Lenovo Partner Learning
   Course code: 020224
2. ThinkSystem High Performance File Offering with WEKA
   2023-10-25 | 25 minutes | Employees and Partners
   Details
   ThinkSystem High Performance File Offering with WEKA
   
   

   This course introduces you the file system solution with WEKA. By the end of this course, you will be able to: describe what is WEKA, identify the high-performance workloads where WEKA can improve the performance, review the file storage, identify the customer’s concerns, and describe the process to sell & win.

   Published: 2023-10-25
   Length: 25 minutes
   Employee link: Grow@Lenovo
   Partner link: Lenovo Partner Learning
   Course code: DSOLO203
3. Benefits and the positioning of Lenovo ThinkSystem file offerings with WEKA, DSS, and DM+ONTAP
   2023-05-08 | 10 minutes | Employees and Partners
   Details
   Benefits and the positioning of Lenovo ThinkSystem file offerings with WEKA, DSS, and DM+ONTAP
   
   

   Course objectives:
   1. Learn about file storage
   2. Understand the different types of storage
   3. Recognize the Lenovo data management portfolio

   Published: 2023-05-08
   Length: 10 minutes
   Employee link: Grow@Lenovo
   Partner link: Lenovo Partner Learning
   Course code: DSOLS207
4. Partner Technical Webinar - WEKA Data Platform
   2023-03-28 | 60 minutes | Employees and Partners
   Details
   Partner Technical Webinar - WEKA Data Platform
   
   

   In this 60-minute replay, Ed Stickney, WEKA Director of Strategic Partnerships and Chak Singamsetti, WEKA Senior Sales Engineered introduced WEKA and the WEKA Data Platform. They presented who is WEKA, the key use cases, the architecture, the competitors and what WEKA looks like in DCSC.

   Published: 2023-03-28
   Length: 60 minutes
   Employee link: Grow@Lenovo
   Partner link: Lenovo Partner Learning
   Course code: 032423
5. ThinkSystem High Performance File offering with WEKA Preview
   2023-02-07 | 3 minutes | Employees and Partners
   Details
   ThinkSystem High Performance File offering with WEKA Preview
   
   

   This preview introduces the ThinkSystem High Performance File offering with WEKA. Lenovo and WEKA are announcing a strategic partnership to deliver enterprise class solutions to accelerate HPC and AI workloads. Lenovo ThinkSystem Software Defined Storage (SDS) Ready Nodes with WEKA Data Platform software deliver a high performance solution that supports every step of the organizations’ data lifecycle – from ingest and pre-processing, to analyzing, storing, and archiving.
   Note: This course is presented as audio only. There are no slides or video.

   Published: 2023-02-07
   Length: 3 minutes
   Employee link: Grow@Lenovo
   Partner link: Lenovo Partner Learning
   Course code: SXXW2515a
6. Technical Champions Webinar: WEKA - Delivering the Data Platform for the Modern Workload
   2023-01-26 | 93 minutes | Employees Only
   Details
   Technical Champions Webinar: WEKA - Delivering the Data Platform for the Modern Workload
   
   

   Have you heard about WEKA lately and are wondering what exactly “WEKA” is? Maybe your customer’s data needs (both performance and capacity) are spiraling out of control, as is the complexity of managing multiple data platforms?
   
   Join Ed Stickney and Chak Singamsetti as they clear up the confusion and provide the training and tools you’ll need to become a WEKA-selling powerhouse. Ed will cover an overview of WEKA, sales positioning, licensing, and more. Chak will take over in the second half for a deeper dive into WEKA’s technological differentiation and how to build Lenovo-WEKA solutions.
   In addition to being a value differentiator and a new market space for Lenovo, this is an EXCITING offering that will provide you with a new and different conversation with your customers. Please join us for what will certainly be an informative and profitable enablement investment in your future.

   Published: 2023-01-26
   Length: 93 minutes
   Employee link: Grow@Lenovo
   Course code: DSOLS206

To learn more about Lenovo Data Management solutions, please visit our Data Management solutions web page. More details on the WEKA Data Platform software can be found at weka.io. You can also contact your Lenovo sales representative or authorized channel partner.