Lenovo High Performance File System Solution with WEKA Data Platform

Big data has created new opportunities for innovation but also unforeseen problems for data-intensive applications, particularly in artificial intelligence (AI), machine learning (ML), deep learning (DL), financial analytics, genomics, life sciences, and high-performance computing (HPC). These compute and data-intensive applications demand maximum performance even while their datasets scale to extreme capacities. Legacy storage and data management solutions were designed to solve yesterday’s problems and cannot take on the storage challenges that organizations running leading-edge applications are facing now or will in the future. These demanding new workloads can place significant stress on legacy solutions, which were not originally designed to handle these new challenges.

The Lenovo High Performance File System Storage Solution with the WEKA® Data Platform, built on Lenovo ThinkSystem SR630 V2, SR635, SR650 V2, and SR655 servers, is a modern data infrastructure solution that is uniquely built to solve big problems that previously had no solution. It eliminates the complexity and compromises associated with legacy data storage and offers unmatched performance at ANY scale while still providing all the necessary enterprise features and benefits at a fraction of the cost. By meeting the stringent storage demands of key industry workloads, the Lenovo and WEKA solution accelerates the process of obtaining insight from mountains of data and enhances the promise of discovering breakthrough innovation and helping solve humanity’s greatest challenges.

Figure 1. WEKA Unified Data Platform

The workloads that address these challenges require a modern infrastructure with extreme performance in compute, storage, and networking resources. The engineered solution from Lenovo and WEKA addresses these needs and helps you gain the insights needed to gain a competitive advantage, deliver customer value, and grow your business.

The Lenovo ThinkSystem server platforms and the WEKA Data Platform work together to maximize acceleration, reduce training times, and deliver unmatched performance and reliability at scale. This solution is ideal for all of today’s data-intensive applications. This storage solution has proven scalable performance, delivering over 10x more performance than all-flash scale-out NAS and 3x more than locally attached NVMe SSDs. Together, Lenovo and WEKA support high-performance workloads in AI, finance, genomics, and technical computing. Because these workloads consist of billions of files, small and large, the storage system must deliver high throughput, low latency, and excellent metadata performance at extreme scale to keep these high-performance applications saturated with data. This is unattainable by legacy storage systems, but it is made possible with WEKA’s modern NVMe‐optimized, distributed, and parallel file system running on reliable and performance optimized Lenovo servers.

This software defined storage solution leverages WEKA software to support a hybrid cloud model, allowing enterprises to leverage on-demand public compute resources for cloud-bursting, remote backup, and disaster recovery.

Figure 2. WEKA Data Platform for AI/ML and HPC

With Lenovo ThinkSystem servers and WEKA Data Platform software, you can take advantage of a plug‐and‐play solution that helps extract greater value from data. The solution allows you to start small and flexibly scale to meet your performance and capacity requirements.

The Lenovo ThinkSystem servers offer uncompromising performance, scalability, reliability, and density in your choice of a 1U or 2U form factors. At the core of the WEKA Data Platform is WekaFS, a POSIX-compliant file system that distributes both data and metadata evenly across the entire storage cluster to ensure massively parallel access. This enables the WEKA platform to reduce time to innovation by delivering more data to the applications that need it, faster than any other available solution.  With a single namespace that offers on-premises storage and cloud connectivity, the software delivers simplified storage management and data protection. ThinkSystem adapters provide Ethernet and InfiniBand connectivity to deliver up to 200 Gb/sec bandwidth in a single network port, the lowest available latency, and application hardware offloads, satisfying even the most demanding application requirements.

This high-performance storage solution not only addresses your specific needs based on workload types and performance and capacity requirements, but goes beyond that to provide greater overall benefits for your business:

• Optimization of your IT environments with a modern storage infrastructure and simplified storage management
• Datacenter agility with faster data access and effective resource utilization
• Data transformation for AI, analytics, and technical computing with faster time to value and insight

Lenovo recommends the following minimum entry-level building block for your initial storage deployment, giving you the ability to scale out to greater capacities and performance levels by adding more servers.

6x Lenovo ThinkSystem SR630 V2 or SR635 servers, each configured with the following:

• 256GB or 512GB of memory
• 2x Mellanox ConnectX-6 network adapters
• 4-10x PCIe 4.0 NVMe SSD drives

Lenovo and WEKA deliver a differentiated solution that goes beyond the current market and performance standards for storage and deliver the best solutions for your IT and business challenges.

The following table lists the minimum recommended configuration for the Lenovo HPFS Solution.

Notes:

1. 4+2 stripe size with one configured hot spare. Configurations may vary based upon tiering customer network configuration.  Please speak with your Lenovo representative for solution guidance appropriate to your environment.
2. N+4 is available only for a cluster with 14 or more servers
3. Performance estimates are for a 6-server cluster on a 200Gb network

Configuration details are subject to change without notice. Additional capacity, drive, and server configurations are available. Please call your Lenovo representative for more information.

The following table lists the available Ready Nodes.

Topics in this section:

• License types
• License quantity
• Calculating the XPS Net Capacity
• Part numbers: WEKA Extreme Performance and Scale (XPS) Edition
• Part numbers: WEKA Data Tiering Option-DTO
• Part numbers: WEKA Data Protection Option-DPO
• Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)
• Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)
• Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)

License types

WEKA has tiered subscription-based licensing which is licensed per TB per year. Licenses for WEKA are not perpetual. Discounts are provided based on the tiering level and the number of advance years of purchase.

There are three types of licenses which can be applied to a WEKA cluster:

• WEKA Extreme Performance and Scale (XPS) Edition

  XPS is the core foundation license required for a WEKA deployment which covers licensing of the flash tier embedded within the WEKA servers.

• WEKA Data Tiering Option-DTO

  WEKA DTO is an additional add-on license on top of XPS. This license is necessary when data is tiered to an object backend behind the XPS layer.

• WEKA Data Protection Option-DPO

  WEKA DPO is an additional add-on license on top of XPS, this covers data protection functionality of WEKA including snapshot to object, incremental snapshot sync and secure backup features. DPO also includes the tiering functions of DTO.

Where a system is currently licensed with XPS+DTO and features of DPO are required, contact your local Lenovo sales team to discuss trade-up options.

XPS licenses are calculated using the Net Capacity of the cluster (usable capacity) and varies depending on the number of servers, RAID encoding, hot spares, and capacity of drives, further details and example calculations are provided below. DTO and DPO licenses are calculated based on the usable capacity of the object store presented to be used by WEKA.

License quantity

To calculate the number of XPS licenses required:

1. Determine the Net Capacity of the WEKA cluster in TB (see below)
2. Use the Net Capacity of the cluster to determine the tiering level
3. Determine how long the license is required for to select the correct part number

Discounted part numbers are available for qualified educational (EDU) customers. Check with your Lenovo sales organization to determine if you are eligible.

Monthly license options are provided to support co-terming when expanding the size of a cluster. For example, to add an additional node providing an additional 100TB of capacity, where the cluster has 2 years and 3 months of support remaining.

Calculating the XPS Net Capacity

The capacity required is the Net Capacity of the cluster. To calculate the XPS net capacity of a WEKA Storage cluster, use the following calculation:

Where the File System Overhead is a static value of 0.9. The calculation should then be rounded UP to the nearest whole number to give the SSD Net Capacity license requirement.

When deploying a WEKA Storage cluster using Lenovo ThinkSystem servers, the number of failure domains is typically the number of servers.

For example, when using 6 servers, each with 10x 3.2TB NVMe drives with 4+2P RAID code:

This is exactly 96TB Net Capacity.

Or when using 10 servers, each with 10x 7.68TB NVMe drives with 8+2P RAID code:

Round UP to the next integer value – 498TB to calculate license Net Capacity requirement.

For more information on calculating Net Capacity, refer to https://docs.weka.io/overview/ssd-capacity-management

Part numbers: WEKA Extreme Performance and Scale (XPS) Edition

Part numbers: WEKA Data Tiering Option-DTO

Part numbers: WEKA Data Protection Option-DPO

Part numbers: WEKA Extreme Performance and Scale (XPS) Edition - State, Local, Education (EDU)

Part numbers: WEKA Data Tiering Option-DTO - State, Local, Education (EDU)

Part numbers: WEKA Data Protection Option-DPO - State, Local, Education (EDU)

The following table lists the available installation/professional services.

Lenovo recommends the use of Red Hat Enterprise Linux (RHEL) as the operating system to use on WEKA storage servers. When coupled with a premium subscription, the software includes Long Term Support (LTS) updates to the operating system helping to ensure security updates are available for the operating system without the need for regular minor version upgrades. The following table outlines the options for Lenovo ThinkSystem servers. One subscription is needed per storage server.

To learn more about Lenovo Data Management solutions, please visit our Data Management solutions web page. More details on the WEKA Data Platform software can be found at weka.io. You can also contact your Lenovo sales representative or authorized channel partner.