ThinkSystem SR950 Memory Decisions

The ThinkSystem SR950 supports a large amount of system memory. The four-socket (4S) configuration supports up to 48 memory slots up to 6TB of memory and the eight-socket (8S) configuration supports up to 96 memory slots up to 12TB of memory. With larger system memory configurations, memory can be the most expensive component of a servers cost as well as a key element driving the server performance.

Because of these factors, it very important to understand the memory choices and build the most optimal memory configuration for your SR950 server.
 

Figure 1. Lenovo ThinkSystem SR950

The Lenovo ThinkSystem SR950 uses Lenovo TruDDR4 memory operating at up to 2666 MHz. The server supports 12 DIMMs per processor, which corresponds to 48 DIMMs with four processors installed and 96 DIMMs when eight processors are installed. Each processor has six memory channels with two DIMMs per channel. With 128 GB 3DS RDIMMs installed, an 8-socket server supports a total of 12 TB of system memory.

All DIMMs operate at a speed of 2666 MHz, both at 1 DIMM per channel and 2 DIMMs per channel. However, if the processor selected has a lower memory bus speed (eg 2400 MHz), the DIMMs will operate at that lower speed. For the SR950, only the Gold 51xx Scalable Family CPUs operate at the lower 2400 MHz speeds

Figure 2. Compute system board with 2 processors and 24 total DIMMs

Lenovo TruDDR4 memory uses the highest quality components that are sourced from Tier 1 DRAM suppliers and only memory that meets the strict requirements of Lenovo is selected. It is compatibility tested and tuned to maximize performance and reliability. From a service and support standpoint, Lenovo TruDDR4 memory automatically assumes the system warranty, and Lenovo provides service and support worldwide.

Hitting your exact desired system memory level might limit your DIMM choices but in some cases you can choose between two and three different DIMMs to meet the requirements.

Many end users configure server memory based on a capacity requirement. Historically, RDIMMs have been favored for lower capacity requirements and LRDIMMs for higher capacity requirements.

Choosing between the two types of 16GB DIMMs

• Choose the 16GB 2R x8 for better performance
• Choose the 16GB 1R x4 for better RAS as its supports Chipkill (DRAM fail support and more self-healing features)

To hit a certain memory level and allow for memory growth, it might make sense to populate the SR950 with larger DIMM sizes instead of filling up the server with smaller DIMMs. This way you have empty DIMM slots allowing expansion. A 4S example would be using 24x 32GB DIMMs instead of using 48x 16GB DIMMs for a 768 GB total system memory.

The 128 GB DIMM is a 3DS RDIMM. 3DS (three dimensional stacking) is a relatively new technology which stacks DRAM components vertically and allows for greater capacity DIMMs while not increasing the load on the memory bus.

The cost of the different memory options should also be taken into consideration:

• The two 16GB DIMMs have essentially the same cost
• A single 32GB DIMM is typically lower cost than two 16GB DIMMs
• A single 64GB DIMM is currently only 1-5% higher cost than two 32GB DIMMs.

A single 128GB DIMM is currently quite a bit more expensive than two 64GB DIMMs but cost on 128GB DIMM will decrease as adoption of this new DIMM increases.

The following figure shows the 16 GB and 64 GB DIMMs.

Figure 3. SR950 memory options (16 GB RDIMM and 64 GB RDIMM)

Configuring a server with balanced memory is important for maximizing its memory bandwidth and overall performance. The SR950 has six memory channels per processor and up to two DIMMs per channel, so it is important to understand what is considered a balanced configuration and what is not.

Balanced memory configurations enable optimal interleaving across all attached memory channels so memory bandwidth is maximized. Bandwidth can be optimized if both memory channels on the same physical processor socket are identically configured. System level memory performance can be further optimized if each physical processor socket has the same physical memory capacity.

The basic guidelines for a balanced memory subsystem are as follows:

• Populate memory DIMMs in quantities of 6 or 12 per processor, so that all memory channels are used.
• When mixing 16 GB and 32 GB DIMMs in the same configuration, use 16GB 2Rx8 DIMMs instead of 16 GB 1Rx4 DIMMs for better performance.
• Populate memory channels so they all have the same total memory capacity
• Ensure all memory controllers on a processor socket have the same DIMM configuration.
• All processor sockets on the same physical server should have the same DIMM configuration.

If the SR950 requires a specific memory capacity, sometimes smaller DIMMs or a mix of DIMM sizes are needed to produce a balanced memory configuration.

Overall server performance is affected by the memory subsystem which can provide both high memory bandwidth and low memory access latency when properly configured. Balancing memory across the memory channels and CPUs produces memory configurations which can efficiently interleave memory references among its DIMMs producing the highest possible memory bandwidth.

For a detailed discussion of configurations with balanced memory, see the Lenovo Press paper, Intel Xeon Scalable Family Balanced Memory Configurations.

The best way to ensure a balanced memory configuration that meets your capacity and upgrade requirements, use the Memory Wizard as described in the Lenovo Memory Configurator section below.

Examples of Balanced Memory configurations for each processor:

Figure 4. 384 GB (4S) or 768 GB (8S)

Figure 5. 768 GB (4S) or 1.5 TB (8S)

Figure 6. 1 TB (4S) or 2 TB (8S)

Figure 7. 1.5 TB (4S) or 3 TB (8S)

Figure 8. 1.5 TB (4S) or 3 TB (8S)

Figure 9. 3 TB (4S) or 6 TB (8S)

Figure 10. 6 TB (4S) or 12 TB (8S)

If memory channel mirroring is used, then DIMMs must be installed in pairs or sets of three (minimum of one pair or set of three per processor), and all DIMMs in the pair or set of three must be identical in type and size. 50% of the installed capacity is available to the operating system.

If memory rank sparing is used, then a minimum of two single-rank or dual-rank DIMMs must be installed per populated channel (the DIMMs do not need to be identical). In rank sparing mode, one rank of a DIMM in each populated channel is reserved as spare memory. The largest rank in the channel will be automatically selected as the spare rank. The amount of memory available to the operating system depends on the number, capacity and rank counts of the DIMMs installed.

The XClarity Provisioning Manager application embedded in the Unified Extensible Firmware Interface (UEFI) of the SR950 can if requested, force memory to a speed lower than the maximum speed determined by the processor and the memory subsystem. Memory speed might be set lower to save energy in environments with little memory performance sensitivity

Memory speed options are as follows:

• Maximum Performance (default): Memory runs at the maximum supported speed determined by processor model and memory subsystem.
• Balanced: Memory runs at one step lower than the maximum speed.
• Minimal Power: Memory runs at the lowest speed supported by the architecture. 1866 MHz is the lowest speed supported on SR950.

Learning about the memory choices for the SR950 helps you optimize system memory for performance and cost. To make it easier, Lenovo has also developed a Lenovo Memory Configuration tool to optimize system memory for each ThinkSystem server.

The user can provide the amount of memory required and how they would like to optimize based on performance or by price. The memory configuration tool will then calculate the amount of DIMMs required as well as the location of the DIMMs based on the users input.

To use the tool to configure SR950 memory, do the following:

1. Launch Lenovo Memory Configuration tool
2. Select Mission-Critical > ThinkSystem SR950
3. Click Specify processor and select the type and number of CPUs (2, 4 or 8)
4. Click Next (you do not specify any existing memory on a new system)
5. Enter the desired system memory (ie 1535 GB). The tool will round up to the next available capacity based on available DIMMs.
6. Select the configuration priority (performance, price or custom).
7. Review the available configuration options. Each configuration shows which memory DIMMs are installed in every DIMM slot of the server.

The tool provides an export function to save the selected configuration to an XLS spreadsheet and an XML file for use in the LESC system configurator.

This article is one in a series on the ThinkSystem SR950 and SR850 servers:

• Five Highlights of the ThinkSystem SR950
• Five Highlights of the ThinkSystem SR850
• Choosing between Lenovo ThinkSystem SR850 and SR950
• Workloads for 4-Socket and 8-Socket Servers
• Usability in the Design of the ThinkSystem SR950
• The Value of Refreshing Your 4-Socket Servers with the ThinkSystem SR950
• ThinkSystem SR950 Memory Decisions
• ThinkSystem SR950 Server Configurations
• The Value of Refreshing Your 8-Socket Servers with the ThinkSystem SR950
• Lenovo ThinkSystem SR950 New Options and Features - December 2017
• ThinkSystem SR950 Performance Leadership
• Lenovo Servers for Mission Critical Workloads
• Microsoft and Lenovo ThinkSystem SR950 – A Perfect Match
• Accelerate Your 4- and 8-Socket Server Refresh Cycle
• SAP Business Process Applications and Lenovo ThinkSystem SR950 – A Perfect Match
• ThinkSystem SR950 New Options - March 2018
• SAP HANA and Lenovo ThinkSystem SR950 – A Perfect Match
• ThinkSystem SR950 Performance Leadership Continues
• New Solution for SAP HANA - Lenovo ThinkAgile HX
• The Advantages of Keeping Mission Critical Workloads On-Premises vs Going to the Cloud
• SQL Server Migration and Lenovo ThinkSystem SR950

Randall Lundin is a Senior Product Manager in the Lenovo Infrastructure Solution Group. He is responsible for planning and managing ThinkSystem servers. Randall has also authored and contributed to numerous Lenovo Press publications on ThinkSystem products.