Intel Optane Persistent Memory 100 Series

Intel Optane DC Persistent Memory represents a new class of memory and storage technology explicitly architected for data center usage. It offers three main benefits:

• Significantly lower latency than fetching data from system storage
• High capacities
• Affordable cost

Using Lenovo ThinkSystem servers running applications that are tuned for Intel Optane DC Persistent Memory will result in lower data latency compared to solid-state drive technology. When data is stored closer to the processor on nonvolatile media, applications can see significant overall improvement in performance. 

An Intel Optane DC Persistent Memory Module (DCPMM) is shown in the following figure.

Figure 1. Intel Optane DC Persistent Memory Module (DCPMM)

Intel Optane DC Persistent Memory modules (DCPMMs) have the form factor of a DDR4 DIMM, but the persistence and capacity of data storage of a solid-state drive. This means the DCPMMs have performance characteristics similar to that of TruDDR4 DIMMs, the storage capacity of an SSD, and the ability to stay active after a power cycle or reboot of the server. These features open up a new way of performing data I/O to application developers and new levels of server performance to customers.

The following table lists the ordering information for the persistent memory options.

* Configure-to-order only, not available as a field upgrade. Planned for late 2Q/2019

Intel Optane DC Persistent Memory operates in one of three modes:

• Memory Mode

  In this mode, the DCPMMs act as large capacity DDR4 memory modules. In such a configuration, the memory that the operating system recognizes is the DCPMMs; the installed TruDDR4 DIMMs are hidden from the operating system and act as a caching layer for the DCPMMs. In this mode, the persistence feature of the DCPMMs is disabled. This mode does not require the application to be DCPMM-aware.

• App Direct Mode

  In this mode, the DCPMMs provide all persistence features to the operating system and applications that support them. The operating system presents both TruDDR4 DIMMs and DCPMMs to the applications, as system memory and persistent storage respectively.

  Depending on the configuration in UEFI and the operating system, the DCPMMs appear as one of two types of namespaces:

  • Direct access (DAX): byte-addressable storage accessible via an API. The applications must be DCPMM-aware and use the published APIs to implement the DCPMM features.
  • Block storage: the persistent memory is presented to applications is seen as a block storage device, similar to an SSD. The operating system needs to be DCPMM-aware, however the applications do not.

  Applications with planned support include:

  • SAP HANA
  • Aerospike Enterprise Edition
  • Gigaspaces
  • Apache Cassandra
  • Apache Spark SQL with OAP
  • Apache HBase Bucket Cache
  • Apache Hadoop HDFS Cache
• Mixed Memory Mode

  Mixed Memory Mode is a combination of Memory Mode and App Direct Mode, where a portion of the capacity of the DCPMMs is used for the Memory Mode operations, and the remaining capacity of the DCPMMs is used for the App Direct Mode operations. In this mode, all installed TruDDR4 DIMMs are hidden from the operating system and act as a caching layer for portion of the DCPMMs in Memory Mode.

In App Direct mode (and the persistent portion of Mixed mode), the persistent memory can be configured in one of two ways:

• Interleaved, where all DCPMMs are seen as one single monolithic space. This is similar in concept to RAID-0 in storage.
• Non-interleaved, where each DCPMM is seen as a separate space. This is similar in concept to JBOD in storage.

For configure-to-order (CTO) configurations, the feature codes listed in the following table specify the DCPMM mode you wish to enable.

The following figure shows a Lenovo TruDDR4 DIMM and an Intel Optane DC Persistent Memory Module.

Figure 2. Intel Optane DC Persistent Memory Module (top) and Lenovo TruDDR4 DIMM (bottom)

Intel Optane DC Persistent Memory provides benefits in the following application types:

• Larger memory footprint: For applications with performance characteristics that place greater emphasis on memory capacity over memory bandwidth or memory latency, the use of DCPMMs can mean a significant increase in overall system performance compared to the use of TruDDR4 DIMMs. 
  • Cloud and Infrastructure-as-a-service (IaaS) applications
  • More virtual machines and cloud containers per server
  • Larger memory allocation to each VM
• In-memory databases: With DCPMMs, database applications have store much larger databases in persistent memory rather than on disk, and database performance will be improved significantly. For existing applications that use system RAM for in-memory databases, the use of persistent memory will mean no delays at boot time having to copy the databases from disk into memory.
• Storage caching layers: TruDDR4 DIMMs can be used for the fastest memory access - best throughput and lowest latency, and DCPMMs can be used for the caching layer that offers memory-like performance with the persistence of SSD storage.
• NFV infrastructure: Network Function Virtualization (NFV) can make use of increased memory capacity and performance with the addition of DCPMMs.
  • High capacity non-volatile cache for enterprise and cloud storage
  • High capacity local cache for network storage App Direct direct-attach storage

Intel Optane DCPMMs have the following specifications:

• DCPMMs are installed in standard memory slots in supported servers
• 2666 MHz memory bus speed. Any 2933 MHz TruDDR4 DIMMs installed will also operate at 2666 MHz.
• Optional data encryption using AES 256-bit encryption
• Optional data security in App Direct mode, including secure erase functionality
• Firmware updates through XClarity Administrator and other Lenovo support tools

DCPMMs offer the following memory protection technologies:

• ECC
• SDDC
• DDDC
• Patrol scrubbing
• Demand scrubbing

The following are the requirements when selecting the number of DIMMs and DCPMMs:

• DCPMMs require second generation Intel Xeon Scalable Family processors. First generation Xeon Scalable processors are not supported.
• All Platinum processors, all Gold processors and the Silver 4215 processor support DCPMM.
• All installed DCPMMs must be the same size. Mixing DCPMMs of different capacities is not supported
• All installed DIMMs must be the same size and structure (ie same part number). Mixing different DIMMs is not supported
• The use of 1Rx8 DIMMs with DCPMMs is not supported. See the Memory DIMM support section for specifics.
• Maximum 6 DCPMMs per processor (install 1 in each memory channel)
• Minimum 2 TruDDR4 DIMMs per processor (1 per memory controller)
• For Memory Mode, minimum 2 DCPMMs per processor (install 1 per memory controller)
• For App Direct Mode, minimum 1 DCPMM installed in the server (any processor)
• When either Memory Mode or Mixed Mode is used, the ratio of memory to DCPMMs must be between 1:16 and 1:2, and the recommended ratio is 1:4 for the best performance. For example, 6x 16GB DIMMs + 2x 256GB DCPMMs is a ratio of 1:5.33. In Mixed Mode, the ratio is between memory and only the volatile portion of the DCPMMs. This ratio requirement does not apply to App Direct mode.
• For each memory channel with both a DCPMM and a TruDDR4 DIMM installed, the DCPMM is installed in channel slot 1 (closest) and the DIMM is installed in channel slot 0
• To maximize performance, balance all memory channels
• In configurations with DCPMMs installed, memory mirroring is supported, with two restrictions:
  • Mirroring is only enabled on the DRAM DIMMs installed in the server; The DCPMMs themselves do not support mirroring.
  • Only App Direct mode is supported. Memory mirroring cannot be enabled when DCPMMs are in Memory Mode or Mixed Mode.
• Memory sparing is not supported with DCPMMs installed

The following figure shows DCPMMs and TruDDR4 DIMMs installed in the system board of a ThinkSystem SR950. In this full configuration, there is one DCPMM and one TruDDR4 DIMM installed in each memory channel (6 DCPMMs and 6 DIMMs per processor).

Figure 3. Intel Optane DCPMMs installed in a ThinkSystem SR950 system board

The following table lists the supported combinations in App Direct mode.

In Memory Mode, the DCPMMs are seen by the operating system as system memory. The memory DIMMs are hidden from the operating system and are used as a high-speed cache for the DCPMMs.

Key to performance is the ratio of total DIMMs to total DCPMMs. The recommended range of DIMMs:DCPMMs is between 1:2 and 1:16:

• 1:16 means 1GB of DIMMs (used for cache) for every 16GB of DCPMM capacity (used as system memory). A larger number than 16 means potentially worse performance since the chances of a cache hit in memory will be lower.
• 1:2 means 1GB of DIMMs for every 2GB of DCPMM capacity. 1:2 maximizes the performance of the memory subsystem while still using persistent memory for increased capacity. Note: Ratios of between 1:2 and 1:4 require the latest firmware.

DCPMMs are only supported in quantities of 1, 2, 4 and 6 per processor, and only certain quantities of memory DIMMs are supported alongside the DCPMMs, as listed in the following table.

The specific memory part numbers that are supported with DCPMMs is listed in the Memory DIMM support section.

The following tables show the allowed percentage for each DCPMM part number and what the effective amount of App Direct persistent memory will be available to applications. Only a set number of percentages are available to choose from and the amount of App Direct persistent memory that is allocated will be in increments of 32 GB multiplied by the number of DCPMMs installed.

The following table shows the supported combinations of DCPMMs and DIMMs. The key requirement for support is ensuring that the ratio of the total memory DIMMs capacity to the total of the volatile (memory) portion of DCPMMs should be between 1:2 and 1:16. Ratios between 1:2 and 1:4 require the latest firmware.

The following table lists which TruDDR4 DIMMs can be installed with DCPMMs. 2933 MHz DIMMs will operate at 2666 MHz with installed with DCPMMs.

Notes:

• RDIMMs, LRDIMMs and 3DS RDIMMs cannot be mixed.
• Not all servers support all memory options. See the Lenovo ThinkSystem Memory Summary for details:
  https://lenovopress.com/lp1021-lenovo-thinksystem-memory-summary

Intel Optane DC Persistent Memory is supported by these processors:

• All second-generation Xeon Platinum (8200 series) processors
• All Second-generation Xeon Gold (6200 and 5200 series) processors
• The following second-generation Xeon Silver (4200 series) processor:
  • Intel Xeon Silver 4215 8C 85W 2.5GHz Processor
  • Intel Xeon Silver 4215R 8C 130W 3.2GHz Processor

Second-generation Xeon Scalable processors are limited to the amount of memory they can address, and this memory maximum also includes DCPMMs in the calculation:

• Processors with an L suffix (eg 8280L): 4.5 TB maximum per processor
• Processors with an M suffix (eg 8280M): 2 TB maximum per processor
• All other processors: 1 TB per processor

For example:

• A configuration using 12x 64GB DIMMs per processor is a total of 768 GB, which means that neither an M nor an L processor is required
• A configuration using 12x 256GB DIMMs per processor is a total of 3 TB, which means that an L processor is required
• A configuration using 6x 32GB DIMMs + 6x 256GB DCPMMs is a total of 1.69 TB which means an M processor is required (an L processor may also be used)
• A configuration using 6x 64GB DIMMs + 6x 512GB DCPMMs is a total of 3.375 TB which means an L processor is required
• A configuration using 6x 256GB DIMMs + 6x 512GB DCPMMs is a total of 4.5 TB which means an L processor is required

Intel Optane DC Persistent Memory is only supported in servers with second-generation Intel Xeon Scalable processors. The following table lists the ThinkSystem servers that are compatible.

The following tables list the ThinkSystem servers that are compatible.

Most ThinkSystem servers have 12 DIMM slots per processor and 2 DIMMs per channel across all channels, thereby supporting 1 DCPMM in every memory channel. However, some servers have fewer slots, and as a result, not all combinations of DIMMs and DCPMMs are supported.

When all 12 DIMMs are installed, this is referred to as a "2-2-2" configuration, where each 2 corresponds to the number of DIMMs per channel in 1 memory controller. When fewer DIMMs are installed, it is referred to a "2-2-1" or "2-1-1" or similar. The table shows with combinations are supported with each ThinkSystem server.

The following operating systems support the use of persistent memory modules:

• Microsoft Windows Server 2019
• Red Hat Enterprise Linux 7.6
• SUSE Linux Enterprise Server 12.4
• SUSE Linux Enterprise Server 15
• VMware vSphere Hypervisor (ESXi) 6.7 U1
• Ubuntu LTS 18.04

For details about VMware support, see these VMware KB articles:

• Lenovo DCPMM ESXi support
• VMware ESXi DCPMM support

The ThinkSystem Intel Optane DC Persistent Memory Modules carry a 1-year limited warranty. When installed in a supported ThinkSystem server, the DCPMMs assume the server’s base warranty and any warranty upgrades.

The following papers have been published on the use of Persistent Memory:

• Implementing Intel Optane DC Persistent Memory on Windows Server 2019
  https://lenovopress.com/LP1192
• Enabling Intel Optane DC Persistent Memory in a Linux Virtual Machine
  https://lenovopress.com/LP1224
• Implementing Intel Optane DC Persistent Memory with VMware vSphere
  https://lenovopress.com/LP1225
• Introducing the Programming Model of Intel Optane DC Persistent Memory
  https://lenovopress.com/LP1194
• Analyzing the Performance of Intel Optane DC Persistent Memory in App Direct Mode in Lenovo ThinkSystem Servers
  https://lenovopress.com/LP1083
• Analyzing the Performance of Intel Optane DC Persistent Memory in Memory Mode in Lenovo ThinkSystem Servers
  https://lenovopress.com/LP1084
• Analyzing the Performance of Intel Optane DC Persistent Memory in Storage over App Direct Mode
  https://lenovopress.com/LP1085

For more information, see these additional web resources:

• Lenovo web page for Persistent Memory
  https://www.lenovo.com/us/en/data-center/servers/server-options/thinksystem-options/Intel-Optane-DC-Persistent-Memory/p/WMD00000388
• Intel Optane DC Persistent Memory web page
  https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html
• Intel Optane Technology web page:
  https://www.intel.com/content/www/us/en/architecture-and-technology/intel-optane-technology.html
• Lenovo ThinkSystem Information Center (User Manuals):
  • DCPMM installation order
  • DCPMM setup
  • Adding memory modules with DCPMMs
  • Configuring DCPMMs