ThinkSystem Hidden UEFI Parameters

The Lenovo server Unified Extensible Firmware Interface (UEFI) provides the interface between an operating system and the server firmware. Many UEFI parameters are exposed to server administrators in UEFI menus that can be accessed by pressing F1 when a server is rebooted, through the XClarity Controller (XCC) service processor, or through command line utilities such as Lenovo’s Advanced Settings Utility (ASU) or OneCLI.

These parameters are exposed because they are commonly changed from their default values to fine tune server performance for a wide variety of customer use cases. Other UEFI settings are more limited in their applicability to customer use cases and are not exposed in UEFI menus but can be accessed using the command line utilities.

In this paper, we describe the following commands that are only accessible via command-line utilities:

• Snoop Response Hold Off
• Dead Line LLC Allocation
• Stale/Directory AtoS

These commands are available for ThinkSystem servers that use the Intel Xeon Scalable Family processors.

This hidden parameter is accessible on Lenovo ThinkSystem servers that use the Intel Xeon Scalable Family processors, as well as System x servers with Intel Xeon E5 v3 or E5 v4 processors.

For some workloads in which throughput and latency are critical, it is better to constantly poll the status of an I/O device rather than use an interrupt. Network adapter device drivers commonly use a thread to continuously poll in a fast loop so that incoming requests can be handled as fast as possible.

This can create contention between a processor core running the polling thread and the processor’s Integrated I/O feature (IIO) for an I/O-owned line in cache. This contention can cause an I/O operation to lose ownership of the cache line it has just acquired. It must then spend more time reacquiring the cache line to write it back.

When there are a large number of network ports each servicing small packets, the system may not be able to achieve the full throughput required due to excessive I/O and core contentions of cache lines. For this situation, the I/O operation should delay its response to core snoops and hold onto its cache lines until it successfully completes its write.

The Snoop Response Hold Off parameter allows the I/O operation to delay its snoop response by a selected amount to achieve this delay. It is possible to adjust this parameter using Lenovo’s Advanced Settings Utility (ASU) or the OneCLI tool as follows.

asu64 show Processors.SnoopResponseHoldOff --override
asu64 set Processors.SnoopResponseHoldOff value --override

onecli config show Processors.SnoopResponseHoldOff --override
onecli config set Processors.SnoopResponseHoldOff value --override

This hidden parameter is accessible on Lenovo ThinkSystem servers that use the Intel Xeon Scalable processors.

With the Intel Xeon Scalable processors non-inclusive cache scheme, mid-level cache (MLC) evictions are filled into the last level cache (LLC) if the data is shared across processor cores. When cache lines are evicted from the MLC, the processor core can flag them as “dead” meaning they are not likely to be read again.

With this command, the LLC can be configured to drop dead lines and not fill them in the LLC. By setting LLCdeadlinealloc to Disabled, dead lines will be dropped. This provides better utilization in the LLC and prevents the LLC from evicting useful data. By setting LLCdeadlinealloc to Enabled, then the processor will determine will determine whether to keep or drop deadlines. The default (initial value) is Enabled.

ASU command:

To show the current setting or to change the setting with ASU, use these commands:

asu64 show Processors.LLCdeadlinealloc --override
asu64 set Processors.LLCdeadlinealloc value --override

Where value is Auto, Disable or Enable. The parameter --override is required when viewing or changing the setting.

OneCLI command:

To show the current setting or to change the setting with OneCLI, use these commands:

onecli config show Processors.LLCdeadlinealloc --override
onecli config set Processors.LLCdeadlinealloc value --override

Where value is Auto, Disable or Enable. The parameter --override is required when viewing or changing the setting.

This hidden parameter is accessible on Lenovo ThinkSystem servers that use the Intel Xeon Scalable processors.

The in-memory directory has three states:

• Invalid: The invalid (I) state means the data is clean and does not exist in any other processor’s cache.
• snoopAll: The snoopAll (A) state means the data may exist in another processor in an exclusive or modified state.
• Shared: The shared (S) state means the data is clean and may be shared across one or more processor’s caches.

When executing a read to memory, if the directory line is in the A state, the processor must snoop all other processors because another processor may have the line in modified state. If this is the case, the snoop will return the modified data. However, it may be the case that a line is read in A state and all the snoops come back a miss. This can happen if another processor read the line earlier and then silently dropped it from its cache without modifying it.

By setting StaleAtoS to Enabled, a line in A state returns only snoop misses and the line will transition to S state. Subsequent reads to the line will encounter it in S state and not have to snoop, saving latency and snoop bandwidth. When StaleAtoS is disabled, it is possible that a line is read in the A state and all the snoops come back a miss. This can happen if another socket read the line earlier and then silently dropped it from its cache without modifying it. This consumes snoop bandwidth unnecessarily. The default (initial value) is Disabled.

ASU command:

To show the current setting or to change the setting with ASU, use these commands:

asu64 show Processors.StaleAtoS --override
asu64 set Processors.StaleAtoS value --override

Where value is Auto, Disable or Enable. The parameter --override is required when viewing or changing the setting.

OneCLI command:

To show the current setting or to change the setting with OneCLI, use these commands:

onecli config show Processors.StaleAtoS --override
onecli config set Processors.StaleAtoS value --override

Where value is Auto, Disable or Enable. The parameter --override is required when viewing or changing the setting.