Lenovo ThinkSystem Firmware and Driver Update Best Practices - An Introduction

Firmware is essential to the operation of a server. It controls every single hardware component found in the server; from the onboard chipset, to the built-in modules, to the add-on adapters and drives, and all the way to the entire system, including the self-contained management controller.

Updating the firmware and drivers on a regular schedule is the recommended best practice for several reasons:

• Achieves the highest level hardware availability.
• Enables you to proactively apply the latest bug fixes before your systems are affected by them.
• Increases security, compatibility, and system uptime.

Lenovo has the richest portfolio of firmware and drivers update tools in the industry. These tools provide easy-to-use graphical utilities to perform updates interactively. They also provide a complete set of utilities to meet more complex scenarios to acquire and apply updates. The utilities support:

• Graphical and command line modes.
• Interactive or unattended updates.
• Local or remote updates.
• Apply updates to a single target or multiple targets.
• Run updates within or outside of the operating system.
• Perform task-based or policy-based updates.
• Use local or online repositories.

This guide provides introductory instruction for users who are new to Lenovo ThinkSystem products. If you are managing a few servers interactively and have no operational constraints, such as Internet access, this is the place to start. This guide will present three simple interactive tools, which are part of the XClarity Essentials toolset:

• Bootable Media Creator
• UpdateXpress
• OneCLI

A subsequent guide will provide advanced level instruction to users managing a complex mix of servers and operating systems. The advanced guide will describe how to update firmware in a distributed environment while minimizing downtime to your workload.

Consider these recommendations:

1. Use of UpdateXpress System Packs

   Lenovo recommends that you update the entire system to the latest UpdateXpress System Pack (UXSP) level before you deploy the server into a production environment. This includes system firmware, all adapter and hard-drive firmware, and the corresponding device drivers in the operating system.

   Tip: Install all the hardware components (modules, adapters, and drives) and power on the system at least once before updating the entire system, so that everything will be activated, detected, and updated together.

2. Installing system firmware

   If new system management controller firmware (IMM or XCC) is applied, either a system management controller restart (via the XCC/IMM web interface or CLI) or a full power cycle (unplug the server) will be required to activate the pending updates. A virtual reseat will also restart the controller (if the function is available in your server).

   If new UEFI firmware is applied, a server reboot is required to activate the updates. If delayed activation is being used, such as in XClarity Administrator, then the updates will remain as pending (unapplied) on the system until server is restarted.

   If the system management controller firmware update package also includes updated FPGA firmware (as indicated in the change history for the update), then both the system management controller will need to be restarted (via XCC/IMM web interface/CLI) and the server will need to be rebooted before the FPGA change becomes effective. A full power cycle (unplug the server) will achieve both.

3. Installing new hardware

   If you install or upgrade hardware components later, make sure that you perform a full system update to ensure that the system can handle the new hardware, and that the newly installed components have the proper firmware and drivers.

4. Updating firmware manually

   Updating firmware manually: If you are updating individual firmware manually or via your own script but are not using the XClarity tools mentioned above, you should always update the BMC (XCC or IMM) first, restart the BMC and wait 5 minutes, then update UEFI, reboot the server, then update the rest of the system. This order ensures that critical dependencies are satisfied.

5. Subscribe to updates on the Lenovo support site

   Make sure that you visit the Lenovo Support web site regularly, or that you subscribe to product notifications to be informed of critical updates for your devices. Then, plan your maintenance schedule accordingly.

Every Lenovo server that is shipped to a customer contains a set of firmware that has been comprehensively tested together to provide optimal performance. Due to a variety of factors, firmware for one or more components may need to be updated. Firmware updates to individual components may be planned or unplanned, e.g. a fix to a problem not identified previously. To ensure continued optimal performance, the firmware updates are integrated and thoroughly re-tested together. The updated firmware and drivers are available on the Lenovo Support web site.

While it is safe to receive a new server, install a certified operating system, and put the server into production, Lenovo recommends that you update the server to the latest firmware and driver pack before putting the server into production.

There are several reasons for updating firmware and drivers:

• The key advantage of Intel x86 platform is its openness and fast-paced innovation. New hardware technologies, new device classes, new adapters, emerging standards and protocols, new operating systems, and major applications become available in the marketplace on a regular basis.
• Lenovo partners with many hardware component suppliers who update their firmware and drivers continuously to enhance functionality or resolve issues. The Lenovo development team must periodically release new firmware and drivers, and incorporate suppliers’ new firmware and drivers into our update pack. While all systems receive the latest firmware at time of manufacturing, there could be additional updates between manufacturing date and system installation at your site.
• Lenovo service and solution teams work directly with customers around the world to meet their business requirement. As we help customers fine tune the servers for their workloads, or address technical issues, we’re eager to share the benefit to all other customers. We’re continuously enhancing firmware to improve security, compatibility and system uptime.

Lenovo actively works with the major software partners to certify our servers for their software, and vice-versa. We also provide our device drivers to be included in their official software release, such as Windows Server, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and VMware ESXi. However, device drivers that are shipped in the official OS release might be out of date or might not support the complete set of features and functions advertised by Lenovo. Therefore, the drivers need to be refreshed at the same time as the firmware.

UpdateXpress System Pack (UXSP) is a bundle of firmware and device drivers that are validated together as a whole for compatibility and stability across all of the supported system models, and that are released on a predefined schedule (typically, 3 times a year).

The UXSP contains additional metadata for the Lenovo tools to optimize sequencing and minimize reboots while applying the updates. Unless explicitly mentioned, this guide always recommends that you use the UXSP for updates, not the individual packages.

Use the following flow chart to determine the best tool to be used when updating the firmware and device drivers, based on your environment.

Figure 1. Update process flow (click for a larger version)

This introductory guide is focused on the following tools:

XClarity Essentials Bootable Media Creator

Bootable Media Creator (BoMC) is ideal for pre-operating-system firmware updates and for updates to “desktop-less” operating systems, such as VMware ESXi, Windows Server Core, or Nano:

• Supports bare-metal update of entire system firmware, without the operating system drivers.
• Acquires UXSP automatically from Lenovo Support web site.
• Creates a bootable DVD, USB, or ISO image, which includes the update tool and firmware files.

We describe BoMC in the Bootable Media Creator section below.

XClarity Essentials UpdateXpress

UpdateXpress is ideal for post-OS update, and periodic refresh of firmware and OS drivers:

• Runs inside the Windows and Linux GUI environment to update the entire system firmware and OS drivers.
• Acquires UXSP automatically from the Lenovo Support web site.

We describe UpdateXpress in the UpdateXpress section below.

XClarity Essentials OneCLI

OneCLI is a command-line utility that is ideal for a non-GUI environment:

• Provides a command line interface to the same functions available with UpdateXpress.
• Enables the scripting of updates.

We describe OneCLI in the OneCLI section below.

Other tools can be used to update firmware, but they are not in the scope of this guide. For example, users can always update device drivers through the operating system. Windows Server and VMware have their own way to update drivers individually and on a large scale.

Additional Lenovo tools that can be used to update firmware include:

• XClarity Provisioning Manager. Available by pressing F1 when the server boots, XClarity Provisioning Manager can be used to update UEFI, BMC firmware, and the XClarity Provisioning Manager software.
• XClarity Controller. Available on the server, XClarity Controller provides a browser-based and a command-line interface to update system firmware and some adapter firmware. However, it does not support using a UXSP; you must update each firmware package separately.
• XClarity Administrator. XClarity Administrator is recommended for customers working in a remotely managed environment. It provides the centralized system management for servers, storage, and switches. XClarity Administrator supports policy-based updates to the system and adapter firmware for multiple servers from a local repository.

Bootable Media Creator (BoMC) is used for pre-operating-system firmware updates and for updates to “desktop-less” operating systems, such as VMware ESXi, Windows Server Core, or Nano

BoMC runs in any Windows or Linux system (for example a Windows 10 laptop), and it requires Internet access to acquire firmware online.

Download BoMC from the Lenovo support site:
https://datacentersupport.lenovo.com/us/en/solutions/lnvo-bomc

This tool runs on your Windows computer or Linux-based workstations, to acquire the UXSP online based on your selected system model(s). It creates an image that includes a bootable operating system, the update tool, and all the firmware files applicable to the selected system model(s). It can write the image directly to a DVD or USB drive that you can carry to the target server and boot.

Alternatively, if you are working remotely from the target server, the ISO image that it creates can be remotely mounted to the management controller as a virtual DVD, if the function is available on your server. Then you can boot the server from the mount and apply the updates.

To use BoMC, do the following:

1. Download the tool from the link provided above
2. Start the application:

   In Windows, open Windows Explorer and navigate to the folder where you downloaded the executable file and double click to launch it: lnvgy_utl_bomc_version number_windows_i386.exe

   Tip: To write the image directly to a DVD or USB drive, rather than to an ISO file, you must right-click the icon and then click Run as administrator.

   In Linux, open a terminal session, change to the directory where you downloaded the application and start the application as root with the command:

   ./lnvgy_utl_bomc_version number_linux_distro_x86-64.bin

   Tip: To write the image directly to a DVD or USB drive, ensure that you have write access privileges to the resource.

3. Step through the wizard, and choose one or more system models for which you intend to create the image. You can choose one or more ThinkSystem models, or one or more System x models, but not both ThinkSystem and System x models at the same time.
4. At the Acquire Location step, choose to acquire the UXSP from Lenovo Support web site.

   Note: Downloads for each system model may exceed 1 GB of downloaded data.

5. At the Media Format step, choose to write the output to an ISO image file, or write directly to the DVD or USB drive. The tool creates an image first, and then writes the image to the device. You will be prompted to format the media on the chosen device at a later time.
6. Step through the wizard to confirm the selection and check back once in a while to answer the prompt.

After the image is created or written to a drive, you can boot from it on the target server(s). The server does the following:

1. Boots an operating system.
2. Launches the update tool.
3. Collects current hardware inventory and current firmware versions.
4. Compares those to the applicable updates.
5. Shows the comparison result on screen while waiting for user confirmation.

Upon confirmation or timeout, the tool automatically applies the updates and reboots the system. The update process runs unattended. However, you can also intervene manually after the comparison result is displayed, if you want to.

If the server to be updated has the same or newer firmware than what is in the image, the update process automatically skips those items, unless you intervene and select an item manually. Likewise, If you boot the image on a server model that is different from the server for which the image was create, the tool skips the update process and indicates that there is nothing to update.

If you just bought a batch of servers of the same or various models, you can create one image for all of them and use it among those servers. You can also burn the image onto several DVDs or several USB keys to update them all at the same time. If you have similar servers across a number of remote branch offices, you can also send the image to the remote branch office. You can get someone in the remote branch office to burn a DVD, and then boot the server to update it locally.

For customers who are deploying VMware ESXi, Lenovo provides Lenovo’s-VMware Certified-Custom ESXi Images, and Lenovo strongly recommends that customers use these images. They include the latest Lenovo-qualified I/O drivers and, when your system is updated with the BoMC process, match entries on VMware’s Hardware Combability List. Lenovo’s VMware ESXi Custom Images can be used for initial ESXi deployments or for updates/upgrades to existing deployments. Go to https://vmware.lenovo.com/content/custom_iso/ for additional details.

UpdateXpress is used to apply a firmware/driver update within the target server operating system, and to periodically refresh of firmware and drivers.

UpdateXpress runs on supported versions of Windows Server, RHEL and SLES. It requires Internet access to acquire firmware and drivers online.

Download UpdateXpress from the Lenovo support site:
https://datacentersupport.lenovo.com/us/en/solutions/lnvo-xpress

This tool runs in the target server’s operating system desktop session as Administrator or as root. It automatically detects the hardware inventory, firmware version, and operating system. It can acquire a UXSP online, and then update the firmware and device drivers. It also implements an update sequence to minimize reboots; the server is rebooted one time after all updates have been applied.

To use UpdateXpress, do the following:

1. Download the tool from the link provided above; unzip the file in a temporary directory.
2. Start the application:
   • In Windows, right-click the executable file lxce_ux.exe and then click Run as administrator.
   • In Linux, open a terminal session, change to the directory where you unziped the file, and execute: ./start_lxce_ux.sh
3. At the Update Task step, choose to update the local machine.
4. At the Update Type step, choose UXSP as the update type.
5. Step through the wizard and follow the instructions.
6. The tool downloads firmware metadata to compare against the current hardware inventory; then it downloads the firmware payload for components that need to be updated.
7. Firmware and driver updates will be applied in a sequence that minimizes reboot. Changes will be effective after the reboot is completed.

This is the easiest tool for updating the entire system if your environment has no constraint on Internet access or running the management tool. This tool does not need to be installed, nor does it create any permanent daemon or background service at runtime. Therefore, you can run it from a temporary folder then delete it at the end.

OneCLI is used to apply firmware/driver updates within the operating system for a target server, and to periodically refresh of firmware and drivers.

OneCLI runs on supported versions of Windows Server, RHEL and SLES. It requires Internet access to acquire the UXSP packages online.

Download OneCLI from the Lenovo support site:
https://datacentersupport.lenovo.com/us/en/solutions/lnvo-tcli

This tool runs in Windows Command Prompt as Administrator, or in Linux terminal as root. It acquires the UXSP packages online, and then updates the firmware and device drivers. It also implements an update sequence to minimize reboots; the server is rebooted one time after all updates have been applied.

To use OneCLI if the target server is connected to the Internet:

1. Download the tool specific to the server type/operating system from the link provided above.
2. Expand the compressed file download into a new directory.
3. Open a command prompt/terminal window and change into the directory where the expanded version of OneCLI was written.
4. Acquire updates using the command:

   onecli[.exe] update acquire --mt XXXX --ostype ostype --dir uxsp

   where:

   • The XXXX parameter is the four digit machine type of targeted server
   • The ostype parameter is one of the following: win2012r2, win2016, rhel6, rhel7, sles11, or sles12
5. Apply updates using the command:

   onecli[.exe] update flash --dir uxsp

Firmware and driver updates are applied in a sequence that minimizes the number of reboots. Changes will be effective after the user reboot the server.

If the target server is not connected to the Internet, you can run UpdateXpress or OneCLI on an Internet-connected workstation to acquire the firmware packages to build your own repository. Then you can copy the folder to the target servers using a USB drive or over the network. Finally, you can run UpdateXpress or OneCLI on the target servers by pointing to that folder.

To use OneCLI from your own update repository:

1. Expand the compressed file, designated for your workstation, into a new directory.
2. Open a command prompt/terminal window and change into the directory where the expanded version of OneCLI was written.
3. Acquire updates using the command:

   onecli[.exe] update acquire --mt XXXX --ostype ostype --dir uxsp

   where:

   Note: The uxsp directory is a subdirectory of the expanded OneCLI directory.

   • The XXXX parameter is the four digit machine type of targeted server
   • The ostype parameter is one of the following: win2012r2, win2016, rhel6, rhel7, sles11, or sles12
4. Transfer the expanded OneCLI directory, which includes the files in the uxsp directory, to a new directory on the targeted server. Apply updates using the command:

   onecli[.exe] update flash --dir uxsp

Changes will be effective after you reboot the server.

As you try out the tools described in this guide, you will find that they have more functions than what was described here, and you are encouraged to try them out thoroughly. You will notice too that BoMC and UpdateXpress fully support the entire Lenovo System x M5 and X6 family of servers. So do not hesitate to apply these best practices to your existing servers as well.

As you get more comfortable with updating the entire system firmware using these tools, and you start to apply the process to more servers, you will want to look for ways to increase productivity, minimize downtime, and schedule a specific reboot time after an update. The advanced guide will show you how to accomplish these procedures.

For more information, see the following resources:

ToolsCenter for Lenovo x86 servers:
https://datacentersupport.lenovo.com/us/en/documents/lnvo-center

XClarity Essentials online documentation:
http://sysmgt.lenovofiles.com/help/topic/xclarity_essentials/overview.html