Lenovo ThinkSystem V3 and V4 Server Firmware and Drivers Update Best Practices - Advanced Guide

Firmware is essential to the operation of a server. It controls every important hardware component in the server, such as the many chipsets on the system board, the power supply unit (PSU), the fan controller, the storage controller, the disk drive, the drive backplane, the PCIe adapter, the PCIe riser, and of course the most important system firmware, the baseboard management controller (BMC), the FPGA, and the UEFI.

Updating the firmware and drivers on a regular schedule is the recommended best practice for several reasons:

• Achieves the highest level hardware availability.
• Enables you to proactively apply the latest bug fixes before your systems are affected by them.
• Increases security, compatibility, and system uptime.

Lenovo has the richest portfolio of firmware and drivers update tools in the industry. These tools provide easy-to-use graphical utilities to perform updates interactively. They also provide a complete set of utilities to meet more complex scenarios to acquire and apply updates.

The utilities support:

• Graphical web interface or command line modes.
• Interactive wizard or unattended updates.
• Apply updates to a single target or multiple targets.
• Run updates within or outside of the operating system.
• Perform task-based or policy-based updates.
• Activate the changes immediately or on trigger.
• Use local or online repositories.

There is an introductory guide for users who are new to Lenovo ThinkSystem servers. If you are managing a few servers interactively and have no operational constraints such as OS administrative privilege and have Internet access, that is the place to start.

This advanced guide will help the users who want to maximize productivity in a large server farm, or minimize maintenance window for critical production server, or need to update a specific piece of firmware for troubleshooting.

The tools presented in this guide are either embedded in every server or can be downloaded free of charge:

• XClarity Essentials (LXCE) UpdateXpress and OneCLI
• XClarity Controller 2 (XCC2) and XClarity Controller 3 (XCC3)
• XClarity Provisioning Manager (LXPM)

There are other tools that also update firmware in large scale, namely XClarity Administrator, XClarity Orchestrator, XClarity Integrator and XClarity One. Those are Lenovo licensed software with a comprehensive management console and additional capabilities such as policy-based updates and cluster rolling updates. They are not being covered in this guide.

The capabilities presented in this guide are dependent on the server having a BMC, more specifically the XCC2 or XCC3. The entire family of ThinkSystem V3 servers have the XCC2 including both Intel-based and AMD-based models, likewise for the ThinkSystem V4 with XCC3. Servers that have no BMC or have the XCC (version one) cannot fully support the capabilities covered here.

Lenovo recommends that you update the entire system to the latest service pack bundle before you deploy the server into a production environment. This includes system firmware, all adapter and hard-drive firmware, and the corresponding device drivers in the operating system.

Install all the hardware components (modules, adapters, and drives) and power on the system at least once before updating the entire system, so that everything will be activated, detected, and updated together.

If you install or upgrade hardware components later, make sure that you perform a full system update to ensure that the system can handle the new hardware, and that the newly installed components have the proper firmware and drivers.

Make sure that you visit the Lenovo Support web site regularly, or that you subscribe to product notifications to be informed of critical updates for your devices. Then, plan your maintenance schedule accordingly.

Lenovo ThinkSystem V3 server is equipped with a new management controller, XClarity Controller 2 (XCC2), that is more powerful and has larger internal storage. The XCC2 has several new features compared to the previous XCC, including a new firmware update mechanism.

The XCC2 can now handle a full system firmware update by storing all the payload in the internal storage. It can apply updates to components by out-of-band mechanism (OOB) and in-band mechanism (IB). It is more desirable to update firmware by OOB while the hardware component continues to operate, and it will pick up the new code when the system reboots.

However, some hardware components are not supporting OOB yet, for example all disk drives (old HDD and new SSD alike), some NIC, some GPU, they must be updated in-band. In the past, those “IB firmware” can only be updated from within the host OS but that could be challenging if the host OS is a hypervisor or is not in the list of supported OS. With XCC2, this constraint has been addressed by the new update mechanism using a Linux-based embedded OS to update the IB firmware. The embedded OS is called Lenovo XClarity Update Manager (LXUM) with its own version and is refreshed along with the rest of the system firmware periodically.

Thanks to the new update mechanism and the larger storage capacity, it is now possible to transfer the updates payload to XCC2 staging area as step 1 and apply the updates as step 2 at another time. This preparation can effectively cut the maintenance window by half because the time-consuming file transfer task can be done hours or days prior to the activation time. This is especially useful for remote server that have a slower or unstable link to the datacenter, such as edge deployment, branch office or retail store.

In concert with the new update mechanism, the update package format was changed too. Now we are calling it a service pack bundle, in place of the previous UpdateXpress System Pack (UXSP). The concept remains same, about twice a year, we’re taking all the cumulative firmware and drivers updates during that period and test them together and release it as a qualified coherent service pack bundle.

Individual updates for various components are still being released to address pressing customer issues, it is called an update bundle, or simply bundle. In the past, an update package is often delivered in self-extracting executable format (.exe or .bin) that can run directly in the OS to extract or install. The new bundle is now delivered in compressed format (.zip) that includes JSON metadata and built-in installation script in native shell language, such as installWindowsDrivers.ps1 for Windows, installLinuxDrivers.sh for Linux. This makes the bundle more transparent to user scrutiny or to security software scanning.

Note that only bundles that are manually downloaded from Lenovo datacenter support website have built-in installation script. If you acquire the bundles using XClarity Essentials then you must install it using the same tool.

The following sections will illustrate how to make good use of the new update capabilities.

The ThinkSystem V4 servers come with a new firmware stack and many improvements in the XCC3 while preserving the same user-friendly web UI with only minor adjustments. The instruction steps for XCC2 are equally applicable to XCC3.

From a firmware update perspective, the ThinkSystem V4 server uses the same structure of service pack bundle and individual update bundle. The methods to update firmware and OS drivers are identical.

There is one difference that should be noted here. On all the servers, there are one or more FPGA modules that work alongside the BMC to regulate the server operation. These FPGA modules have their own firmware but it used to be packaged together with the XCC/XCC2 update bundle, until now.

From ThinkSystem V4 onward, the FPGA firmware is no longer packaged inside XCC3, it will be a separate item visible to the user, with its own version, and updatable individually. This will enable more efficient troubleshooting when the issue might be caused by the FPGA functions, it was harder to do when that was hidden from the user.

That being said, we don’t recommend to update the FPGA firmware alone, unless you’re being told by Lenovo Support to do so, because it has intricate dependency with the XCC3 and UEFI version. The best practice is to update it in a service pack bundle together with everything else.

A common scenario is to update the entire system including all the firmware and device drivers.

• Updates using UpdateXpress
• Updates using OneCLI

To use UpdateXpress:

1. Download the tool from the UpdateXpress web page, and unzip the file in any directory that has enough free space (a few gigabytes will do).
2. Start the application in Windows by launching exe (run as Administrator role) or in Linux graphical desktop by executing the following (run as root):

   ./start_lxce_ux.sh 
   

3. At the Update Task step, choose to update the local machine.
4. At the Update Type step, choose UXSP as the update type, this is equivalent to a bundle.
5. Step through the wizard and follow the instructions.

The tool downloads online metadata to compare against the current hardware inventory, then it acquires only the firmware and device drivers payload that need to be updated. This reduces the amount of download substantially if the server firmware is reasonably recent. Some components are updated immediately and some components will be updated at reboot time (i.e., disk drives updates); so it is expected that the next reboot might take several minutes longer than usual.

Alternatively, you can launch UpdateXpress from any Windows or Linux workstation to update a remote system by selecting to “Manage the remote server” in the wizard step 2, then input the remote management controller IP address and credentials. The subsequent wizard steps are similar to the above but the outcome will not include device driver updates.

You can also use UpdateXpress to acquire the updates of multiple machine types to build your own local repository and then share it through the common network file sharing protocol (CIFS, NFS, HTTP, HTTPS, SFTP). To do that, select “Create a repository of updates” in the wizard step 2, then select the machine types and OS that you want to acquire for.

Figure 1. XClarity Essential UpdateXpress wizard step 2 showing multiple function selections

Suppose that you want to update two server models, SR650 V3 (7D75) and SR630 V3 (7D72), you have to acquire the service pack bundle first, then run update locally or remotely. Using OneCLI, the steps would be the following:

1. Download the tool from the OneCLI web page, and unzip the file in a temporary directory.
2. Open a Windows command prompt (as Administrator role) or a Linux terminal shell as root.
3. Acquire the bundles for two models and OS drivers using this command:

   onecli.exe update acquire --scope bundle --mt 7D75,7D72 --ostype win2022 --dir D:\Download\2024a
   

4. When the download is completed, you can update the firmware and drivers directly from there:

   onecli.exe update flash --dir D:\Download\2024a
   

5. You could also put the downloaded content in a share directory and map to it from each target server, e.g., map the U: drive to it, then you can run update with the following command:

   onecli.exe update flash --dir U:\Download\2024a
   

6. When the updates are completed, reboot the system and it’s good to go.

Remember that some firmware components might need to be updated in-band which will happen in the embedded OS, so the reboot might take several minutes longer than usual. The reboot could take up to half hour if prerequisite update and intermediate reboot is necessary.

The above example does not require the user to access the XCC2 or to know its password. In fact it could work even if XCC2 is disconnected from the network.

Now suppose that the two servers have no operating system installed yet, or the OS is proprietary, you’d have to run the update from a remote workstation toward the XCC2 over the management network. There are different ways to do it depending on your priority.

If your workstation can connect to the target XCC2 by LAN at good network speed, you could initiate the firmware update from where you acquired the bundles with these commands:

onecli.exe update flash --bundle --dir D:\Download\2024a --bmc USERID:PASSW0RD@server1
onecli.exe update flash --bundle --dir D:\Download\2024a --bmc USERID:PASSW0RD@server2

During the update, the payload will be transferred from the workstation to server1 and then to server2, then XCC2 will perform the firmware update from there.

If you have a short maintenance window and don’t want that time to be consumed by file transfer operation, use the --applytime OnStartUpdateRequest option to tell it NOT to do the update immediately but wait until another command is sent. That way you can prepare the target servers many hours or days ahead of the maintenance time.

For example:

onecli.exe update flash --bundle --dir D:\Download\2024a --bmc USERID:PASSW0RD@server1 --applytime OnStartUpdateRequest

The command result clearly indicates the task ID and the firmware that are staged for future update.

Figure 2. Use the --applytime option to staged updates to XCC2 and trigger the task later

When you’re ready to apply the update, use the option startstaged as follows:

onecli.exe update startstaged --bmc USERID:PASSW0RD@server1

If you forgot the task ID, or you’re not sure if anything was being staged, use the checktask option to find out:

onecli.exe update checktask --bmc USERID:PASSW0RD@server1

If you decide to cancel the staged task, use the canceltask option as follows:

onecli.exe update canceltask --taskid d17b7d42-c665-4c9e-93bf-333d69e7332b --bmc USERID:PASSW0RD@server1

Regarding the file transfer operation, if you’re updating 2 servers from your workstation that is probably fine. If you’re updating 20 servers then the workstation will likely be a bottleneck due to disk I/O and network bandwidth. Also the workstation has to be there during the whole time.

If the update repository is on a file server instead of local disk, then your workstation is getting it from the file server and sending it away to the target XCC2, that is twice the network traffic for each server! In that case, it is better to tell the XCC2 to connect to the repository directly without passing through your workstation. That is more scalable if you’re updating many servers in parallel. That is also more desirable if your workstation must go through a WAN or VPN to reach to the XCC2.

Use the --fileserver option to specify a SFTP, HTTP and HTTPS share path, for example:

onecli.exe update flash --bundle --bmc USERID:PASSW0RD@server1 --fileserver sftp://sftpuser:password@sftpserver/Download/2024a

If you are sharing through CIFS or NFS, you can use the XCC2 web interface to submit the update. Login to XCC2 web interface, go to Firmware Update page and scroll to the Update from Repository section. Select a share protocol then input the path and credential.

Figure 3. Using Update from Repository to connect to CIFS or NFS shared repository

If you’re working at a highly restricted area where you can’t use your own workstation to run the update, and there is no file server that can be used. You can go to the Lenovo support site, input the machine type, select Drivers & Software, then Update Bundles (Service Packs), then select the bundle for "Any OS" and download the System Firmware Bundle Payload. You only need to deal with that one file.

Figure 4. Manually downloading Service Pack Bundle from Lenovo support site

Put that file on a thumb drive or any storage media, then use XCC2 web interface to upload the bundle file onto the XCC2 internal storage. To do that, login to XCC2 web interface, go to Firmware Update page and scroll to the Update from Repository section. Select Internal Storage and then click Import Firmware Package. The given bundle zip file will be staged in XCC2 and it will update the entire system. The host server will be restarted during the update. Then you’ll have to restart XCC2 manually to make new XCC2 update effective.

Notice that the service pack bundle includes prerequisite updates. For instance, if your server’s UEFI firmware is old and must be updated to an intermediary version first, that will be taken care of without user intervention, in some cases an extra server reboot will occur automatically.

Figure 5. Using XCC2 internal storage to stage and apply updates

Actually you can upload any individual firmware update to the XCC2 internal storage for applying, it is not limited to a service pack bundle. This is the most convenient way to update firmware in small quantity.

Now that server firmware is up to date, the next step is to update OS drivers. We will use OneCLI to do the updates.

If you can access the same network shared repository illustrated in the previous section (e.g., U: drive), you can point to it and specify the --type dd option to update only device drivers. Run this command from the target OS:

onecli.exe update flash --dir U:\Download\2024a --type dd

Then reboot the OS for the new drivers to be effective.

If you cannot access the existing shared repository or want to download only the drivers updates for a specific OS, run this command to acquire and to update:

onecli.exe update acquire --mt 7D72 --dir D:\Download\2024a_drivers --ostype win2022 --scope latest --type dd
onecli.exe update flash --dir U:\Download\2024a_drivers

Use these commands for Linux environment:

./onecli update acquire --mt 7D72 --dir /Download/2024a_drivers --ostype rhel9,sles15 --scope latest --type dd
./onecli update flash --dir /Download/2024a_drivers

Reboot the OS and your server is good to go.

For customers who are deploying VMware ESXi, Lenovo provides several options to update the drivers inside ESXi. You can get the latest VMware-certified Lenovo custom ESXi image from there. You can also get driver bundles for VMware Update Manager. Please visit vmware.lenovo.com for more information.

While the general recommendation is to update the entire server using service pack bundle, there will be time where you want to update only a specific component firmware for troubleshooting a problem, or when you’re plugging in a component that has older firmware.

The XCC2 web interface is the best tool for this purpose. You can download the desired individual firmware bundle from Lenovo support site then login to XCC2, go to Firmware Update page.

If you need to update a system firmware, such as UEFI or XCC2, look for the System Firmware section, click on Update Firmware, upload the file, then you’ll be able to choose if you want to update the XCC2 primary or backup image or both.

Figure 6. Use XCC2 to update System Firmware and select individual target as needed

Likewise, if you need to update the FPGA firmware on a ThinkSystem V4 server, upload the FPGA update bundle file, then you’ll be guided by the user interface accordingly. FPGA update requires rebooting the host and the BMC to be effective.

Figure 7. On ThinkSystem V4 server, the FPGA modules are displayed under System Firmware

Note that the SCM and HPM are different firmware bundle that you can download on Lenovo data center support site under the Core System Firmware category. The SCM bundle is called “DCSCM Firmware”. Some server models might not have the HPM module.

If you have multiple same adapters in the system (e.g., 2 RAID controllers, 4 NIC cards) but only want to update one of them, look for the Adapter Firmware section, then click on Update Firmware.

Figure 8. Use XCC2 to update Adapter Firmware and select individual adapter as needed

Likewise if you have multiple PSU but want to update a specific one, look for the PSU Firmware section then click on Update Firmware. You’ll be able to select one specific PSU to update.

If you want to update an in-band firmware component such as disk drive, GPU, some legacy NIC, or any component mentioned previously that doesn’t need specific selection, you can scroll to the Update from Repository section, select the Internal Storage, click on Import Firmware Package, provide the bundle zip file, then XCC2 will take the bundle and apply immediately.

Figure 9. Use XCC2 Update from Repository to update any individual component easily

Depending on the situation, you might need to reboot the server or the XCC2 after the update.

You can use this interface to rollback to an older firmware version as well. Note that there’s a BMC setting at BMC Configuration > Security > Prevent System Firmware Down-Level that is disabled by default. If it is enabled you won’t be allowed to downgrade XCC2, UEFI and LXPM in the system.

In some rare situation where the system firmware cannot be updated via the XCC2 web interface, or if you cannot connect to the XCC2 altogether, there’s another way to update, or downgrade, the system firmware through the XClarity Provisioning Manager.

To do that, reboot the system and press <F1> during power-on self-test (POST), you’ll see the button “F1:System Setup” highlighted in blue while the server continues to initialize. Eventually it will launch the Provisioning Manager interface, go to the Firmware Update page, click the Browse Update Package button to provide the update bundle file from a USB drive. Then you can select the component to update.

Figure 10. Updating system firmware via XClarity Provisioning Manager interface

If your troubleshooting is successful and you want to keep the new piece of firmware, make sure that the corresponding OS drivers is also in sync. For that, go to Lenovo support site to download the driver bundle. Unzip the bundle into a temporary directory in the target OS. Then run the embedded install script within the expanded bundle directory:

Run in Windows PowerShell:

./installWindowsDrivers.ps1

Run in Linux bash shell:

./installLinuxDrivers.sh

Figure 11. Use the embedded installation script to update Windows or Linux driver bundle

It will detect the machine type, hardware component, OS version, then update the appropriate drivers accordingly.

Here’s a quick summary of the best practices:

1. LXCE UpdateXpress can update all firmware and device drivers when running in the target OS; it minimizes download by comparing metadata first.
2. LXCE OneCLI can be used to acquire service pack bundles of several models; then remotely update many servers from one place. It can also update drivers when running in the target OS.
3. If you’re updating a large number of servers, it is preferrable to build a local repository by acquiring the updates once and put them on a network share, then have XCC2 fetch the updates from the network share directly.
4. With LXCE OneCLI, you can use the parameter --applytime OnStartUpdateRequest or
   --applytime OnReboot to transfer the firmware payload to XCC2 and later on trigger the update process at the desired time, or cancel it. This can be very useful for server located at remote site with slow network connection.
5. Use XCC2 Firmware Update web interface to update a specific component, such as system firmware, adapter firmware and PSU firmware.
6. Use XCC3 Firmware Update web interface to update the FPGA firmware. It requires rebooting the host and the BMC to be effective.
7. Use XCC2 Update from Repository wizard to update in-band disk drive firmware, legacy NIC, GPU, or to update an entire service pack bundle.
8. You can also update system firmware using LXPM by pressing <F1> at server boot time. For that, you’d have to provide the updates in a USB stick or by connecting to a network share inside LXPM.
9. Device driver bundles that are downloaded from Lenovo website have an installation script that can run directly in the OS; bundles that are downloaded using OneCLI must be installed with OneCLI.

In this guide, we explained the benefits of keeping firmware and drivers up to date, and the enhancements in ThinkSystem V3 server that lead to a better update experience. We provided many examples on using XClarity Essentials UpdateXpress, OneCLI, XClarity Controller 2 and XClarity Provisioning Manager to update firmware and drivers in a way that is easy to do, minimize downtime, minimize bandwidth, and updating individual component in case of troubleshooting.

Updating firmware is a necessary burden to keep your servers in good shape. Once you’re comfortable with the tools and commands illustrated here, hopefully you’ll cultivate a habit of keeping your servers up to date every 6 months or every year.

For more information, see these resources:

• Lenovo ThinkSystem Firmware and Driver Update Best Practices - An Introduction
  https://lenovopress.lenovo.com/lp0656-lenovo-thinksystem-firmware-and-driver-update-best-practices
• Lenovo data center support website
  https://datacentersupport.lenovo.com
• Lenovo XClarity Essentials UpdateXpress
  https://datacentersupport.lenovo.com/us/en/solutions/lnvo-xpress

• Lenovo XClarity Essentials OneCLI
  https://datacentersupport.lenovo.com/us/en/solutions/lnvo-tcli

CheKim Chhuor (CK) is a Principal Designer in the Lenovo Infrastructure Solutions Group. He is responsible for the user experience and visual design of several product families including ThinkSystem, ThinkEdge, ThinkAgile, and XClarity. CK has 20+ years of experience in the enterprise system management space. He has authored many articles on software product review and utilization.