Top
Author
Updated
31 Oct 2023Form Number
LP0901PDF size
15 pages, 268 KBSubscribed to LP0901.
Thank you for your feedback.
Table of Contents
Abstract
The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter delivers low sub-600µs latency, extremely high message rates, RoCE v2, NVMe over Fabric offloads and embedded PCIe switch. Mellanox provides the highest performance and lowest latency for the most demanding applications: High frequency trading, Machine Learning, Data Analytics, and more. This adapter ships ready to accept 10GbE, 25GbE or 40GbE network connections.
This product guide provides essential presales information to understand the ConnectX-5 Ex adapter and its key features, specifications, and compatibility. This guide is intended for technical specialists, sales specialists, sales engineers, IT architects, and other IT professionals who want to learn more about the adapter and consider its use in IT solutions.
Change History
Changes in the October 31, 2023 update:
- Added the following transceiver - Supported transceivers and cables section:
- ThinkSystem Finisar Dual Rate 10G/25G SR SFP28 Transceiver, 4TC7A88638
Introduction
The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter delivers low sub-600µs latency, extremely high message rates, RoCE v2, NVMe over Fabric offloads and embedded PCIe switch. Mellanox provides the highest performance and lowest latency for the most demanding applications: High frequency trading, Machine Learning, Data Analytics, and more. This adapter ships ready to accept 10GbE, 25GbE or 40GbE network connections.
The following figure shows the ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter.
Figure 1. ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter (heatsink removed)
Did you know?
NVMe storage solutions are gaining popularity, offering very fast storage access. The evolving NVMe over Fabric (NVMe-oF) protocol leverages the Mellanox RDMA connectivity for remote access. ConnectX-5 offers further enhancements by providing NVMe-oF target offloads, enabling very efficient NVMe storage access with no CPU intervention, and thus improved performance and lower latency.
Part number information
The ordering information is listed in the following table.
| Part number |
Feature code |
Mellanox equivalent |
Description |
|---|---|---|---|
| 4XC7A08229 | B31C | MCX516A-BDAT | ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter |
The adapter, when shipped as a stand-alone option, includes the following items:
- One Mellanox ConnectX-5 Ex EN adapter
- Two QSFP28 to SFP28 transceiver adapters (QSA28)
- Full-height (3U) bracket attached with low-profile (2U) bracket included in the box
- Documentation flyer
QSA28 adapters: The option part number includes the QSA28 adapters, however when ordering the adapter as part of a server CTO build, the feature code B31C does not include the adapters and they must be ordered separately.
Supported transceivers and cables
The adapter has two empty QSFP28 cages. The adapter also includes two QSA28 adapters that convert the QSFP28 cages to SFP28 for 25Gb connectivity.
The adapter either supports a connection to a 1Gb, 10Gb, 25Gb or 40Gb switch or can share a connection to a 100Gb switch using a 4:1 breakout cable.
The following table lists the supported transceivers.
Configuration note:
- Two QSA 100G to 25G adapters are included with the ConnectX-5 Ex adapter part number, 4XC7A08229, however they are not automatically included with the adapter when ordered with a server as part of a CTO build. For CTO orders, select the QSA adapters (feature B306) separately if needed.
25Gb transceivers: When installed in this 25Gb Ethernet adapter, 25Gb transceivers are designed to operate at either 25 Gb/s or 10 Gb/s speeds as listed in the description of the transceiver, however the speed also depends on the negotiation with the connected switch. In most configurations, this negotiation is automatic, however in some configurations you may have to manually set the link speed or FEC mode.
The following table lists the supported fiber optic cables and Active Optical Cables.
The following table lists the supported direct-attach copper (DAC) cables.
Features
The ThinkSystem Mellanox ConnectX-5 Ex 25/40GbE 2-port Low-Latency Adapter has the following key features:
Ultra-low latency
Mellanox ConnectX-5 adapter cards feature exceedingly low latency and hardware-based RDMA technology that is perfect for meeting the needs of the financial services industry. Based on advanced technology providing sub-microsecond latency and a capacity of 200 million messages per second, ConnectX-5 offers the next generation of connectivity today.
Mellanox’s Messaging Accelerator (VMA) transparently boosts performance for socket-based applications without having to modify the applications. By offloading network processing from the OS, traffic passes directly to the network adapter from the application user space, bypassing the kernel IP stack and minimizing context switches, buffer copies, and interrupts. The result is a reduction in latency by 500% and an increase in application throughput, as compared to applications running on a standard network kernel stack.
Cloud, SDN, and NFV environment support
Mellanox Accelerated Switching and Packet Processing (ASAP2) Direct technology allows to offload vSwitch/vRouter by handling the data plane in the NIC hardware while maintaining the control plane unmodified. As a result there is significantly higher vSwitch/vRouter performance without the associated CPU load
The vSwitch/vRouter offload functions that are supported by ConnectX-5 include Overlay Networks (for example, VXLAN, NVGRE, MPLS, GENEVE, and NSH) headers’ encapsulation and de-capsulation, as well as stateless offloads of inner packets, packet headers’ re-write enabling NAT functionality, and more.
Moreover, the intelligent ConnectX-5 flexible pipeline capabilities, which include flexible parser and flexible match-action tables, can be programmed, which enable hardware offloads for future protocols.
ConnectX-5 SR-IOV technology provides dedicated adapter resources and guaranteed isolation and protection for virtual machines (VMs) within the server. With ConnectX-5 Network Function Virtualization (NFV), a VM can be used as a virtual appliance. With full data-path operations offloads as well as hairpin hardware capability and service chaining, data can be handled by the Virtual Appliance with minimum CPU utilization.
With these capabilities data center administrators benefit from better server utilization while reducing cost, power, and cable complexity, allowing more Virtual Appliances, Virtual Machines and more tenants on the same hardware.
Specifications
The adapter has the following technical specifications:
- PCIe 4.0 x16 host interface
- Support for MSI/MSI-X mechanisms
- Two QSFP28 external connectors supporting a transceiver, converter, direct-attach copper (DAC) cable or active optical cable (AOC).
- Data rates supported: 1, 10, 25 and 40 Gb/s Ethernet
- TLP (Transaction Layer Packet) Processing Hints (TPH)
- Embedded PCIe Switch: Up to 8 bifurcations
- PCIe switch Downstream Port Containment (DPC) enablement for PCIe hot-plug
- Access Control Service (ACS) for peer-to-peer secure communication
- Advance Error Reporting (AER)
- Process Address Space ID (PASID) Address Translation Services (ATS)
- IBM CAPI v2 support (Coherent Accelerator Processor Interface)
- Support for MSI/MSI-X mechanisms
Ethernet:
- 40GbE / 25GbE / 10GbE / 1GbE
- IEEE 802.3by, Ethernet Consortium 25, supporting all FEC modes
- IEEE 802.3ba 40 Gigabit Ethernet
- IEEE 802.3ae 10 Gigabit Ethernet
- IEEE 802.3az Energy Efficient Ethernet
- IEEE 802.3ap based auto-negotiation and KR startup
- IEEE 802.3ad, 802.1AX Link Aggregation
- IEEE 802.1Q, 802.1P VLAN tags and priority
- IEEE 802.1Qau (QCN) Congestion Notification
- IEEE 802.1Qaz (ETS)
- IEEE 802.1Qbb (PFC)
- IEEE 802.1Qbg
- IEEE 1588v2
- Jumbo frame support (9.6KB)
- IPv4 (RFQ 791)
- IPv6 (RFC 2460)
Enhanced Features:
- Hardware-based reliable transport
- Collective operations offloads
- Vector collective operations offloads
- PeerDirect RDMA (GPUDirect) communication acceleration
- 64/66 encoding
- Extended Reliable Connected transport (XRC)
- Dynamically Connected Transport (DCT)
- Enhanced Atomic operations
- Advanced memory mapping support, allowing user mode registration and remapping of memory (UMR)
- On demand paging (ODP)
- MPI Tag Matching
- Rendezvous protocol offload
- Out-of-order RDMA supporting Adaptive Routing
- Burst buffer offload
- In-Network Memory registration-free RDMA memory access
CPU Offloads:
- RDMA over Converged Ethernet (RoCE)
- TCP/UDP/IP stateless offload
- LSO, LRO, checksum offload
- RSS (also on encapsulated packet), TSS, HDS, VLAN and MPLS tag insertion/stripping, Receive flow steering
- Data Plane Development Kit (DPDK) for kernel bypass applications
- Open VSwitch (OVS) offload using ASAP2:
- Flexible match-action flow tables
- Tunneling encapsulation/ de-capsulation
- Intelligent interrupt coalescence
- Header rewrite supporting hardware offload of NAT router
Storage Offloads:
- NVMe over Fabric offloads for target machine
- Erasure Coding offload – offloading Reed Solomon calculations
- T10 DIF – Signature handover operation at wire speed, for ingress and egress traffic
- Storage protocols: SRP, iSER, NFS RDMA, SMB Direct, NVMe-oF
Overlay Networks:
- RoCE over Overlay Networks
- Stateless offloads for overlay network tunneling protocols
- Hardware offload of encapsulation and decapsulation of VXLAN, NVGRE, and GENEVE overlay networks
Hardware-Based I/O Virtualization
- Single Root IOV
- Address translation and protection
- VMware NetQueue support
- SR-IOV: Up to 1K Virtual Functions
- SR-IOV: Up to 16 Physical Functions per host
- Virtualization hierarchies (e.g., NPAR and Multi-Host, when enabled):
- Virtualizing Physical Functions on a physical port
- SR-IOV on every Physical Function
- Configurable and user-programmable QoS
- Guaranteed QoS for VMs
HPC Software Libraries:
- Open MPI, IBM PE, OSU MPI (MVAPICH/2), Intel MPI
- Platform MPI, UPC, Open SHMEM
Management and Control:
- PLDM for Monitor and Control DSP0248
- PLDM for Firmware Update DSP0267
- SDN management interface for managing the eSwitch
- I2C interface for device control and configuration
- General Purpose I/O pins
- SPI interface to Flash
- JTAG IEEE 1149.1 and IEEE 1149.6
Remote Boot:
- Remote boot over Ethernet
- Remote boot over iSCSI
- Unified Extensible Firmware Interface (UEFI)
- Pre-execution Environment (PXE)
Server support
The following tables list the ThinkSystem servers that are compatible.
Operating system support
The adapter supports the following operating systems:
Tip: This table is automatically generated based on data from Lenovo ServerProven.
Physical specifications
The adapter is a low-profile adapter with the following dimensions:
- Length: 14.2 cm (5.6 in)
- Height: 6.9 cm (2.71 in)
Operating environment
The adapter is supported in the following environment:
- Temperature (operating): 0 to 55 °C (32 to 131 °F)
- Temperature (storage): -40 to 70 °C (-40 to 158 °F)
- Humidity (operating): 10 to 90% non-condensing
- Humidity (storage): 5 to 90% non-condensing
Warranty
One-year limited warranty. When installed in a supported server, the adapter assumes the system’s base warranty and any warranty upgrade.
Agency approvals
The adapter conforms to the following standards:
- Safety: CB / cTUVus / CE
- EMC: CE / FCC / VCCI / ICES / RCM
- RoHS: RoHS-R6
Top-of-rack Ethernet switches
The following table lists the Ethernet LAN switches that are offered by Lenovo.
For more information, see the list of Product Guides in the following switch categories:
- 1 Gb Ethernet switches: http://lenovopress.com/networking/tor/1gb?rt=product-guide
- 10 Gb Ethernet switches: http://lenovopress.com/networking/tor/10gb?rt=product-guide
- 25 Gb Ethernet switches: http://lenovopress.com/networking/tor/25gb?rt=product-guide
- 40 Gb Ethernet switches: http://lenovopress.com/networking/tor/40gb?rt=product-guide
- 100 Gb Ethernet switches: https://lenovopress.com/networking/tor/100Gb?rt=product-guide
Related publications
For more information, see the following resources:
- Mellanox page for ConnectX-5 EN Ethernet adapters:
https://www.nvidia.com/en-us/networking/ethernet/connectx-5/ - Mellanox User Manuals:
https://docs.nvidia.com/networking/display/ConnectX5EN - Mellanox Messaging Accelerator (VMA)
http://www.mellanox.com/page/software_vma?mtag=vma - Networking Options for ThinkSystem Servers
https://lenovopress.com/lp0765-networking-options-for-thinksystem-servers - Lenovo ServerProven compatibility information:
http://www.lenovo.com/us/en/serverproven/ - Support page for the adapter:
https://datacentersupport.lenovo.com/us/en/search?query= 4XC7A08229
Trademarks
Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.
The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
ServerProven®
ThinkSystem®
The following terms are trademarks of other companies:
AMD is a trademark of Advanced Micro Devices, Inc.
Intel® and Xeon® are trademarks of Intel Corporation or its subsidiaries.
Linux® is the trademark of Linus Torvalds in the U.S. and other countries.
Microsoft®, Windows Server®, and Windows® are trademarks of Microsoft Corporation in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Configure and Buy
Please select a locale
Full Change History
Changes in the October 31, 2023 update:
- Added the following transceiver - Supported transceivers and cables section:
- ThinkSystem Finisar Dual Rate 10G/25G SR SFP28 Transceiver, 4TC7A88638
Changes in the September 17, 2023 update:
- Withdrawn cables are now hidden; click Show Withdrawn Products to view them - Supported transceivers and cables section
Changes in the December 15, 2022 update:
- Added the following transceiver - Supported transceivers and cables section:
- ThinkSystem Accelink 40G SR4 QSFP+ Ethernet transceiver, 4TC7A85336
Changes in the November 8, 2022 update:
- Added the following breakout cables - Supported transceivers and cables section:
- Lenovo 1.5m 100G to 4x25G Breakout SFP28 Breakout DAC Cable, 4Z57A85043
- Lenovo 2m 100G to 4x25G Breakout SFP28 Breakout DAC Cable, 4Z57A85044
Changes in the June 16, 2022 update:
- The following transceiver is withdrawn - Supported transceivers and cables section:
- Lenovo QSFP+ 40GBASE-iSR4 Transceiver, 00D9865
Changes in the December 14, 2021 update:
- Added support for the following transceivers - Supported transceivers and cables section:
- ThinkSystem Accelink 10G SR SFP+ Ethernet transceiver, 4TC7A78615
- Lenovo 25Gb SR SFP28 Ethernet Transceiver, 4M27A67041
Changes in the October 20, 2021 update:
- Added a note that the 25GbE transceivers can operate at 25 Gb/s or 10 Gb/s speeds when installed in a 25GbE adapter - Supported transceivers and cables section
Changes in the February 28, 2021 update:
- The server support tables are now automatically updated - Server support section
Changes in the October 13, 2020 update:
- Added the SR850 V2 and SR860 V2 servers - Server support section (no support)
Changes in the May 5, 2020 update:
- Added SR645 and SR665 to the server support table - Server support section (no support)
Changes in the January 7, 2020 update:
- Added SR850P to the server support table - Server support section
Changes in the September 25, 2019 update:
- Added SE350, SR635 and SR655 to the Server support table (no support)
Changes in the December 10, 2018 update:
- Added SFP+ 10Gb Active Optical Cables - Table 3 in the Supported transceivers and cables section
Changes in the November 5, 2018 update:
- Added new servers: ThinkSystem ST50, ST250, SR150 and SR250
- Updated the list of supported operating systems
First published: 17 July 2018
Course Detail
Employees Only Content
The content in this document with a is only visible to employees who are logged in. Logon using your Lenovo ITcode and password via Lenovo single-signon (SSO).
The author of the document has determined that this content is classified as Lenovo Internal and should not be normally be made available to people who are not employees or contractors. This includes partners, customers, and competitors. The reasons may vary and you should reach out to the authors of the document for clarification, if needed. Be cautious about sharing this content with others as it may contain sensitive information.
Any visitor to the Lenovo Press web site who is not logged on will not be able to see this employee-only content. This content is excluded from search engine indexes and will not appear in any search results.
For all users, including logged-in employees, this employee-only content does not appear in the PDF version of this document.
This functionality is cookie based. The web site will normally remember your login state between browser sessions, however, if you clear cookies at the end of a session or work in an Incognito/Private browser window, then you will need to log in each time.
If you have any questions about this feature of the Lenovo Press web, please email David Watts at dwatts@lenovo.com.
