ThinkSystem Mellanox ConnectX-5 EN 10/25GbE SFP28 Ethernet Adapter

The ThinkSystem Mellanox ConnectX-5 EN 10/25GbE SFP28 Ethernet Adapter is a high performance 25Gb Ethernet network adapter that offers multiple network offloads including RoCE v2, NVMe over Ethernet and Open vSwitch.

The following figure shows the ThinkSystem Mellanox ConnectX-5 EN 10/25GbE SFP28 Ethernet Adapter (the standard heat sink has been removed in this photo).

Figure 1. ThinkSystem Mellanox ConnectX-5 EN 10/25GbE SFP28 Ethernet Adapter

NVMe storage devices are gaining popularity by offering very fast storage access. The evolving NVMe over Fabric (NVMe-oF) protocol leverages the RDMA connectivity for remote access. ConnectX-5 offers further enhancements by providing NVMe-oF target offloads, enabling very efficient NVMe storage access with no CPU intervention, and thus improving performance and reducing latency.

The following table shows the part number for the adapter.

The part numbers include the following:

• One Mellanox adapter
• Low-profile (2U) and full-height (3U) adapter brackets
• Documentation

The following table lists the supported transceivers.

The following table lists the supported optical cables.

The Mellanox ConnectX-5 EN 10/25GbE SFP28 Ethernet Adapter offers a number of features that are well suited for workloads, including the following:

Cloud and Web 2.0 Environments

ConnectX-5 adapter cards enable data center administrators to benefit from better server utilization and reduced costs, power usage, and cable complexity, allowing for more virtual appliances, virtual machines and tenants to co-exist on the same hardware.

Supported vSwitch/vRouter offload functions include:

• Overlay Networks (e.g., VXLAN, NVGRE, MPLS, GENEVE, and NSH) header encapsulation & decapsulation.
• Stateless offloads of inner packets and packet headers re-write, enabling NAT functionality and more.
• Flexible and programmable parser and match-action tables, which enable hardware offloads for future protocols.
• SR-IOV technology, providing dedicated adapter resources, guaranteed isolation and protection for virtual machines (VMs) within the server.
• Network Function Virtualization (NFV), enabling a VM to be used as a virtual appliance. The full data- path operation offloads, hairpin hardware capability and service chaining enables data to be handled by the virtual appliance, with minimum CPU utilization.

Cloud and Web 2.0 customers developing platforms on Software Defined Network (SDN) environments are leveraging the Virtual-Switching capabilities of the server operating systems to achieve maximum flexibility. Open vSwitch (OvS) is an example of a virtual switch that allows Virtual Machines to communicate with each other and with the outside world. Traditionally residing in the hypervisor where switching is based on twelve-tuple matching onflows, the virtual switch, or virtual router software-based solution, is CPU-intensive. This can negatively affect system performance and prevent the full utilization of available bandwidth.

Mellanox ASAP2 (Accelerated Switching and Packet Processing) technology enables offloading the vSwitch/vRouter by handling the data plane in the NIC hardware, without modifying the control plane. This results in significantly higher vSwitch/vRouter performance without the associated CPU load.

Storage Environments

NVMe storage devices are gaining popularity by offering very fast storage access. The evolving NVMe over Fabric (NVMe-oF) protocol leverages the RDMA connectivity for remote access. ConnectX-5 offers further enhancements by providing NVMe-oF target offloads, enabling very efficient NVMe storage access with no CPU intervention, and thus improving performance and reducing  latency.

The embedded PCIe switch enables customers to build standalone storage or Machine Learning appliances. As with earlier generations of ConnectX adapters, standard block and file access protocols leverage RoCE for high-performance storage access. A consolidated compute and storage network achieves significant cost-performance advantages over multi-fabric networks.

ConnectX-5 enables an innovative storage rack design, Host Chaining, which enables different servers to interconnect without involving the top-of-rack switch. Leveraging Host Chaining, ConnectX-5 lowers the data center’s total cost of ownership by reducing CAPEX (cables, NICs, and switch port expenses). OPEX is also reduced by cutting down on switch port management and overall power usage.

Telecommunications

Telecommunications service providers are moving towards disaggregation, server virtualization, and orchestration as key tenets to modernize their networks. Likewise, they’re also moving towards Network Function Virtualization (NFV), which enables the rapid deployment of new network services. With this move, proprietary dedicated hardware and software, which tend to be static and difficult to scale, are being replaced with virtual machines running on commercial off-the-shelf (COTS) servers.

For telecom service providers, choosing the right networking hardware is critical to achieving a cloud-native NFV solution that is agile, reliable, fast and efficient. Telco service providers typically leverage virtualization and cloud technologies to better achieve agile service delivery and efficient scalability; these technologies require an advanced network infrastructure to support higher rates of packet processing. However, the resultant east-west traffic causes numerous interrupts as I/O traverses from kernel to user space, eats up CPU cycles and decreases packet performance. Particularly sensitive to delays are voice and video applications which often require less than 100ms of latency.

ConnectX-5 adapter cards drive extremely high packet rates, increased throughput and drive higher network efficiency through the following technologies; Open vSwitch Offloads (OvS), OvS over DPDK or ASAP2, Network Overlay Virtualization, SR-IOV, and RDMA. This allows for secure data delivery through higher-performance offloads, reducing CPU resource utilization, and boosting data center infrastructure efficiency. The result is a much more responsive and agile network capable of rapidly deploying network  services.

Physical adapter:

• PCIe 3.0 x8 host interface
• Low profile form factor (142mm length, 69mm height)
• Two SFP28 cages

Ethernet:

• 25GbE / 10GbE
• IEEE 802.3bj, 802.3bm 25 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium 25 Gigabit
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3ad Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau Congestion Notification
• IEEE 802.1Qaz D0.2 ETS
• IEEE 802.1Qbb D1.0 Priority-based Flow Control
• IEEE 1588v2
• Jumbo frame support (9600 byte)
• IPv4 (RFQ 791)
• IPv6 (RFC 2460)

Enhanced Features:

• Hardware-based reliable transport
• Collective operations offloads
• Vector collective operations offloads
• Mellanox PeerDirect RDMA (aka GPUDirect) communication acceleration
• 64/66 encoding
• Extended Reliable Connected transport (XRC)
• Dynamically Connected Transport (DCT)
• Enhanced Atomic operations
• Advanced memory mapping support, allowing user mode registration and remapping of memory (UMR)
• On demand paging (ODP)
• MPI Tag Matching
• Rendezvous protocol offload
• Out-of-order RDMA supporting Adaptive Routing
• Burst buffer offload
• In-Network Memory registration-free RDMA memory access

CPU Offloads:

• RDMA over Converged Ethernet (RoCE)
• TCP/UDP/IP stateless offload
• LSO, LRO, checksum offload
• RSS (also on encapsulated packet), TSS, HDS, VLAN and MPLS tag insertion/stripping, Receive flow steering
• Data Plane Development Kit (DPDK) for kernel bypass applications
• Open VSwitch (OVS) offload using ASAP2
  • Flexible match-action flow tables
  • Tunneling encapsulation/ de-capsulation
• Intelligent interrupt coalescence
• Header rewrite supporting hardware offload of NAT router

Hardware-Based I/O Virtualization - Mellanox ASAP²:

• Single Root IOV
• Address translation and protection
• VMware NetQueue support
  • SR-IOV: Up to 512 Virtual Functions
  • SR-IOV: Up to 8 Physical Functions per host
• Virtualization hierarchies (e.g., NPAR when enabled)
  • Virtualizing Physical Functions on a physical port
  • SR-IOV on every Physical Function
• Configurable and user-programmable QoS
• Guaranteed QoS for VMs

Management and Control:

• NC-SI over MCTP over SMBus and NC-SI over MCTP over PCIe - Baseboard Management Controller interface
• PLDM for Monitor and Control DSP0248
• PLDM for Firmware Update DSP0267
• SDN management interface for managing the eSwitch
• I2C interface for device control and configuration
• General Purpose I/O pins
• SPI interface to Flash
• JTAG IEEE 1149.1 and IEEE 1149.6

Remote Boot:

• Remote boot over Ethernet
• Remote boot over iSCSI
• Unified Extensible Firmware Interface (UEFI)
• Pre-execution Environment (PXE)

The following tables list the ThinkSystem servers that are compatible.

The following table indicates which operating systems can be preloaded in the Lenovo factory for CTO server orders where this adapter is included in the server configuration.

The adapter supports the operating systems listed in the following table.

• Safety: CB / cTUVus / CE
• EMC: CE / FCC / VCCI / ICES / RCM
• RoHS compliant

Power consumption:

• Typical power: 9.5W (passive cable)
• Maximum power: 11.8 W (passive cable), 15.1W (1.5W active cable)

Maximum power through external connectors: 1.5W

Temperature:

• Operational 0°C to 55°C
• Non-operational -40°C to 70°C

Humidity: 90% relative humidity

One year limited warranty. When installed in a Lenovo server, this adapter assumes the server’s base warranty and any warranty upgrades.

For more information, refer to these documents:

• Networking Options for ThinkSystem Servers:
  https://lenovopress.com/lp0765-networking-options-for-thinksystem-servers
• ServerProven compatibility
  http://www.lenovo.com/us/en/serverproven
• Mellanox product page for ConnectX-5 EN
  https://www.mellanox.com/products/ethernet-adapters/connectx-5-en
• User Manual for ConnectX-5 EN
  https://docs.mellanox.com/display/ConnectX5EN