Top
Author
Updated
6 Sep 2024Form Number
LP1692PDF size
17 pages, 218 KBSubscribed to LP1692.
Thank you for your feedback.
Table of Contents
Abstract
The ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand/400GbE OSFP Adapters offer 400 Gb/s connectivity for high-performance connectivity when running HPC, cloud, storage and machine learning applications.
This product guide provides essential presales information to understand the ConnectX-7 NDR adapters and their key features, specifications, and compatibility. This guide is intended for technical specialists, sales specialists, sales engineers, IT architects, and other IT professionals who want to learn more about the ConnectX-7 NDR adapters and consider their use IT solutions.
Change History
Changes in the September 6, 2024 update:
- Added the following adapter:
- ThinkSystem NVIDIA ConnectX-7 NDR400 OSFP 1-port PCIe Gen5 VPI Adapter, 4XC7A95508
Introduction
The ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand/400GbE OSFP Adapters offer 400 Gb/s connectivity for high-performance connectivity when running HPC, cloud, storage and machine learning applications. Adapters support either NDR InfiniBand only, or both NDR InfiniBand and 400Gb Ethernet, depending on the adapter.
The following figure shows the ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (the standard heat sink has been removed in this photo).
Figure 1. ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (4XC7A80289)
Did you know?
The ConnectX-7 NDR adapters are optimized to deliver accelerated networking for modern cloud, artificial intelligence, and traditional enterprise workloads. ConnectX-7 provides a broad set of software-defined, hardware-accelerated networking, storage, and security capabilities which enable organizations to modernize and secure their IT infrastructures.
Part number information
The following table shows the part numbers for the adapters.
Non-DWC adapter part numbers includes the following:
- One NVIDIA adapter with full-height (3U) adapter bracket attached
- Low-profile (2U) adapter bracket
- Documentation
Supported transceivers and cables
The ConnectX-7 NDR adapters have an empty OSFP400 cage for connectivity. The following table lists the supported transceivers.
The following table lists the supported transceivers.
The following table lists the supported optical cables.
The following table lists the supported direct-attach copper (DAC) cables.
Features
The adapters have the following features:
- Accelerated Networking and Security
ConnectX-7 provides a broad set of software-defined, hardware-accelerated networking, storage, and security capabilities which enable organizations to modernize and secure their IT infrastructures. Moreover, ConnectX-7 empowers agile and high-performance solutions from edge to core data centers to clouds, all while enhancing network security and reducing the total cost of ownership.
- Accelerate Data-Driven Scientific Computing
ConnectX-7 provides ultra-low latency, extreme throughput, and innovative NVIDIA In-Network Computing engines to deliver the acceleration, scalability, and feature- rich technology needed for today’s modern scientific computing workloads.
- Accelerate Software- Defined Networking
NVIDIA ASAP2 technology accelerates software-defined networking, delivering line-rate performance with no CPU penalty.
- Enhance Storage Performance
ConnectX-7 enables high- performance and efficient data storage by leveraging RDMA/RoCE, GPUDirect Storage, and hardware-based NVMe-oF offload engines.
- SharedIO (4XC7A86670 only)
SharedIO (Shared I/O or Multi-Host) is an implementation of Sockets Direct that is offered in Lenovo direct water-cooled (DWC) servers such as the SD650 V3, where there are two server nodes per DWC tray. In this implementation, the ConnectX-7 adapter is installed in a PCIe slot in one node and the adapter is connected via a cable to a PCIe connector on the second node. The result is that the two nodes share the network connection of the adapter with significant savings both in the cost of the adapters but also the cost of switch ports.
Technical specifications
The adapters have the following technical specifications:
- Networking interfaces
- One OSPF400 cage for 400 Gb/s connectivity
- InfiniBand connectivity
- Support for InfiniBand NDR
- Supports NDR 400Gb/s, HDR 200Gb/s, EDR 100Gb/s
- InfiniBand Trade Association Spec 1.5 compliant
- RDMA, send/receive semantics
- 16 million input/output (IO) channels
- 256 to 4Kbyte maximum transmission unit (MTU), 2Gbyte messages
- Enhanced InfiniBand Networking
- Hardware-based reliable transport
- Extended Reliable Connected (XRC)
- Dynamically Connected Transport (DCT)
- GPUDirect RDMA
- GPUDirect Storage
- Adaptive routing support
- Enhanced atomic operations
- Advanced memory mapping, allowing user mode registration (UMR)
- On-demand paging (ODP), including registration-free RDMA memory access
- Enhanced congestion control
- Burst buffer offload
- Single root IO virtualization (SR-IOV)
- Optimized for HPC software libraries including: NVIDIA HPC-X, UCX, UCC, NCCL, OpenMPI, MVAPICH, MPICH, OpenSHMEM, PGAS
- Collective operations offloads
- Support for NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)
- Rendezvous protocol offload
- In-network on-board memory
- Ethernet connectivity (only 4XC7A95508 - see Table 1)
- Support for 400Gb Ethernet
- Supports 400GbE, 200GbE, 100GbE, 50GbE, 25GbE, 10GbE
- Support for NRZ, PAM4 (50G and 100G)
- RDMA over Converged Ethernet (RoCE)
- Enhanced Ethernet Networking (only 4XC7A95508)
- Zero-Touch RoCE
- ASAP² Accelerated Switch and Packet Processing™ for SDN and VNF acceleration
- Single Root I/O Virtualization (SR-IOV)
- VirtIO acceleration
- Overlay network acceleration: VXLAN, GENEVE, NVGRE
- Programmable flexible parser
- Connection tracking (L4 firewall)
- Flow mirroring, sampling and statistics
- Header rewrite
- Hierarchical QoS
- Stateless TCP offloads
- Storage Accelerations
- Block-level encryption: XTS-AES 256/512-bit key
- NVMe over Fabrics (NVMe-oF)
- NVMe over TCP (NVMe/TCP)
- T10 Data Integrity Field (T10-DIF) signature handover
- SRP, iSER, NFS over RDMA, SMB Direct
- Management and Control
- NC-SI, MCTP over SMBus, and MCTP over PCIe
- PLDM for Monitor and Control DSP0248
- PLDM for Firmware Update DSP0267
- PLDM for Redfish Device Enablement DSP0218
- PLDM for FRU DSP0257
- SPDM DSP0274
- Serial Peripheral Interface (SPI) to flash
- JTAG IEEE 1149.1 and IEEE 1149.6
- Remote Boot
- Remote boot over InfiniBand
- Remote boot over iSCSI
- UEFI
- PXE over InfiniBand
- Cybersecurity
- Platform security: secure boot with hardware root-of-trust, secure firmware update, flash encryption, and device attestation
- PCI Express Interface
- PCIe Gen 5.0 x16 host interface
- Support for PCIe bifurcation
- SharedIO (NVIDIA Multi-Host) supports connection of up to two hosts (on supported water-cooled nodes only)
- Transaction layer packet (TLP) processing hints (TPH)
- PCIe switch Downstream Port Containment (DPC)
- Support for MSI/MSI-X mechanisms
- Advanced error reporting (AER)
- Access Control Service (ACS) for peer-to-peer secure communication
- Process Address Space ID (PASID)
- Address translation services (ATS)
- Support for SR-IOV
NVIDIA Unified Fabric Manager
NVIDIA Unified Fabric Manager (UFM) is InfiniBand networking management software that combines enhanced, real-time network telemetry with fabric visibility and control to support scale-out InfiniBand data centers.
The two offerings available from Lenovo are as follows:
- UFM Telemetry for Real-Time Monitoring
The UFM Telemetry platform provides network validation tools to monitor network performance and conditions, capturing and streaming rich real-time network telemetry information, application workload usage, and system configuration to an on-premises or cloud-based database for further analysis.
- UFM Enterprise for Fabric Visibility and Control
The UFM Enterprise platform combines the benefits of UFM Telemetry with enhanced network monitoring and management. It performs automated network discovery and provisioning, traffic monitoring, and congestion discovery. It also enables job schedule provisioning and integrates with industry-leading job schedulers and cloud and cluster managers, including Slurm and Platform Load Sharing Facility (LSF).
The following table lists the subscription licenses available from Lenovo.
For more information, see the following web page:
https://www.nvidia.com/en-us/networking/infiniband/ufm/
Server support
The following tables list the ThinkSystem servers that are compatible.
Operating system support
The adapters support the operating systems listed in the following tables.
- ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter, 4XC7A80289
- ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter DWC, 4XC7A86671
- ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (SharedIO) DWC, 4XC7A86670
Tip: These tables are automatically generated based on data from Lenovo ServerProven.
Regulatory approvals
The adapters have the following regulatory approvals:
- Safety: CB / cTUVus / CE
- EMC: CE / FCC / VCCI / ICES / RCM / KC
- RoHS: RoHS Compliant
Operating environment
The adapters have the following operating characteristics:
- Maximum power available through OSFP port: 17W
- Temperature
- Operational: 0°C to 55°C
- Non-operational: -40°C to 70°C
- Humidity: 90% relative humidity
Warranty
One year limited warranty. When installed in a Lenovo server, the adapter assumes the server’s base warranty and any warranty upgrades.
Related publications
For more information, refer to these documents:
- Networking Options for ThinkSystem Servers:
https://lenovopress.com/lp0765-networking-options-for-thinksystem-servers - ServerProven compatibility:
http://www.lenovo.com/us/en/serverproven - NVIDIA InfiniBand product page:
https://www.nvidia.com/en-us/networking/infiniband-adapters/ - ConnectX-7 user manual:
https://docs.nvidia.com/networking/display/ConnectX7VPI/Specifications
Trademarks
Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.
The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
ServerProven®
ThinkSystem®
The following terms are trademarks of other companies:
AMD is a trademark of Advanced Micro Devices, Inc.
Intel® and Xeon® are trademarks of Intel Corporation or its subsidiaries.
Linux® is the trademark of Linus Torvalds in the U.S. and other countries.
Microsoft®, Windows Server®, and Windows® are trademarks of Microsoft Corporation in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Configure and Buy
Please select a locale
Full Change History
Changes in the September 6, 2024 update:
- Added the following adapter:
- ThinkSystem NVIDIA ConnectX-7 NDR400 OSFP 1-port PCIe Gen5 VPI Adapter, 4XC7A95508
Changes in the August 16, 2024 update:
- The adapters support PXE boot support over InfiniBand - Technical specifications section
Changes in the December 6, 2023 update:
- Removed the restriction that these adapters were only orderable in DCSC using the HPC & AI mode of the configurator. Now orderable via the General Purpose mode.
Changes in the September 29, 2023 update:
- The adapters support secure boot, however they do not include embedded cryptography functions - Technical specifications section
First published: May 2, 2023
Course Detail
Employees Only Content
The content in this document with a is only visible to employees who are logged in. Logon using your Lenovo ITcode and password via Lenovo single-signon (SSO).
The author of the document has determined that this content is classified as Lenovo Internal and should not be normally be made available to people who are not employees or contractors. This includes partners, customers, and competitors. The reasons may vary and you should reach out to the authors of the document for clarification, if needed. Be cautious about sharing this content with others as it may contain sensitive information.
Any visitor to the Lenovo Press web site who is not logged on will not be able to see this employee-only content. This content is excluded from search engine indexes and will not appear in any search results.
For all users, including logged-in employees, this employee-only content does not appear in the PDF version of this document.
This functionality is cookie based. The web site will normally remember your login state between browser sessions, however, if you clear cookies at the end of a session or work in an Incognito/Private browser window, then you will need to log in each time.
If you have any questions about this feature of the Lenovo Press web, please email David Watts at dwatts@lenovo.com.
