NvidiaMellanoxOFED驱动解决方案

简介

这篇文章显示了 Mellanox Linux 驱动程序解决方案的相关文档及链接

Linux Driver

• 快速开始
  • Getting started with ConnectX-4 100Gb/s Adapter for Linux
  • Getting Started with ConnectX-5 100Gb/s Adapters for Linux
  • Mellanox Adapters - Comparison Table 网卡比较表

  Class	Feature	ConnectX-3	ConnectX-3 Pro	ConnectX-4	ConnectX-4 Lx	ConnectX-5	ConnectX-6	References and Notes
  Interface	Port/Speed options	2 ports of 10/40/56GbE	2 ports of 10/40/56GbE	2 ports of 100/56/50/40/25/10GbE	2 ports of 10/25GbE	2 ports of 100/56/50/40/25/10GbE	2 ports of 200/100/56/50/40/25/10GbE	Note: There various of adapter cards. Some OPNs for example could be speed limited, e.g.ConnectX-4 FDR (that won’t link up EDR) or Connect-IB that is x8 only and not x16.
  					1 port of 40/50GbE
  Interface	PCIe	x8 Gen3	x8 Gen3	x8, x16 Gen3	x8 Gen3	x16 Gen3	2x Gen3 x16
  						x16 Gen4	Gen4 x16
  Interface	PCIe Switch					PCIe x16	PCIe x32
  Interface	Coherent Accelerator Processor Interface (CAPI)	-	-	-	-	Supported (v2)	Supported (v2)	IBM - Coherent Accelerator Processor Interface (CAPI)
  								http://openpowerfoundation.org/blogs/interconnect-your-future-mellanox-100gb-edr-capi-infiniband-and-interconnects/
  IB RDMA / RoCE	IB RDMA / RoCE	IB RDMA, RoCE*	IB RDMA, RoCE	IB RDMA, RoCE	RoCE	IB RDMA, RoCE	IB RDMA, RoCE	RDMA/RoCE Solutions
  RoCE	RoCE Congestion Control	-	Supported	Supported	Supported	Supported	Supported	RDMA/RoCE Solutions
  								Understanding RoCEv2 Congestion Management
  								RoCEv2 CNP Packet Format Example
  								HowTo Configure RoCE Congestion Control for Windows 2012
  HPC	CORE-Direct®	Supported	Supported	Supported	Supported	Supported	Supported	Exploited by FCA and extended by Mellanox SHARP technology.
  								http://www.mellanox.com/related-docs/whitepapers/TB_CORE-Direct.pdf
  HPC	PeerDirect®	Supported	Supported	Supported	Supported	Supported	Supported	GPU Direct Uses PeerDirect.
  								http://www.mellanox.com/page/products_dyn?product_family=116
  HPC	Dynamically Connected Transport (DCT)	-	-	Supported	Supported	Supported	Supported	http://www.mellanox.com/related-docs/applications/SB_Connect-IB.pdf
  CPU Offloads	Stateless Ethernet offload	Supported	Supported	Supported + LRO, LSOv2	Supported + LRO, LSOv2	Supported + LRO, LSOv2	Supported + LRO, LSOv2	LRO = Large Receive Offload.
  								LSO = Large Send offload.
  								See MLNX_OFED User Manual for more info.
  CPU Offloads	RSS (MAC, VLAN 5 Tuple)	Supported	Supported	Supported +	Supported +	Supported +	Supported +
  Virtualization	SR-IOV	Supported	Supported	Supported	Supported	Supported	Supported	Virtualization Solutions
  Virtualization	Multi Host	-	-	4 hosts	4 hosts	4 hosts	8 hosts	http://www.mellanox.com/page/products_dyn?product_family=210&mtag=multihost
  Open V-Switch (OVS)	OVS Offload	-	-	-	Supported	Supported +	Supported +	Describes whether the adapters can run the OVS data-plane in HW. Supported means that at least matching for a specific flow is performed in HW. Supported+ means that HW Encap/Decap is also included.
  								See http://www.mellanox.com/page/asap2.
  Overlay Network	Stateless offload	-	Supported	VXLAN / NVGRE / GENEVE	VXLAN / NVGRE / GENEVE	VXLAN / NVGRE / GENEVE and FlexParse	VXLAN / NVGRE / GENEVE and FlexParse	Virtualization Solutions
  Overlay Network	Encap/Decap (in HW)	-	-	-	Supported	Supported	Supported
  (VXLAN/NVGRE)
  Storage	Erasure Coding Offload	-	-	Supported	Supported	Supported	Supported	Reed Solomon Erasure Coding hardware offload is supported by the adapters.
  								Verbs API are available.
  								Understanding Erasure Coding Offload
  Storage	T10/DIF Signature Handover	-	-	Supported	-	Supported	Supported	HowTo Enable T10-PI (T10-DIF) Data Integrity Protection in iSER with LIO Target
  Storage	NVMe oF Target Offload	-	-	-	-	Supported	Supported +
  	(also for Burst Buffer)
  Storage	Host Chaining	-	-	-	-	Supported	Supported
  Cloud Integration	Mirantis Fuel	Supported Fuel 7.0/8.0	Supported Fuel 7.0/8.0	Supported Fuel 8.0	Supported Fuel 8.0	Supported	Supported	Cloud Solutions
  Media & Entertainment	Packet Pacing	-	-	Supported	Supported	Supported	Supported
  Security	Secure Firmware update	-	-	Supported	Supported	Supported	Supported
  Security	Secure Boot	-	-	-	-	-	Supported

* Partial support 部分支持

• Mellanox Linux Driver Modules Relationship (MLNX_OFED) Mellanox Linux 驱动程序模块关系 (MLNX_OFED)

HowTo Install MLNX_OFED Driver 如何安装MLNX_OFED驱动

• 重新编译OFED/rebuild ofed: https://enterprise-support.nvidia.com/s/article/howto-compile-mlnx-ofed-drivers--160---mlnx-ofa-kernel-example-x 编译其他系统的MLNX_OFED: https://enterprise-support.nvidia.com/s/article/howto-compile-mlnx-ofed-for-different-linux-kernel-distribution--160---160-x
• HowTo Install MLNX_OFED for ConnectX-4/ConnectX-5 via ISO Image
• Ethtool Commands
• How To Extract MLNX_OFED Source Files

• Port
  • HowTo Change Port Type in Mellanox ConnectX-3 Adapter
  • HowTo Find the Logical-to-Physical Port Mapping (Linux)
  • HowTo Change the Ethernet Port Speed of Mellanox Adapters (Linux)
  • HowTo Enable LLDP on Linux Servers for Link Discovery
  • HowTo Find Cable Info on Mellanox Adapters and Switches
• QoS
  • HowTo Set Egress Priority VLAN on Linux
  • HowTo Configure Packet Pacing on ConnectX-4
  • HowTo Auto-Config PFC and ETS on ConnectX-4 via LLDP DCBX
  • Default ToS to skprio mapping on Linux
  • HowTo Tune Receive Buffers on Mellanox Adapter Cards
• Maintenance and Monitoring 监控和维护
  • HowTo Enable SNMP Server (snmpd) on Linux
  • HowTo Find Mellanox Adapter Type and Firmware/Driver version (Linux)
  • HowTo Compile MLNX_OFED for Different Linux Kernel Distribution
  • HowTo manage Mellanox Linux Driver modules and RPMs
  • HowTo Dump Driver Configuration Using Ethtool
  • How-To Dump RDMA traffic Using the Inbox tcpdump tool (ConnectX-4)
• Counters 计数器
  • Counters Troubleshooting for Linux Driver
  • MLNX_OFED ConnectX-3 RDMA Diagnostic Counters
  • Understanding mlx5 ethtool Counters 理解mlx5的以太计数器
  • Understanding mlx5 Linux Counters and Status Parameters
  • Understanding mlx4 Linux Counters
• Time Synchronization 时间同步
  • Running TimeKeeper with ConnectX-4
  • Running Linux PTP with ConnectX-4/ConnectX-5
  • Running PTPd with ConnectX-4
  • HowTo Steer PTP traffic to single RX ring (via ethtool)
  • HowTo Test 1PPS on Mellanox Adapters
• Flow Steering 流量控制
  • Understanding Linux debug-fs Flow Steering rules
• Offload Features
  • How To Enable Large Receive Offload (LRO)
• Tools
  • ibdev2netdev 物理接口到逻辑接口的映射
  • mlx_fs_dump
  • mlnx_qos
• Other Features
  • MLNX_OFED driver configuration file: openib.conf
  • HowTo Increase Memory Size used by Mellanox Adapters
  • HowTo Configure aRFS on ConnectX-4
  • HowTo Implement PeerDirect Client using MLNX_OFED

Firmware Tools 固件工具

• Getting started with Mellanox Firmware tools (MFT) for Linux
• mlxlink

PXE(Pre-boot Execution Environment，预启动执行环境)

• Expansion ROM for BIOS and UEFI on Mellanox cards
• HowTo Configure PXE Server in UEFI Mode for IPoIB Network

For other Linux related topics refer to: 其他技术主题

• Virtualization Solutions 虚拟化解决方案
• RDMA/RoCE Solutions RoCE解决方案
• Performance Tuning for Mellanox Adapters MLX网卡性能调优
• Programming Solutions 编程示例/ODP/单边/双边操作等

总结

• 本文总结了NvidiaMLX驱动相关的文章和链接, 后期可基于此索引页面更新
• 对于RoCE协议, 可抓包分析, 如可通过自己编译驱动源码中的ibdump对MLX5系列网卡进行抓包, make WITHOUT_FW_TOOLS=yes [install]

https://github.com/Mellanox/ibdump

参考

Linux驱动解决方案: https://enterprise-support.nvidia.com/s/article/linux-driver-solutions

晓兵(ssbandjl)

博客: https://cloud.tencent.com/developer/user/5060293/articles | https://logread.cn | https://blog.csdn.net/ssbandjl | https://www.zhihu.com/people/ssbandjl/posts

DPU专栏

https://cloud.tencent.com/developer/column/101987

技术会友: 欢迎对DPU/智能网卡/卸载/网络,存储加速/安全隔离等技术感兴趣的朋友加入DPU技术交流群