Enabling Infiniband and RDMA support in Linux

This article will help you enable Infiniband and RDMA support in Linux using the upstream drivers or the MLNX OFED drivers from Mellanox.

RHEL/CentOS 7

Upstream Drivers

Install the relevant packages

# yum groups install Infiniband\ Support
# yum install opensm rdma infiniband-diags

Enable subnet manager on all nodes

# systemctl enable opensm --now

NOTE: opensm only needs to run on a single node in the fabric. I’m enabling it on all nodes for redundancy. opensm will detect that other instances are running in the fabric and shut itself down.

At this point, you should have a working infiniband fabric. However, you can create TCP/IP interfaces over the Infiniband network (IPoIB) by creating configuration files in /etc/sysconfig/network-scripts similar to this:

DEVICE=ib0 
TYPE=Infiniband 
BOOTPROTO=static 
BROADCAST=192.168.0.255 
IPADDR=192.168.0.1 
NETMASK=255.255.255.0 
NETWORK=192.168.0.0 
MTU=65520
ONBOOT=yes

Mellanox (MLNX OFED)

Download and install the OFED stack from MLNX: https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed

The downloaded package should come with instructions and an installation script. Please see the README there for details.

After installed, enable the opensmd (same as opensm in upstream instructions) on all nodes:

# systemctl enable opensmd --now

You can use the same IPoIB configuration from the upstream instructions to configure TCP/IP on top of these devices as well.

Reviewed 2020/12/01 - DGT