Setting up an NFSoRDMA share for use with hipFile

Setting up an NFSoRDMA share for use with hipFile#

The following prerequisites are required to use an RDMA-capable NFS server over RDMA or RoCE with hipFile:

  • amdgpu-dkms version 31.40 or later.

  • An RDMA or RoCE fabric connecting the NFS server and client.

  • nfs-kernel-server installed on the server, with the svcrdma kernel module available.

  • nfs-common installed on the client, with the xprtrdma kernel module available.

Configure the NFS server#

Configure a static IP address on the RDMA or RoCE interface.

Note

The example commands use netplan. Use the appropriate distribution network configuration tools.

Create /etc/netplan/config.yaml, replacing <rdma_interface> and <server_ip>/<prefix> with values that match the fabric, then run netplan apply:

network:
  version: 2
  renderer: networkd
  ethernets:
    <rdma_interface>:
      dhcp4: false
      dhcp6: false
      accept-ra: false
      addresses:
        - <server_ip>/<prefix>

sudo netplan apply

Load the RDMA-capable NFS server module:

sudo modprobe svcrdma

Add the directories to export to /etc/exports, where EXPORT_PATH is the path to each directory to export:

EXPORT_PATH *(rw,async,insecure,no_root_squash)

Reload the NFS server to pick up the new exported files:

sudo systemctl reload nfs-server

Add an RDMA port for NFS to listen on:

echo rdma 20049 | sudo tee /proc/fs/nfsd/portlist

Confirm that the RDMA port is active:

$ cat /proc/fs/nfsd/portlist
rdma 20049
tcp 2049

Configure the client#

Load the RDMA-capable NFS client module:

sudo modprobe xprtrdma

Create a mount point and mount the share using RDMA:

sudo mkdir /mnt/nfs
sudo mount -o rdma,port=20049 <server_ip>:<export_path> /mnt/nfs

Set HIPFILE_UNSUPPORTED_FILE_SYSTEMS=true before running hipFile I/O.

Verify that hipFile has access to the mounted share by copying a file using aiscp.cpp:

sudo mkdir /mnt/nfs/"${USER}"
sudo chown "${USER}":"${USER}" /mnt/nfs/"${USER}"
# Create a random input file
dd if=/dev/urandom of=/mnt/nfs/"${USER}"/source bs=4K count=16
# Copy file
HIPFILE_ALLOW_COMPAT_MODE=false HIPFILE_UNSUPPORTED_FILE_SYSTEMS=true ./aiscp /mnt/nfs/"${USER}"/source /mnt/nfs/"${USER}"/dest
md5sum /mnt/nfs/"${USER}"/source /mnt/nfs/"${USER}"/dest

Use the Linux umount command to unmount the share when it’s no longer needed:

sudo umount /mnt/nfs
Contents