MORI network configuration file reference for Cluster Validation Suite (CVS)#
2026-09-24
6 min read time
MORI (Memory-Oriented RDMA Interface) tests validate RDMA communication performance across multi-node AMD GPU clusters. These tests ensure optimal bandwidth, latency, and reliability for distributed workloads that require high-speed inter-node communication.
The MORI tests check:
Container orchestration: Docker setup with MORI libraries for RDMA communication
RDMA device configuration: Proper setup of RDMA devices and network interfaces
Read/Write operations: Bandwidth and latency metrics for RDMA read and write operations
Network interface types: Support for various NICs (AINIC, Thor2, CX7)
Multi-node coordination: Master-worker communication and synchronization
Result verification: Expected bandwidth and latency thresholds
Change the parameters as needed in the MORI configuration file: mori_config.json for multi-node RDMA testing.
Note
{user-id}resolves to the current username at runtime. You can also manually change this value to your username.Replace all
<changeme>placeholders with actual values for your cluster.
mori_config.json#
Here’s a code snippet of the mori_config.json file for reference:
mori_config.json
{
"no_of_nodes": "2",
"container_image": "<changeme>",
"container_name": "mori_container",
"oob_port": "<changeme>",
"mori_device_list": "<changeme>",
"mori_dir": "<changeme>",
"torchlib_dir": "/usr/local/lib/python3.12/dist-packages/torch/lib",
"master_addr": "<changeme>",
"master_port": "1234",
"nic_type": "<changeme>",
"log_dir": "/home/{user-id}/LOGS/mori",
"container_config": {
"device_list": [ "/dev/dri", "/dev/kfd" ],
"volume_dict": {
"/home/{user-id}": "/home/{user-id}",
"/it-share/models": "/root/models",
"/usr/lib/x86_64-linux-gnu/libionic.so.1.0.54.0-164.g21c72dcad":
"/usr/lib/x86_64-linux-gnu/libionic.so.1.0.54.0-164.g21c72dcad",
"/usr/lib/x86_64-linux-gnu/libionic.so.1":
"/usr/lib/x86_64-linux-gnu/libionic.so.1",
"/usr/lib/x86_64-linux-gnu/libionic.so":
"/usr/lib/x86_64-linux-gnu/libionic.so",
"/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so":
"/usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so",
"/etc/libibverbs.d/ionic.driver":
"/etc/libibverbs.d/ionic.driver"
},
"env_dict": {}
},
"expected_results": {
"read": {
"16384,128,1": {
"524288": {
"max_bw": "45.0",
"avg_lat": "1500"
},
"1048576": {
"max_bw": "46.0",
"avg_lat": "2500"
}
}
},
"write": {
"128": {
"524288": {
"max_bw": "45.0",
"avg_lat": "1500"
},
"1048576": {
"max_bw": "46.0",
"avg_lat": "2500"
}
}
}
}
}
Parameters#
Use the parameters in this table to configure the MORI configuration file.
Configuration parameters |
Default values |
Description |
|---|---|---|
|
2 |
Number of nodes in the MORI test cluster |
|
|
Docker container image with MORI libraries for RDMA testing (e.g., rocm/sgl-dev:sglang-0.5.6.post1-rocm700-mi35x-mori-1224) |
|
mori_container |
Name of the Docker container instance |
|
|
Out-of-band network interface for control plane communication (e.g., eno0, eno1, enp0s0) |
|
|
Comma-separated list of RDMA devices to use for testing (e.g., rdma0,rdma1,rdma2,rdma3,rdma4,rdma5,rdma6,rdma7) |
|
|
Directory containing MORI binaries and libraries (e.g., /sgl-workspace/mori) |
|
/usr/local/lib/python3.12/dist-packages/torch/lib |
Directory containing PyTorch libraries (for dependencies) |
|
|
IP address or hostname of the master node |
|
1234 |
Port for master-worker communication |
|
|
Network interface card type: ainic (AMD Pensando AINIC), thor2 (Thor2 adapter), or cx7 (Mellanox ConnectX-7) |
|
|
Directory for MORI test logs |
|
Values: |
List of device paths to mount in the container for GPU access |
|
Multiple mappings |
Dictionary mapping host paths to container paths for volume mounts |
|
|
User home directory mount |
|
|
Models directory mount |
|
|
AMD Pensando AINIC library (specific version) |
|
|
AMD Pensando AINIC library (versioned symlink) |
|
|
AMD Pensando AINIC library (unversioned symlink) |
|
|
RDMA verbs provider for AMD Pensando AINIC |
|
|
Driver configuration file for AMD Pensando NICs |
|
Empty |
Dictionary of environment variables to set in the container |
|
Nested dictionary |
Expected results for RDMA read operations with format: |
|
45.0 |
Expected maximum bandwidth (GB/s) for read operation with 16384 threads, block size 128, 1 block, and 524288 byte transfer |
|
1500 |
Expected average latency (microseconds) for read operation with 16384 threads, block size 128, 1 block, and 524288 byte transfer |
|
46.0 |
Expected maximum bandwidth (GB/s) for read operation with 16384 threads, block size 128, 1 block, and 1048576 byte transfer |
|
2500 |
Expected average latency (microseconds) for read operation with 16384 threads, block size 128, 1 block, and 1048576 byte transfer |
|
Nested dictionary |
Expected results for RDMA write operations with format: |
|
45.0 |
Expected maximum bandwidth (GB/s) for write operation with 128 threads and 524288 byte transfer |
|
1500 |
Expected average latency (microseconds) for write operation with 128 threads and 524288 byte transfer |
|
46.0 |
Expected maximum bandwidth (GB/s) for write operation with 128 threads and 1048576 byte transfer |
|
2500 |
Expected average latency (microseconds) for write operation with 128 threads and 1048576 byte transfer |