Page History

The following assumes that underlying network connectivity has been tested (using iperf for example) as and was found to be working as expected for reliability and performance.

...

• pick another node on the same LNet as A and run lnet_selftest between A and this node (A1)
• pick another node on the same LNet as B and run lnet_selftest between B and this node (B1)
• pick another router node and run lnet_selftest beween R and the other router R1, exercise LNet1 and LNet2 nids as separate tests

LND Tuning

...

If at this point the performance results are not satisfactory, there's still a chance that certain parameter adjustments can make yield improvements. 

This section discusses SockLND LND tuning.

Current limitation is that generally all NIDs served by a particular LND on a given node shall use the same set of LND tunables. With the exception of sockLND conns_per_peer, it is currently not possible to have parameters specific to a NID, so, for examle, all "tcpX" NIDs on the same node will be configured the same even though they may be connected to different fabrics.

It is recommended that all NIDs talking to eachother over the same LNet have the same set of tunables applied.

Tuning sockLND

conns_per_peer

For peers on "tcpxtcpX" lnets, check the conns_per_peer value in the "lnetctl net show -v 4" output. Heuristically determined optimal settings are: 4 for 100Gbps link and higher, 3 for 50Gbps link, 2 for 5-10Gbps and 1 for anything less. It is possible that in some situations increasing this parameter beyond the recommended value may help improve performance. Note that this can be set per individual tcp NID using lnetctl.

nscheds            

Use "top" to check on socklnd threads while lnet_selftest (or any other test, e.g. FIO, is running). If socklnd threads are seen to be fully loaded, it may be beneficial to increase nscheds value. It makes sense to increase it to a value between conns_per_peer and (conns_per_peer x 2)

peer_credits

Default sockLND peer_credits is 8. There's a chance that increasing this and consequently credits can improve performance. As shown above, changing these affects optimal router buffer number choice.

Tuning o2ibLND

OPA

Default tunings for OPA can be found in /etc/modprobe.d/ko2iblnd.conf  and are as follows:

options ko2iblnd-opa peer_credits=128 peer_credits_hiw=64 credits=1024 concurrent_sends=256 ntx=2048 map_on_demand=32 fmr_pool_size=2048 fmr_flush_trigger=512 fmr_cache=1 conns_per_peer=4

They are applied if OPA device is detected.

MLNX

Default tunings for MLNX are hardcoded. Theyreare notable differences from OPA settings:

peer_credits: 32, peercredits_hiw: 16, concurrent_sends: 64, fmr_pool_size: 512, fmr_flush_trigger: 384, ntx: 512, conns_per_peer: 1

peer_credits

Default peer_credits can be decreased if, for example, it is seen that remote network is slower and can't keep up. As shown above, changing these affects optimal router buffer number choice.

Troubleshooting

• Check MTU size configured for the interfaces facing certain LNet. It should be matching across the LNet nodes as well as the network switch configuration.
• Check CPU utilization/core affinitization on LNet nodes, especially "clients". If LNet is sharing the cores its been assigned to use with some other process, the performance may suffer. If this is the case, cpu_pattern and cpu_npartitions libcfs parameters may need to be changed

...