...

lnetctl net add --net <net-name> --if <interface name>
# Examples
lnetctl net add --net o2ib --if ib0
lnetctl net add --net tcp --if eth0

How do I configure different

...

module parameters per network interface?

Sometimes nodes can have different types of interfaces, for example, MLX and OPA. It is desired to configure both of those with different tunables. To do that, we must use the YAML configuration.

...

1. Configure LNet and its interfaces as shown above
2. Configure a router

   1. Code Block
      # enable routing lnetctl set routing 1

3. Configure route entries on each of the nodes

   1. Code Block

      lnetctl route add --net <destination net> --gateway <gateway nid> #Examples lnetctl route add --net o2ib1 --gateway 10.10.10.2@o2ib # The above says: # any messages destined to o2ib1 should be forwarded to 10.10.10.2@o2ib # o2ib has to be a reachable network   lnetctl route add --net tcp --gateway 10.10.10.3@2o2ib --hop 2 --priority 1 # If there are multiple routes sometimes it's useful to define the priority between these routes # hop should define the number of hops to the gateway. # Unfortunately, due to legacy reasons hop and priority perform the same function # it would have been better to only have one to reduce confusion. # routes with less number of hops or with higher priority are selected first # if routes have the same number of hops and priority they are visited in round-robin.

What

...

module parameters impact routing?

A router can be configured with a set number of buffers. These buffers are used to receive messages to be forwarded

...

The pinger waits for router_ping_timeout for the gateway to respond to a ping health check. 

How can I verify my routing setup works?

What is LNet Multi-Rail?

LNet Multi-Rail allows multiple interfaces to be used for sending LNet messages. This feature boosts performance. A follow up feature, LNet Resiliency, currently being worked on is aimed at increasing resiliency.

...

For more information on the feature refer to the HLD: http://wiki.lustre.org/images/b/bb/Multi-Rail_High-Level_Design_20150119.pdf. Dynamic Behavior section.

Are there any specific consideration when configuring linux routing for LNet Multi-Rail node?

Refer to: MR Cluster Setup

Are there any specific routing consideration with Multi-Rail?

Refer to: Using Routing Resiliency with the Multi-Rail

How do I test LNet performance?

How should I setup a clustre with a combination of Multi-Rail enabled nodes and non-Multi-Rail enabled nodes? (#TODO)

What should I do if I upgrade from a non-Multi-Rail Lustre version (<=2.10 ) to a Multi-Rail Lustre version (> 2.10) with regards to Multi-Rail?

(#TODO)

Are there any specific consideration when configuring linux routing for LNet Multi-Rail node?

Refer to: MR Cluster Setup

Are there any specific routing consideration with Multi-Rail?

Refer to: Using Routing Resiliency with the Multi-Rail

How do I test LNet performance?

lnet_selftest is available for performance lnet_selftest is available for performance testing. Refer to: https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lnetselftest

...

lctl set_param debug=+net
lctl set_param debug=+neterror
 
# make sure to use an absolute path
lctl debug_daemon start /root/log
tail -f /root/log
 
# or... NOTE doesn't have to be an absolute path
lctl dk > /root/log

I can't mount the FS and I think it's a networking issue, what should I do?

1. Check that you can ping your MGS first.
2. Check that you can lnetctl ping the MGS NID.
   1. If you're able to lnetctl ping the MGS NID, then check if you can RDMA using ib_write_bw:
      1. Start ib_write_bw  on your server. This will start a receiver process
      2. Run ib_write_bw <MGS IP address> 
      3. This will run an RDMA traffic test independent of LNet. If this works then RDMA works and move on to further LNet debugging. Otherwise contact your IB service provider.
   2. Next step is to run lnet_selftest to verify LNet traffic.
      1. Look at: https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lnetselftest
   3. If lnet_selftest works, then verify your MGS is setup properly.

If ib_write_bw works, but LNet doesn't work, then check your o2iblnd configuration, as shown below.

I have a routed setup and my clients can't mount? (#TODO)

How do I check my o2iblnd configuration on the different are compatible?

1. Make sure peer_credits are the same across your nodes.
   1. peer_credits are dynamically negotiated, such that the lowest peer_credits are used. However, if it's not your intention to have different peer_credits across the different nodes, it is recommended to ensure they all have the same value.
2. Make sure your peer_credits_hiw are the same across your nodes.
   1. peer_credits_hiw define the High Water Mark value, which when reached the outstanding credits on the connection are returned using a No-op message.
3. Make sure your concurrent_sends are the same across your nodes.
   1. concurrent_sends define the number of concurrent transmits per conneciton
4. The recommended values for the above parameters are:
   1. peer_credits = 32
   2. peer_credits_hiw = 16
   3. concurrent_sends = 64
   4. Generally speaking you want peer_credits_hiw to be half of peer_credits and concurrent_sends to be two times peer_credits.
5. Make sure conns_per_peer are the same across the nodes. It defines the number of IB connections to create to 1 peer.
   1. On OPA it is recommend to set this value to 4
   2. On MLX it is recommended to leave it a at the default value of 1

What is map_on_demand and what should I set it to? (#TODO)

How should I configure LNet Health? (#TODO)

When should I turn off LNet Health? (#TODO)

When should I turn off Dynamic Discovery? (#TODO)