Page History

...

In this manner we separate the functionality of defining fail-over pairs from defining fail-over networks.

Preferred NIDs

Depending on the network topology which the Lustre network is built on, it might be necessary to assign priorities to specific interfaces which are connected to optimized paths. In this way messages don't take more hops than necessary to get to the destination.  Gliffy DiagramnamePreferredNID

In the above diagram you can have a set of clients on the OPA network and a set of clients on the MLX network. The servers have both OPA and MLX interfaces. The OPA clients are connected on o2ib1 and MLX clients are connected on o2ib0. But there also exists a route to route between MLX and OPA. In the above scenario you might want to prefer the green path to avoid an extra hop through the router.

a scenario where servers are being upgraded with new interfaces to be used in Multi-Rail, it's possible to add interfaces, for example MLX-EDR interfaces to the server. The user might want to continue making the existing QDR clients use the QDR interface, while new clients can use the EDR interface or even both interfaces. By specifying rules on the clients that prefer a specific interface this behavior can be achieved.

Preferred local/remote NID pairs

...

1. determine the best network to communicate to the destination peer by looking at all the LNet networks the peer is on.
   1. select the network with the highest priority
2. for each selected network go through all the local NIs and keep track of the best_ni based on:
   1. It's priority
   2. NUMA distance
   3. available credits
   4. round robin
3. As you visit each network select the best_ni from the network with the highest priority. Skip any networks which are lower priority than the "active" one. If there are multiple networks with the same priority then the best_ni is selected from amongst them using the stated above criteria.
4. Once the best_ni has been selected, select the best peer_ni available by going through the list of the peer_nis on the selected network.  Select the peer_ni based on:
   1. The priority of the peer_ni.
   2. if the NID of the best_ni is on the preferred local NID list of the peer_ni. It is placed there through the application of the peer to peer rules.
   3. available credits
   4. round robin

...