...

The main difference is in the routing rules. The rules explicitly cause the route selection algorithm to look at the ib0 or ib1 routing tables based on the source prefix. Therefore any packet with a source address set to ib0 or ib1's IP address triggers the rules and is first matched against the route in the corresponding table. In this way messages are guaranteed to use the correct interface. (ib0 and ib1 are used as examples)

Testing

I conducted some testing to understand which tunable parameter avoids the address resolution error or mis-arp problem

Test procedure

1. load LNet
2. Discover both nodes
3. load lnet_selftest
4. run lnet_selftest
5. on the switch bring down one of the ports
6. stop lnet_selftest
7. on the switch bring up the port
8. look for address resolution errors and watch health statistics

Test matrix

• arp settings only:  -110 address resolution errors observed. Local interface didn't recover
• rp_filter = 0 only:  no -110 address resolution errors observed. Local interfaces recovered
  
• rp_filter = 1 only:  -110 address resolution errors observed. Local interface didn't recover
  
• rp_filter = 2 only: -110 address resolution errors observed. consumer fatal error: probably wrong HW address returned for ARP
• router rules only (delete old routes): -110 address resolution errors observed.
• router rules + rp_filter = 0:no -110 address resolution errors observed. Local interfaces recovered
• router rules + rp_filter = 1: no -110 address resolution errors observed. Local interfaces recovered
• router rules + rp_filter = 2: no -110 address resolution errors observed. Local interfaces recovered

It appears like setting rp_fileter to 0 avoids the -110 error. However, I can not conclusively say that it resolves all issues.

Adding the routing rules in addition to the rp_filter avoids the -110 address resolution error for all rp_filter settings. Therefore, it seems like route rules + rp_filter = 0 is the safest configuration to avoid issues on MR setups with interfaces on the same network.

This configuration needs to be done on all clients and servers which have multiple interfaces configured in Multi-Rail.

trevis-401

401 is the most complicated node in the cluster. It has 2 ETH, 2 OPA and 2 MLX interfaces. 

...