Purpose
This document describes the UDSP feature from the user perspective. The document will cover all Intended use cases, list expected behaviour and potential issues for each.
UDSP Use Cases
Linux Routing
Because all use cases featured below rely on having multiple interfaces configured, it is important to make sure that linux routing is configured according to guidelines listed here: MR Cluster Setup
Local Peer Config
The following configuration is used for local testing.
PeerA | PeerB |
---|---|
Two nids on tcp, one nid on tcp1 | Two nids on tcp, one nid on tcp1 |
lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.110@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.253@tcp status: up interfaces: 0: eth1 - net type: tcp1 local NI(s): - nid: 192.168.122.238@tcp1 status: up interfaces: 0: eth2 | lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.103@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.154@tcp status: up interfaces: 0: eth1 - net type: tcp1 local NI(s): - nid: 192.168.122.127@tcp1 status: up interfaces: 0: eth2 |
Prioritize Source
Local Source NID
UDSP | Behaviour |
---|---|
Prioritize one of PeerA NIDs on tcp | Prioritized nid is used when its net (tcp) is used on peerA Priority of the nid is 0. Other nids' priority is -1 (default, no priority) Every send toggles between the prioritized nid on tcp and the nid on tcp1 on the source. The remaining nid on tcp is not used "netstat -i" command shows tx stats incrementing for eth2 and eth0 on peerA |
lnetctl udsp add --src 192.168.122.110@tcp lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: NA rte: NA action: priority: 0 | PeerA: lnetctl net show -v 4 >>>> - net type: tcp local NI(s): - nid: 192.168.122.110@tcp status: up interfaces: 0: eth0 .... udsp info: net priority: -1 nid priority: 0 <<<< |
Local Source Net
UDSP | Behaviour |
---|---|
Prioritize tcp1 as source on PeerA Note that source NID rule is still in the list. | Prioritized tcp1 is used on PeerA Priority of tcp1 is 0 (highest) Every send on PeerA goes out tcp1. NID rule is overriden. "netstat -i" command shows tx stats incrementing for eth2 on peerA |
lnetctl udsp add --src tcp1 lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: NA rte: NA action: priority: 0 - idx: 1 src: tcp1 dst: NA rte: NA action: priority: 0 | PeerA: lnetctl net show -v 4 >>>> - net type: tcp local NI(s): - nid: 192.168.122.110@tcp status: up interfaces: 0: eth0 .... udsp info: net priority: -1 nid priority: 0 <<<< >>>> - net type: tcp1 local NI(s): - nid: 192.168.122.238@tcp1 status: up interfaces: 0: eth2 .... udsp info: net priority: 0 nid priority: -1 <<<< |
Prioritize Destination
Local Destination NID
UDSP | Behaviour |
---|---|
On PeerA, prioritize one of PeerB's NIDs on tcp | Prioritized NID on PeerB is sent to when tcp is selected on peerA Priority of the prioritized NID is 0 (lnetct peer show -v 4) Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID "netstat -i" command shows rx stats incrementing for eth2 and eth0 on peerB |
lnetctl udsp add --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: 192.168.122.103@tcp rte: NA action: priority: 0 | PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID PeerA: use "lnetct peer show -v 4" to verify dest NID priority. |
Local Destination Net
UDSP | Behaviour |
---|---|
On PeerA, prioritize tcp1 as dest net | Prioritized dest net (tcp1) is used to send from PeerA Priority of the prioritized dest net is 0 (lnetct peer show -v 4) Sends on PeerA use tcp1. "netstat -i" command shows tx stats incrementing for eth2 on peerA |
lnetctl udsp add --dst tcp1 lnetctl udsp show udsp: - idx: 0 src: NA dst: tcp1 rte: NA action: priority: 0 | PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net PeerA: use "lnetctl peer show -v 4" to verify dest net priority. |
NID Pairing
Local Destination NID Pairing
UDSP | Behaviour |
---|---|
On PeerA, pair one of the tcp nids with one off the PeerB tcp nids | When paired nid on PeerA is selected, it always sends to its pair on peerB PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v") |
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: 192.168.122.103@tcp rte: NA | PeerA: use "lnetctl peer show -v 4" to verify nid pairing. |
Remote Peer Config
The following configuration is used for testing that involves remote peers and routing:
PeerA | PeerB | Gateway1 | Gateway2 |
---|---|---|---|
two nids on tcp | two nids on tcp2 | two nids on tcp, two nids on tcp2 | two nids on tcp, two nids on tcp2 |
lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.103@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.154@tcp status: up interfaces: 0: eth1 lnetctl route show route: - net: tcp2 gateway: 192.168.122.132@tcp - net: tcp2 gateway: 192.168.122.102@tcp | lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp2 local NI(s): - nid: 192.168.122.110@tcp2 status: up interfaces: 0: eth0 - nid: 192.168.122.253@tcp2 status: up interfaces: 0: eth1 lnetctl route show route: - net: tcp gateway: 192.168.122.26@tcp2 - net: tcp gateway: 192.168.122.200@tcp2 | lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.102@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.252@tcp status: up interfaces: 0: eth1 - net type: tcp2 local NI(s): - nid: 192.168.122.237@tcp2 status: up interfaces: 0: eth2 - nid: 192.168.122.200@tcp2 status: up interfaces: 0: eth3 | lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.132@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.150@tcp status: up interfaces: 0: eth1 - net type: tcp2 local NI(s): - nid: 192.168.122.26@tcp2 status: up interfaces: 0: eth2 - nid: 192.168.122.244@tcp2 status: up interfaces: 0: eth3 |
Gateway Preference For Destination
Preferred Gateway For Destination NID
UDSP | Behaviour |
---|---|
On PeerB, add UDSP that gives one of the PeerA nids highest priority On PeerB, designate Gateway1 as preferred for the same PeerA nid. | All sends from PeerB go to the prioritized nid of PeerA through the preferred gateway Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, Gateway1 should be listed as preferred for the specified nid of PeerA on tcp "netstat -i" command shows rx stats incrementing for eth0 on peerA |
lnetctl udsp add --dst 192.168.122.103@tcp lnetctl udsp add --dst 192.168.122.103@tcp --rte 192.168.122.102@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: 192.168.122.103@tcp rte: NA action: priority: 0 - idx: 1 src: NA dst: 192.168.122.103@tcp rte: 192.168.122.102@tcp | PeerB: lnetctl peer show -v 4 >>>> - nid: 192.168.122.103@tcp udsp info: net priority: -1 nid priority: 0 Preferred gateway NIDs: NID-0: 192.168.122.102@tcp <<<< |
Preferred Gateway For Destination Net
UDSP | Behaviour |
---|---|
On PeerB, designate Gateway1 as preferred for tcp remote network | The preferred gateway for tcp is used for all traffic Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp: |
lnetctl udsp add --dst tcp --rte 192.168.122.102@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: tcp rte: 192.168.122.102@tcp | PeerB: lnetctl peer show -v 4 >>>> - nid: 192.168.122.103@tcp udsp info: net priority: -1 nid priority: -1 Preferred gateway NIDs: NID-0: 192.168.122.102@tcp <<<<< >>>>> - nid: 192.168.122.154@tcp udsp info: net priority: -1 nid priority: -1 Preferred gateway NIDs: NID-0: 192.168.122.102@tcp <<<< |