Purpose
This document describes the UDSP feature from the user perspective. The document will cover all Intended use cases, list expected behaviour and potential issues for each.
UDSP Use Cases
Linux Routing
Because all use cases featured below rely on having multiple interfaces configured, it is important to make sure that linux routing is configured according to guidelines listed here: MR Cluster Setup
Local Peer Config
The following configuration is used for local testing.
...
UDSP | Behaviour |
---|
Prioritize one of PeerA NIDs on tcp | Prioritized nid is used when its net (tcp) is used on peerA Priority of the nid is 0. Other nids' priority is -1 (default, no priority) Every send toggles between the prioritized nid on tcp and the nid on tcp1 on the source. The remaining nid on tcp is not used "netstat -i" command shows tx stats incrementing for eth2 and eth0 on peerA |
Code Block |
---|
lnetctl udsp add --src 192.168.122.110@tcp
lnetctl udsp show
udsp:
- idx: 0
src: 192.168.122.110@tcp
dst: NA
rte: NA
action:
priority: 0 |
| PeerA: Code Block |
---|
PeerA:
lnetctl net show -v 4
net:>>>>
- net type: lotcp
local NI(s):
- nid: 0@lo192.168.122.110@tcp
status: up
statisticsinterfaces:
send_count: 0
recv_count: 0
drop_count: 0eth0
....
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 0
get: 0
<<<< |
|
Local Source Net
UDSP | Behaviour |
---|
Prioritize tcp1 as source on PeerA Note that source NID rule is still in the list. | Prioritized tcp1 is used on PeerA Priority of tcp1 is 0 (highest) Every send on PeerA goes out tcp1. NID rule is overriden. "netstat -i" command shows tx stats incrementing for eth2 on peerA |
Code Block |
---|
lnetctl udsp add --src tcp1
lnetctl udsp show
udsp:
- idx: 0
src: 192.168.122.110@tcp
replydst: 0NA
rte: NA
ackaction: 0
hellopriority: 0
- received_stats:idx: 1
src: tcp1
putdst: 0NA
rte: NA
getaction: 0
priority: 0 |
| PeerA: Code Block |
---|
lnetctl net show reply: 0-v 4
>>>>
- net type: tcp
local ack: 0NI(s):
hello: 0- nid: 192.168.122.110@tcp
dropped_statsstatus: up
put: 0interfaces:
get0: 0eth0
....
udsp replyinfo:
0
net ackpriority: 0-1
nid hellopriority: 0
<<<<
>>>>
- net healthtype: stats:tcp1
local NI(s):
health- valuenid: 0192.168.122.238@tcp1
interruptsstatus: 0up
dropped: 0
interfaces:
aborted: 0
no route: 0
timeouts: 0eth2
....
udsp error: 0info:
tunables:
peer_timeoutnet priority: 0
nid peer_creditspriority: 0
peer_buffer_credits: 0-1
<<<< |
|
Prioritize Destination
Local Destination NID
UDSP | Behaviour |
---|
On PeerA, prioritize one of PeerB's NIDs on tcp
| Prioritized NID on PeerB is sent to when tcp is selected on peerA Priority of the prioritized NID is 0 (lnetct peer show -v 4) Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID "netstat -i" command shows rx stats incrementing for eth2 and eth0 on peerB |
Code Block |
---|
lnetctl udsp add --dst 192.168.122.103@tcp
lnetctl udsp show
udsp:
|
|
credits dev cpt0 tcp bonding0 CPT"[0]"-nettype tcp localNI(s):
- nid: 192.168.122.110@tcp
| PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID PeerA: use "lnetct peer show -v 4" to verify dest NID priority. |
Local Destination Net
UDSP | Behaviour |
---|
On PeerA, prioritize tcp1 as dest net
| Prioritized dest net (tcp1) is used to send from PeerA Priority of the prioritized dest net is 0 (lnetct peer show -v 4) Sends on PeerA use tcp1. "netstat -i" command shows tx stats incrementing for eth2 on peerA |
Code Block |
---|
lnetctl udsp add --dst tcp1
lnetctl udsp show
udsp: |
|
status: up interfaces: 0eth0 statistics: send_count: 4 recv_count: 4
drop_count: 0
udsp info:
net priority: -1
nid priority: 0
sent_stats:
put: 0
get: 4
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 1
reply: 3
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]"
- nid: 192.168.122.253@tcp
status: up
interfaces:
0: eth1
statistics:
send_count: 2
recv_count: 2
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 0
get: 2
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 1
reply: 1
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]"
- net type: tcp1
local NI(s):
- nid: 192.168.122.238@tcp1
status: up
interfaces:
0: eth2
statistics:
send_count: 5
recv_count: 5
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 2
get: 3
reply: 0
ack: 0
hello: 0
received_stats:
put: 1
get: 0
reply: 3
ack: 1
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]"Local Source Net
...
Prioritize tcp1 as source on PeerA
Note that source NID rule is still in the list.
...
Prioritized tcp1 is used on PeerA
Priority of tcp1 is 0 (highest)
Every send on PeerA goes out tcp1. NID rule is overriden.
Code Block |
---|
lnetctl udsp add --src tcp1
lnetctl udsp show
udsp:
- idx: 0
src: 192.168.122.110@tcp
dst: NA
rte: NA
action:
priority: 0
- idx: 1
src: tcp1
dst: NA
rte: NA
action:
priority: 0 |
PeerA:
Code Block |
---|
lnetctl net show -v 4
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
statistics:
send_count: 0
recv_count: 0
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 0
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 0
peer_credits: 0
peer_buffer_credits: 0
credits: 0
dev cpt: 0
tcp bonding: 0
CPT: "[0]"
- net type: tcp
local NI(s):
- nid: 192.168.122.110@tcp
status: up
interfaces:
0: eth0
statistics:
send_count: 9
recv_count: 9
drop_count: 0
udsp info:
net priority: -1
nid priority: 0
sent_stats:
put: 1
get: 8
reply: 0
ack: 0
hello: 0
received_stats:
put: 1
get: 2
reply: 6
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]"
- nid: 192.168.122.253@tcp
status: up
interfaces:
0: eth1
statistics:
send_count: 2
recv_count: 2
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 0
get: 2
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 1
reply: 1
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]"
- net type: tcp1
local NI(s):
- nid: 192.168.122.238@tcp1
status: up
interfaces:
0: eth2
statistics:
send_count: 19
recv_count: 19
drop_count: 0
udsp info:
net priority: 0
nid priority: -1
sent_stats:
put: 4
get: 15
reply: 0
ack: 0
hello: 0
received_stats:
put: 1
get: 0
reply: 15
ack: 3
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 1000
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
dev cpt: -1
tcp bonding: 0
CPT: "[0]" |
Prioritize Destination
Local Destination NID
...
On PeerA, prioritize one of PeerB's NIDs on tcp
...
Prioritized NID tcp NID on PeerA is used when tcp is selected on peerA
Priority of the prioritized NID is 0 (lnetct peer show -v 4)
Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID
| PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net PeerA: use "lnetctl peer show -v 4" to verify dest net priority. |
NID Pairing
Local Destination NID Pairing
UDSP | Behaviour |
---|
On PeerA, pair one of the tcp nids with one off the PeerB tcp nids
| When paired nid on PeerA is selected, it always sends to its pair on peerB PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v") |
Code Block |
---|
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp
lnetctl udsp show
udsp:
- idx: 0
src: 192.168.122.110@tcp
dst: 192.168.122.103@tcp
rte: NA |
| PeerA: use "lnetctl peer show -v 4" to verify nid pairing. |
Remote Peer Config
Gliffy Diagram |
---|
name | RemoteLayout1 |
---|
pagePin | 3 |
---|
|
The following configuration is used for testing that involves remote peers and routing:
PeerA | PeerB | Gateway1 | Gateway2 |
---|
two nids on tcp | two nids on tcp2 | two nids on tcp, two nids on tcp2 | two nids on tcp, two nids on tcp2 |
Code Block |
---|
lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: tcp
local NI(s) |
|
...
Code Block |
---|
lnetctl udsp add --dst 192.168.122.103@tcp
lnetctl udsp show
udsp:
- idx: 0
src: NA
dst: 192.168.122.103@tcp
rte: NA
action:
priority: 0 |
...
PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID
PeerA: use "lnetct peer show -v 4" to verify dest NID priority.
Local Destination Net
...
On PeerA, prioritize tcp1 as dest net
...
Prioritized dest net (tcp1) is used to send from PeerA
Priority of the prioritized dest net is 0 (lnetct peer show -v 4)
Sends on PeerA use tcp1.
...
Code Block |
---|
lnetctl udsp add --dst tcp1
lnetctl udsp show
udsp:
- idx: 0
src: NA
dst: tcp1
rte: NA
action:
priority: 0 |
...
PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net
PeerA: use "lnetctl peer show -v 4" to verify dest net priority.
NID Pairing
Local Destination NID Pairing
...
On PeerA, pair one of the tcp nids with one off the PeerB tcp nids
...
When paired nid on PeerA is selected, it always sends to its pair on peerB
PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v")
...
Code Block |
---|
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp
lnetctl udsp show
udsp:
- idx: 0
src: 192.168.122.110@tcp
dst: 192.168.122.103@tcp
rte: NA |
...
PeerA: use "lnetctl peer show -v 4" to verify nid pairing.
Remote Peer Config
The following configuration is used for testing that involves remote peers and routing:
...
Code Block |
---|
lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: tcp
local NI(s):
- nid: 192.168.122.103@tcp
status: up
interfaces:
0: eth0
- nid: 192.168.122.154@tcp
status: up
interfaces:
0: eth1 |
Code Block |
---|
lnetctl route show
route:
- net: tcp2
gateway: 192.168.122.132@tcp
- net: tcp2
gateway: 192.168.122.102@tcp |
...
Code Block |
---|
lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: tcp2
local NI(s):
- nid: 192.168.122.110@tcp2
status: up
interfaces:
0: eth0
- nid: 192.168.122.253@tcp2
status: up
interfaces:
0: eth1 |
Code Block |
---|
lnetctl route show
route:
- net: tcp
gateway: 192.168.122.26@tcp2
- net: tcp
gateway: 192.168.122.200@tcp2 |
...
Code Block |
---|
lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: tcp
local NI(s):
- nid: 192.168.122.102@tcp
status: up
interfaces:
0: eth0
- nid: 192.168.122.252@tcp
status: up
interfaces:
0: eth1
- net type: tcp2
local NI(s):
- nid: 192.168.122.237@tcp2
status: up
interfaces:
0: eth2
- nid: 192.168.122.200@tcp2
status: up
interfaces:
0: eth3 |
...
Code Block |
---|
lnetctl net show
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
- net type: tcp
local NI(s):
- nid: 192.168.122.132@tcp
status: up
interfaces:
0: eth0
- nid: 192.168.122.150@tcp
status: up
interfaces:
0: eth1
- net type: tcp2
local NI(s):
- nid: 192.168.122.26@tcp2
status: up
interfaces:
0: eth2
- nid: 192.168.122.244@tcp2
status: up
interfaces:
0: eth3 |
Gateway Preference For Source
Preferred Gateway For Source Net
UDSP | Behaviour |
---|
On PeerB, designate Gateway1 as preferred on tcp2 | All sends from PeerB use Gateway1 |
Code Block |
---|
lnetctl udsp add --src tcp2 --rte 192.168.122.102@tcp
lnetctl udsp show
udsp:
- idx: 0
src: tcp2
dst: NA
rte: 192.168.122.102@tcp |
| Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp2:
Code Block |
lnetctl net show -v 4
net:
- net type: lo
local NI(s):
- nid: 0@lo
status: up
statistics:
send_count: 0
recv_count: 0
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
sent_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats:
health value: 0
interrupts: 0
dropped: 0
aborted: 0
no route: 0
timeouts: 0
error: 0
tunables:
peer_timeout: 0
peer_credits: 0
peer_buffer_credits: 0
credits: 0
dev cpt: 0
tcp bonding: 0
CPT: "[0]"
- net type: tcp2
local NI(s):
- nid: 192.168.122.110@tcp2
status: up
interfaces:
0: eth0
statistics:
send_count: 319
recv_count: 319
drop_count: 0
udsp info:
net priority: -1
nid priority: -1
Preferred gateway NIDs:
NID-0: 192.168.122.102@tcp
sent_stats:
put: 3
get: 316
reply: 0
ack: 0
hello: 0
received_stats:
put: 0
get: 3
reply: 313
ack: 3
hello: 0
dropped_stats:
put: 0
get: 0
reply: 0
ack: 0
hello: 0
health stats health value: 1000- nid: 192.168.122.103@tcp
|
|
interrupts0 dropped: 0
aborted:0
noroute0192.168.122.154@tcp
status: up
|
|
timeouts:0
error:0
tunablesroute show
route:
- net: tcp2
|
|
peer_timeout180
192.168.122.132@tcp
- net: tcp2
gateway: 192.168.122.102@tcp |
| |
peer_credits: 8net show
net:
- net type: lo
|
|
peer_buffer_credits: 0 credits256dev cpt-1 tcp bonding0 CPT: "[0]"local NI(s):
- nid: 192.168.122. |
|
253@tcp2110@tcp2
status: up
interfaces:
0: |
|
eth1 statistics:nid: 192.168.122.253@tcp2
status: |
|
send_count: 319
recv_count: 319
Code Block |
---|
lnetctl route show
route:
- |
|
drop_count0tcp
gateway: 192.168.122.26@tcp2
|
|
udspinfonet: tcp
gateway: 192.168.122.200@tcp2 |
| Code Block |
---|
lnetctl net show
net:
- net |
|
priority-1 priority-1PreferredgatewayNIDs NID-0sent_stats put: 17get302 reply: 0- nid: 192.168.122.252@tcp
|
|
ack0 hello:0
received_stats0: eth1
- net type: tcp2
local |
|
put: 15 get: 2- nid: 192.168.122.237@tcp2
|
|
reply300 ack: 2hello0 dropped_stats:nid: 192.168.122.200@tcp2
status: up
|
|
put:0
get:0
| Code Block |
---|
lnetctl net show
net:
- net |
|
reply0ack0hello:0
health statslocal NI(s):
- nid: 192.168.122.132@tcp
|
|
healthvalue:1000
interrupts 0dropped0eth0
- nid: 192.168.122.150@tcp
|
|
aborted:0
noroute 0timeouts0eth1
- net type: tcp2
local |
|
error: 0 tunables
peer_timeout180 peer_credits: 8peer_buffer_credits0 credits: 256- nid: 192.168.122.244@tcp2
|
|
dev cpt-1tcpbonding:0
CPT: "[0]"Preferred Gateway For Source NID
...
Gateway Preference For Destination
Preferred Gateway For Destination NID
Gliffy Diagram |
---|
name | RemoteUDSPDstNid1 |
---|
pagePin | 3 |
---|
|
UDSP | Behaviour |
---|
On PeerB, add UDSP that gives one of the PeerA nids highest priority On PeerB, designate Gateway1 as preferred for the same PeerA nid. | All sends from PeerB go to the prioritized nid of PeerA through the preferred gateway Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to |
the prioritized nid of PeerA through the preferred gateway PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, Gateway1 should be listed as preferred for the specified nid of PeerA on tcp "netstat -i" command shows rx stats incrementing for eth0 on peerA |
Code Block |
---|
lnetctl udsp add --dst 192.168.122.103@tcp
lnetctl udsp add --dst 192.168.122.103@tcp --rte 192.168.122.102@tcp
lnetctl udsp show
udsp:
- idx: 0
src: NA
dst: 192.168.122.103@tcp
rte: NA
action:
priority: 0
- idx: 1
src: NA
dst: 192.168.122.103@tcp
rte: 192.168.122.102@tcp |
|
Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, Gateway1 should be listed as preferred for the specified nid of PeerA on tcp: | PeerB: Code Block |
---|
lnetctl peer show -v 4
>>>>
- nid: 192.168.122.103@tcp
udsp info:
net priority: -1
nid priority: 0
Preferred gateway NIDs:
NID-0: 192.168.122.102@tcp
<<<< |
|
Preferred Gateway For Destination NetDestination Net
Gliffy Diagram |
---|
| |
---|
name | RemoteUDSPDstNet2 |
---|
pagePin | 1 |
---|
|
UDSP | Behaviour |
---|
On PeerB, designate Gateway1 as preferred for tcp remote network | The preferred gateway for tcp is used for all traffic Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp: |
Code Block |
---|
lnetctl udsp add --dst tcp --rte 192.168.122.102@tcp
lnetctl udsp show
udsp:
- idx: 0
src: NA
dst: tcp
rte: 192.168.122.102@tcp |
|
Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp: | PeerB: Code Block |
---|
lnetctl peer show -v 4
>>>>
- nid: 192.168.122.103@tcp
udsp info:
net priority: -1
nid priority: -1
Preferred gateway NIDs:
NID-0: 192.168.122.102@tcp
<<<<<
>>>>>
- nid: 192.168.122.154@tcp
udsp info:
net priority: -1
nid priority: -1
Preferred gateway NIDs:
NID-0: 192.168.122.102@tcp
<<<< |
|