Versions Compared

Old Version 3

changes.mady.by.user Serguei Smirnov

Saved on

compared with

New Version Current

changes.mady.by.user Serguei Smirnov

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Purpose

This document describes the UDSP feature from the user perspective. The document will cover all Intended use cases, list expected behaviour and potential issues for each.

UDSP Use Cases

Linux Routing

Because all use cases featured below rely on having multiple interfaces configured, it is important to make sure that linux routing is configured according to guidelines listed here: MR Cluster Setup

Local Peer Config

Gliffy Diagram
nameLocalLayout1
pagePin5

The following configuration is used for local testing.

...

UDSPBehaviour
Prioritize one of PeerA NIDs on tcp

Prioritized nid is used when its net (tcp) is used on peerA

Priority of the nid is 0. Other nids' priority is -1 (default, no priority)

Every send toggles between the prioritized nid on tcp and the nid on tcp1 on the source. The remaining nid on tcp is not used

"netstat -i" command shows tx stats incrementing for eth2 and eth0 on peerA

Code Block
lnetctl udsp add --src 192.168.122.110@tcp

lnetctl udsp  show 
udsp:
    - idx: 0
      src: 192.168.122.110@tcp
      dst: NA
      rte: NA
      action:
          priority: 0
Code Block

PeerA:

Code Block

lnetctl net show -v 4

net:>>>>
    - net type: lotcp
      local NI(s):
        - nid: 0@lo192.168.122.110@tcp
          status: up
          statisticsinterfaces:
              send_count: 0
              recv_count: 0
              drop_count: 0eth0
....
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats:
        0
<<<<

Local Source Net

 

UDSPBehaviour

Prioritize tcp1 as source on PeerA

Note that source NID rule is still in the list.

Prioritized tcp1 is used on PeerA

Priority of tcp1 is 0 (highest)

Every send on PeerA goes out tcp1. NID rule is overriden.

"netstat -i" command shows tx stats incrementing for eth2 on peerA

Code Block
lnetctl udsp add --src tcp1

lnetctl udsp show
udsp:
    - idx: 0
      putsrc: 0192.168.122.110@tcp
      dst: NA
       getrte: 0NA
      action:
        reply  priority: 0
    - idx: 1
        acksrc: 0tcp1
      dst: NA
       hellorte: 0NA
          received_statsaction:
          priority: 0




PeerA:
Code Block
lnetctl net show put: 0-v 4

>>>>
    - net type: tcp
      local get: 0NI(s):
        - nid: 192.168.122.110@tcp
    reply: 0
     status: up
        ack  interfaces: 0
              hello0: 0eth0
....
          dropped_statsudsp info:
              putnet priority: 0-1
              getnid priority: 0
<<<<
>>>>    
    - net     replytype: 0tcp1
              ack: 0local NI(s):
              hello: 0- nid: 192.168.122.238@tcp1
          healthstatus: stats:up
              health value: 0interfaces:
              interrupts0: 0eth2
....
          udsp    droppedinfo:
 0
             net abortedpriority: 0
              nonid routepriority: 0
              timeouts: 0
              error: 0
          tunables:-1
<<<<

Prioritize Destination

Local Destination NID

UDSPBehaviour

On PeerA, prioritize one of PeerB's NIDs on tcp 


Prioritized NID on PeerB is sent to when tcp is selected on peerA

Priority of the prioritized NID is 0 (lnetct peer show -v 4)

Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID

"netstat -i" command shows rx stats incrementing for eth2 and eth0 on peerB

Code Block
lnetctl udsp add --dst 192.168.122.103@tcp

lnetctl udsp show
udsp:
    - idx: 0
      src: NA
peer_timeout
dst:
0
192.168.122.103@tcp
      rte: NA
      action:
peer_credits:
0
peer_buffer_credits
priority: 0

PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID 

PeerA: use "lnetct peer show -v 4" to verify dest NID priority.

Local Destination Net

UDSPBehaviour

On PeerA, prioritize tcp1 as dest net


Prioritized dest net (tcp1) is used to send from PeerA

Priority of the prioritized dest net is 0 (lnetct peer show -v 4)

Sends on PeerA use tcp1.

"netstat -i" command shows tx stats incrementing for eth2 on peerA

Code Block
lnetctl udsp add --dst tcp1

lnetctl udsp show
udsp:
credits: 0 dev cpt: 0 tcp bonding: 0 CPT: "[0]" - net type: tcp local NI(s): 
    -
nid
idx:
192.168.122.110@tcp
0
status
src:
up
NA
interfaces:
dst: tcp1
0
rte:
eth0
NA
statistics
action:
send_count: 4 recv_count: 4 drop_count
priority: 0

PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net 

PeerA: use "lnetctl peer show -v 4" to verify dest net priority.

NID Pairing

Local Destination NID Pairing

UDSPBehaviour

On PeerA, pair one of the tcp nids with one off the PeerB tcp nids 


When paired nid on PeerA is selected, it always sends to its pair on peerB

PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v")

Code Block
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp

lnetctl udsp show
udsp:
    - idx: 0
udsp info:
src: 192.168.122.110@tcp
net priority: -1
dst: 192.168.122.103@tcp
      rte: NA

PeerA: use "lnetctl peer show -v 4" to verify nid pairing.



Remote Peer Config

Gliffy Diagram
nameRemoteLayout1
pagePin3

The following configuration is used for testing that involves remote peers and routing:

PeerAPeerBGateway1Gateway2
two nids on tcptwo nids on tcp2two nids on tcp, two nids on tcp2two nids on tcp, two nids on tcp2
Code Block
lnetctl net show
net:
    - net type: lo
nid priority: 0 sent_stats: put: 0 get: 4 reply: 0

      local NI(s):
        -
ack
nid:
0
0@lo
          status: up
hello:
0 
- net type: tcp
received_stats
local NI(s):
        - nid: 192.168.122.103@tcp
put:
0 
   status: up
get
interfaces:
1
reply
0:
3
eth0
ack: 0
- nid: 192.168.122.154@tcp
hello
status:
0
up
dropped_stats
interfaces:
put
0:
0 
eth1
Code Block
lnetctl route show
route:
    - net: tcp2
get
gateway:
0
192.168.122.132@tcp
    - net: tcp2
      gateway:
reply: 0
192.168.122.102@tcp
Code Block
lnetctl net show
net:
    - net type: lo
      local
ack
NI(s):
0 
       - nid: 0@lo
hello:
0 
    status: up
    -
health
net
stats
type: tcp2
health value: 1000
local NI(s):
interrupts: 0
- nid: 192.168.122.110@tcp2
dropped
status:
0
up
aborted: 0
interfaces:
no route: 
0
timeouts
:
0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]" - nid: 192.168.122.253@tcp status: up interfaces: 0: eth1 statistics: send_count: 2 recv_count: 2 drop_count: 0 udsp info: net priority: -1 nid priority: -1 sent_stats: put: 0 get: 2 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 1 reply: 1 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]" - net type: tcp1 local NI(s): - nid: 192.168.122.238@tcp1 status: up interfaces: 0: eth2 statistics: send_count: 5 recv_count: 5 drop_count: 0 udsp info: net priority: -1 nid priority: -1 sent_stats: put: 2 get: 3 reply: 0 ack: 0 hello: 0 received_stats: put: 1 get: 0 reply: 3 ack: 1 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]"

Local Source Net

 

...

Prioritize tcp1 as source on PeerA

Note that source NID rule is still in the list.

...

Prioritized tcp1 is used on PeerA

Priority of tcp1 is 0 (highest)

Every send on PeerA goes out tcp1. NID rule is overriden.

Code Block
lnetctl udsp add --src tcp1

lnetctl udsp show
udsp:
    - idx: 0
      src: 192.168.122.110@tcp
      dst: NA
      rte: NA
      action:
          priority: 0
    - idx: 1
      src: tcp1
      dst: NA
      rte: NA
      action:
          priority: 0
eth0
        - nid: 192.168.122.253@tcp2
          status: up
          interfaces:
              0: eth1
Code Block
lnetctl route show
route:
    - net: tcp
      gateway: 192.168.122.26@tcp2
    - net: tcp
      gateway: 192.168.122.200@tcp2
Code Block
lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
PeerA:
Code Block
lnetctl net show -v 4
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 0
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 0
              peer_credits: 0
              peer_buffer_credits: 0
              credits: 0
          dev cpt: 0
          tcp bonding: 0
          CPT: "[0]"
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.110@tcp
          status: up
          interfaces:
              0: eth0
          statistics:
              send_count: 9
              recv_count: 9
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: 0
          sent_stats:
              put: 1
              get: 8
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 1
              get: 2
              reply: 6
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"
        - nid: 192.168.122.253@tcp
          status: up
          interfaces:
              0: eth1
          statistics:
              send_count: 2
              recv_count: 2
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats:
              put: 0
              get: 2
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 0
              get: 1
              reply: 1
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"
    - net type: tcp1
      local NI(s):
        - nid: 192.168.122.238@tcp1
          status: up
          interfaces:
              0: eth2
          statistics:
              send_count: 19
              recv_count: 19
              drop_count: 0
          udsp info:
              net priority: 0
              nid priority: -1
          sent_stats:
              put: 4
              get: 15
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 1
              get: 0
              reply: 15
              ack: 3
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"

Prioritize Destination

Local Destination NID

UDSPBehaviour

On PeerA, prioritize one of PeerB's NIDs on tcp 

Prioritized NID tcp NID on PeerA is used when tcp is selected on peerA

Priority of the prioritized NID is 0 (lnetct peer show -v 4)

Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID

Code Blocklnetctl udsp add --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: 192.168.122.103@tcp rte: NA action:
priority
status:
0

PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID 

PeerA: use "lnetct peer show -v 4" to verify dest NID priority.

Local Destination Net

UDSPBehaviour

On PeerA, prioritize tcp1 as dest net

Prioritized dest net (tcp1) is used to send from PeerA

Priority of the prioritized dest net is 0 (lnetct peer show -v 4)

Sends on PeerA use tcp1.

Code Blocklnetctl udsp add --dst tcp1 lnetctl udsp show udsp
up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.102@tcp
          status: up
          interfaces:
-
idx:
0
src
0:
NA
eth0
dst
- nid:
tcp1
192.168.122.252@tcp
rte: NA 
    status: up
action
interfaces:
priority: 0

PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net 

PeerA: use "lnetctl peer show -v 4" to verify dest net priority.

NID Pairing

Local Destination NID Pairing

UDSPBehaviour

On PeerA, pair one of the tcp nids with one off the PeerB tcp nids 

When paired nid on PeerA is selected, it always sends to its pair on peerB

PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v")

Code Blocklnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0
   0: eth1
    - net type: tcp2
      local NI(s):
        - nid: 192.168.122.237@tcp2
          status: up
          interfaces:
              0: eth2
src
  - nid: 192.168.122.
110@tcp
200@tcp2
          status: up
dst: 192.168.122.103@tcp 
    interfaces:
rte: NA

PeerA: use "lnetctl peer show -v 4" to verify nid pairing.

Remote Peer Config

The following configuration is used for testing that involves remote peers and routing:

PeerAPeerBGateway1Gateway2two nids on tcptwo nids on tcp2two nids on tcp, two nids on tcp2two nids on tcp, two nids on tcp2
0: eth3
Code Block
lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.
103@tcp
132@tcp
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.150@tcp
          status: up
status
interfaces:
up
interfaces
0: eth1
    - net type: tcp2
      local
0: eth0
NI(s):
        - nid: 192.168.122.
154@tcp
26@tcp2
          status: up
          interfaces:
              0:
eth1 Code Blocklnetctl net show net: 
eth2
-
net type: lo local NI(s): 
   
- nid:
0@lo
192.168.122.244@tcp2
          status: up
-
net
type
interfaces:
tcp2
local
NI(s):
- nid: 192.168.122.110@tcp2 status: up interfaces: 0: eth0 - nid: 192.168.122.253@tcp2 status: up interfaces: 0: eth1 Code Blocklnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status:
0: eth3

Gateway Preference For Destination

Preferred Gateway For Destination NID

Gliffy Diagram
nameRemoteUDSPDstNid1
pagePin3

UDSPBehaviour

On PeerB, add UDSP that gives one of the PeerA nids highest priority

On PeerB, designate Gateway1 as preferred for the same PeerA nid.

All sends from PeerB go to the prioritized nid of PeerA through the preferred gateway 

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, Gateway1 should be listed as preferred for the specified nid of PeerA on tcp

"netstat -i" command shows rx stats incrementing for eth0 on peerA

Code Block
lnetctl udsp add --dst 192.168.122.103@tcp

lnetctl udsp add --dst 192.168.122.103@tcp --rte 192.168.122.102@tcp

lnetctl udsp show
udsp:
up

    -
net type
idx:
tcp
0
local NI(s):
src: NA
- nid
dst: 192.168.122.
102@tcp
103@tcp
      rte: NA
status
action:
up
interfaces
priority: 0
    - idx: 1
0
src:
eth0
NA
- nid
dst: 192.168.122.
252@tcp
103@tcp
      rte: 192.168.122.102@tcp

PeerB:

Code Block
lnetctl peer show
status: up
-v 4

>>>>
        -
interfaces:
nid: 192.168.122.103@tcp
          udsp
0
info:
eth1
-
net
type:
tcp2
local NI(s):
net priority: -1
- nid: 192.168.122.237@tcp2 
          nid
status
priority:
up
0
interfaces: 
    Preferred gateway NIDs:
0:
eth2 
        NID-
nid
0: 192.168.122.102@tcp
<<<<


Preferred Gateway For Destination Net

Gliffy Diagram
nameRemoteUDSPDstNet2
pagePin1

UDSPBehaviour

On PeerB, designate Gateway1 as preferred for tcp remote network

 The preferred gateway for tcp is used for all traffic

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp:

Code Block
lnetctl udsp add --dst tcp --rte 192.168.122.102@tcp

lnetctl udsp show
udsp:
    - idx: 0
.122.200@tcp2 status: up interfaces: 0: eth3 Code Blocklnetctl net show net: - net type: lo local NI(s):
- nid
src:
0@lo
NA
status
dst:
up
tcp
-
net
type
rte:
tcp local NI(s):
192.168.122.102@tcp

PeerB:

Code Block
lnetctl peer show -v 4

>>>>
        - nid: 192.168.122.
132@tcp
103@tcp
          udsp
status
info:
up
interfaces: 
    net priority: -1
0:
eth0 
     nid priority:
- nid: 192.168.122.150@tcp
-1
status:
up 
   Preferred
interfaces
gateway NIDs:
0:
eth1
- net type: tcp2 local NI(s):
NID-0: 192.168.122.102@tcp
<<<<<
>>>>>
        - nid: 192.168.122.
26@tcp2
154@tcp
          udsp
status
info:
up
interfaces: 
    net priority: -1
0:
eth2 
     nid priority:
- nid: 192.168.122.244@tcp2
-1
status:
up 
   Preferred gateway 
NIDs:
interfaces: 
              NID-0:
eth3

Gateway Preference For Source

...

192.168.122.102@tcp

<<<<