Page History

...

UDSP

Behaviour

Prioritize one of PeerA NIDs on tcp

Prioritized nid is used when its net (tcp) is used on peerA

Priority of the nid is 0. Other nids' priority is -1 (default, no priority)

Every send toggles between the prioritized nid on tcp and the nid on tcp1 on the source. The remaining nid on tcp is not used

Code Block
lnetctl udsp add --src 192.168.122.110@tcp lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: NA rte: NA action: priority: 0

PeerA:

Code Block

PeerA:
lnetctl net show -v 4
net:
>>>>
    - net type: lotcp
      local NI(s):
        - nid: 0@lo192.168.122.110@tcp
          status: up
          statisticsinterfaces:
              send_count: 0
              recv_count: 0
              drop_count: 0eth0
....
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats0
<<<<

Local Source Net

UDSP

Behaviour

Prioritize tcp1 as source on PeerA

Note that source NID rule is still in the list.

Prioritized tcp1 is used on PeerA

Priority of tcp1 is 0 (highest)

Every send on PeerA goes out tcp1. NID rule is overriden.

Code Block

lnetctl udsp add --src tcp1

lnetctl udsp show
udsp:
    - idx: 0
        putsrc: 0192.168.122.110@tcp
      dst: NA
       getrte: 0NA
      action:
        reply  priority: 0
    - idx: 1
        acksrc: 0tcp1
      dst: NA
       hellorte: 0NA
          received_statsaction:
          priority: 0

PeerA:

Code Block

lnetctl net show put: 0-v 4

>>>>
    - net type: tcp
      local get: 0NI(s):
        - nid: 192.168.122.110@tcp
    reply: 0
     status: up
        ack: 0
 interfaces:
             hello 0: 0eth0
....
          dropped_statsudsp info:
              net putpriority: 0-1
              nid getpriority: 0
<<<<
>>>>    
    - net type: tcp1
   reply: 0
  local NI(s):
        -   acknid: 0192.168.122.238@tcp1
              hellostatus: 0up
          health statsinterfaces:
              health value0: 0eth2
....
          udsp    interrupts: 0info:
              droppednet priority: 0
              nid aborted: 0
      priority: -1
<<<<

Prioritize Destination

Local Destination NID

UDSP

Behaviour

On PeerA, prioritize one of PeerB's NIDs on tcp

Prioritized NID tcp NID on PeerA is used when tcp is selected on peerA

Priority of the prioritized NID is 0 (lnetct peer show -v 4)

Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID

Code Block
lnetctl udsp add --dst 192.168.122.103@tcp lnetctl udsp show udsp:

no route

- idx: 0
      src: NA

timeouts

dst:

0

192.168.122.103@tcp

error

rte:

0

NA

tunables

action:

peer_timeout

priority: 0

PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID

PeerA: use "lnetct peer show -v 4" to verify dest NID priority.

Local Destination Net

UDSP

Behaviour

On PeerA, prioritize tcp1 as dest net

Prioritized dest net (tcp1) is used to send from PeerA

Priority of the prioritized dest net is 0 (lnetct peer show -v 4)

Sends on PeerA use tcp1.

Code Block
lnetctl udsp add --dst tcp1 lnetctl udsp show udsp:

peer_credits: 0 peer_buffer_credits: 0

credits

- idx: 0

dev cpt

src:

0

NA

tcp bonding

dst:

0

tcp1

CPT

rte:

"[0]"

NA

-

net

type

action:

tcp

local

NI(s):

-

nid: 192.168.122.110@tcp status: up interfaces: 0: eth0 statistics: send_count: 4 recv_count: 4 drop_count: 0 udsp info: net priority: -1 nid priority: 0 sent_stats: put: 0 get: 4 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 1 reply: 3 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]" - nid: 192.168.122.253@tcp status: up interfaces: 0: eth1 statistics: send_count: 2 recv_count: 2 drop_count: 0 udsp info: net priority: -1 nid priority: -1 sent_stats: put: 0 get: 2 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 1 reply: 1 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]" - net type: tcp1 local NI(s): - nid: 192.168.122.238@tcp1 status: up interfaces: 0: eth2 statistics: send_count: 5 recv_count: 5 drop_count: 0 udsp info: net priority: -1 nid priority: -1 sent_stats: put: 2 get: 3 reply: 0 ack: 0 hello: 0 received_stats: put: 1 get: 0 reply: 3 ack: 1 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 dev cpt: -1 tcp bonding: 0 CPT: "[0]"

Local Source Net

...

Prioritize tcp1 as source on PeerA

Note that source NID rule is still in the list.

...

Prioritized tcp1 is used on PeerA

Priority of tcp1 is 0 (highest)

Every send on PeerA goes out tcp1. NID rule is overriden.

Code Block
lnetctl udsp add --src tcp1 lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: NA rte: NA action: priority: 0 - idx: 1 src: tcp1 dst: NA rte: NA action: priority: 0

PeerA:

Code Block

lnetctl net show -v 4
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
          statistics:
              send_count: 0
              recv_count: 0
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 0
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 0
              peer_credits: 0
              peer_buffer_credits: 0
              credits: 0
          dev cpt: 0
          tcp bonding: 0
          CPT: "[0]"
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.110@tcp
          status: up
          interfaces:
              0: eth0
          statistics:
              send_count: 9
              recv_count: 9
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: 0
          sent_stats:
              put: 1
              get: 8
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 1
              get: 2
              reply: 6
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"
        - nid: 192.168.122.253@tcp
          status: up
          interfaces:
              0: eth1
          statistics:
              send_count: 2
              recv_count: 2
              drop_count: 0
          udsp info:
              net priority: -1
              nid priority: -1
          sent_stats:
              put: 0
              get: 2
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 0
              get: 1
              reply: 1
              ack: 0
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"
    - net type: tcp1
      local NI(s):
        - nid: 192.168.122.238@tcp1
          status: up
          interfaces:
              0: eth2
          statistics:
              send_count: 19
              recv_count: 19
              drop_count: 0
          udsp info:
              net priority: 0
              nid priority: -1
          sent_stats:
              put: 4
              get: 15
              reply: 0
              ack: 0
              hello: 0
          received_stats:
              put: 1
              get: 0
              reply: 15
              ack: 3
              hello: 0
          dropped_stats:
              put: 0
              get: 0
              reply: 0
              ack: 0
              hello: 0
          health stats:
              health value: 1000
              interrupts: 0
              dropped: 0
              aborted: 0
              no route: 0
              timeouts: 0
              error: 0
          tunables:
              peer_timeout: 180
              peer_credits: 8
              peer_buffer_credits: 0
              credits: 256
          dev cpt: -1
          tcp bonding: 0
          CPT: "[0]"

Prioritize Destination

Local Destination NID

...

On PeerA, prioritize one of PeerB's NIDs on tcp

...

Prioritized NID tcp NID on PeerA is used when tcp is selected on peerA

Priority of the prioritized NID is 0 (lnetct peer show -v 4)

Sends on PeerA alternate between tcp and tcp1. When tcp is used, send goes to the prioritized dest NID

...

Code Block
lnetctl udsp add --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: 192.168.122.103@tcp rte: NA action: priority: 0

...

PeerB: use "lnetct net show -v" to verify the receive count going up on the prioritized NID

PeerA: use "lnetct peer show -v 4" to verify dest NID priority.

Local Destination Net

...

On PeerA, prioritize tcp1 as dest net

...

Prioritized dest net (tcp1) is used to send from PeerA

Priority of the prioritized dest net is 0 (lnetct peer show -v 4)

Sends on PeerA use tcp1.

priority: 0

PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net

PeerA: use "lnetctl peer show -v 4" to verify dest net priority.

NID Pairing

Local Destination NID Pairing

UDSP

Behaviour

On PeerA, pair one of the tcp nids with one off the PeerB tcp nids

When paired nid on PeerA is selected, it always sends to its pair on peerB

PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v")

Code Block
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: 192.168.122.103@tcp rte: NA

PeerA: use "lnetctl peer show -v 4" to verify nid pairing.

Remote Peer Config

The following configuration is used for testing that involves remote peers and routing:

PeerA

PeerB

Gateway1

Gateway2

two nids on tcp

two nids on tcp2

two nids on tcp, two nids on tcp2

Code Block
lnetctl net show net: - net type: lo local NI(s)

...

Code Block
lnetctl udsp add --dst tcp1 lnetctl udsp show udsp: - idx: 0 src: NA dst: tcp1 rte: NA action: priority: 0

...

PeerB: use "lnetctl net show -v" to verify the receive count going up on the nids of the prioritized dest net

PeerA: use "lnetctl peer show -v 4" to verify dest net priority.

NID Pairing

Local Destination NID Pairing

...

On PeerA, pair one of the tcp nids with one off the PeerB tcp nids

...

When paired nid on PeerA is selected, it always sends to its pair on peerB

PeerA nid is listed as preferred NID for the peerB nid ("lnetctl net show -v")

...

Code Block
lnetctl udsp add --src 192.168.122.110@tcp --dst 192.168.122.103@tcp lnetctl udsp show udsp: - idx: 0 src: 192.168.122.110@tcp dst: 192.168.122.103@tcp rte: NA

...

PeerA: use "lnetctl peer show -v 4" to verify nid pairing.

Remote Peer Config

The following configuration is used for testing that involves remote peers and routing:

...

Code Block

lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.103@tcp
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.154@tcp
          status: up
          interfaces:
              0: eth1

Code Block
lnetctl route show route: - net: tcp2 gateway: 192.168.122.132@tcp - net: tcp2 gateway: 192.168.122.102@tcp

...

Code Block

lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp2
      local NI(s):
        - nid: 192.168.122.110@tcp2
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.253@tcp2
          status: up
          interfaces:
              0: eth1

Code Block
lnetctl route show route: - net: tcp gateway: 192.168.122.26@tcp2 - net: tcp gateway: 192.168.122.200@tcp2

...

Code Block

lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.102@tcp
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.252@tcp
          status: up
          interfaces:
              0: eth1
    - net type: tcp2
      local NI(s):
        - nid: 192.168.122.237@tcp2
          status: up
          interfaces:
              0: eth2
        - nid: 192.168.122.200@tcp2
          status: up
          interfaces:
              0: eth3

...

Code Block

lnetctl net show
net:
    - net type: lo
      local NI(s):
        - nid: 0@lo
          status: up
    - net type: tcp
      local NI(s):
        - nid: 192.168.122.132@tcp
          status: up
          interfaces:
              0: eth0
        - nid: 192.168.122.150@tcp
          status: up
          interfaces:
              0: eth1
    - net type: tcp2
      local NI(s):
        - nid: 192.168.122.26@tcp2
          status: up
          interfaces:
              0: eth2
        - nid: 192.168.122.244@tcp2
          status: up
          interfaces:
              0: eth3

Gateway Preference For Source

Preferred Gateway For Source Net

UDSPBehaviourOn PeerB, designate Gateway1 as preferred on tcp2All sends from PeerB use Gateway1

Code Block
lnetctl udsp add --src tcp2 --rte 192.168.122.102@tcp lnetctl udsp show udsp: - idx: 0 src: tcp2 dst: NA rte: 192.168.122.102@tcp

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp2:

Code Blocklnetctl net show -v 4 net: - net type: lo local NI(s): - nid: 0@lo status: up statistics: send_count: 0 recv_count: 0 drop_count: 0 udsp info: net priority: -1 nid priority: -1 sent_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 received_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 0 interrupts: 0 dropped: 0 aborted: 0 no route: 0 timeouts: 0 error: 0 tunables: peer_timeout: 0 peer_credits: 0 peer_buffer_credits: 0 credits: 0 dev cpt: 0 tcp bonding: 0 CPT: "[0]" - net type: tcp2 local NI(s): - nid: 192.168.122.110@tcp2 status: up interfaces: 0: eth0 statistics

send_count

- nid:

319

0@lo

recv_count

status:

319

up
    -

drop_count

net type:

0

tcp

udsp info

local NI(s):

net priority: -1

- nid: 192.168.122.103@tcp

nid priority

status:

-1

up
          interfaces:

Preferred

gateway

NIDs:

        0: eth0
        -

NID-0

nid: 192.168.122.

102@tcp

154@tcp

sent_stats

status: up

put: 3

interfaces:

get

0:

316

eth1

Code Block
lnetctl route show route:

reply

- net:

0

tcp2

ack: 0

gateway: 192.168.122.132@tcp
    -

hello

net:

0

tcp2
      gateway: 192.168.122.102@tcp

Code Block
lnetctl

received_stats

net show
net:
    - net type: lo
      local

put: 0

NI(s):

get

- nid:

3

0@lo

reply

status:

313

up
    - net type: tcp2
      local

ack: 3

NI(s):

hello: 0

- nid: 192.168.122.110@tcp2

dropped_stats:

   status: up

put

interfaces:

0

get

0:

0

eth0
        - nid: 192.168.122.253@tcp2

reply:

0

   status: up

ack

interfaces:

0

hello

0:

0

eth1

Code Block
lnetctl route show route: - net: tcp

health

stats:

gateway: 192.168.122.26@tcp2
    - net: tcp
      gateway:

health value: 1000

192.168.122.200@tcp2

Code Block
lnetctl net show net: - net type: lo local

interrupts: 0

NI(s):
        - nid: 0@lo

dropped:

0

     status: up
    - net

aborted

type:

0

tcp
      local NI(s):

no

route

nid:

0

192.168.122.102@tcp

timeouts

status:

0

up

error: 0

interfaces:

tunables:

    0: eth0
        -

peer_timeout: 180

nid: 192.168.122.252@tcp

peer_credits

status:

8

up

peer_buffer_credits: 0

interfaces:

credits

0:

256

eth1
    - net type: tcp2

dev

cpt:

-1

 local NI(s):

tcp

bonding

nid:

0

192.168.122.237@tcp2

CPT

status:

"[0]"

up

-

nid

interfaces:

192.168.122.253@tcp2

status

0:

up

eth2
        -

interfaces

nid:

 192.168.122.200@tcp2

0

status:

eth1

up

statistics

interfaces:

send_count

0:

319

eth3

Code Block
lnetctl net show net: - net type: lo

recv_count: 319

local NI(s):
        - nid: 0@lo

drop_count:

0

    status: up
    -

udsp

net

info

type: tcp
      local NI(s):

net

priority

nid:

-1

192.168.122.132@tcp

nid priority

status:

-1

up
          interfaces:

Preferred

gateway

NIDs:

        0: eth0
        -

NID-0

nid: 192.168.122.

102@tcp

150@tcp

sent_stats

status: up
          interfaces:

put:

17

       0: eth1
    - net

get

type:

302

tcp2
      local NI(s):
        -

reply

nid:

0

192.168.122.26@tcp2
          status: up

ack

interfaces:

0

hello

0:

0

eth2
        -

received_stats:

nid: 192.168.122.244@tcp2

put

status:

15

up
          interfaces:

get:

2

         0: eth3

Gateway Preference For Source

Preferred Gateway For Source Net

UDSP

Behaviour

On PeerB, designate Gateway1 as preferred on tcp2

All sends from PeerB use Gateway1

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp2:

Code Block
lnetctl udsp add --src tcp2 --rte 192.168.122.102@tcp lnetctl udsp show udsp: reply: 300 ack: 2 hello- idx: 0 dropped_stats:src: tcp2 putdst: 0NA rte: 192.168.122.102@tcp

PeerB:

Code Block

lnetctl net show -v 4

>>>>       get: 0
    - net type: tcp2
      local reply: 0NI(s):
              ack: 0- nid: 192.168.122.110@tcp2
              hellostatus: 0up
          health statsinterfaces:
              health value0: 1000eth0
....
          udsp    interruptsinfo:
 0
             net droppedpriority: 0-1
              nid abortedpriority: 0-1
              Preferred nogateway routeNIDs:
    0
              timeoutsNID-0: 0192.168.122.102@tcp
<<<<
>>>>
        - nid: 192.168.122.253@tcp2
         error status: 0up
          tunablesinterfaces:
              peer_timeout0: 180eth1
....
              peer_credits: 8
udsp info:
              net peer_buffer_creditspriority: 0-1
              nid creditspriority: 256-1
          dev cpt: -1
  Preferred gateway NIDs:
      tcp bonding: 0
          CPT: "[0]"NID-0: 192.168.122.102@tcp
>>>>

Preferred Gateway For Source NID

...

UDSP

Behaviour

On PeerB, add UDSP that gives one of the PeerA nids highest prioritypriority

On PeerB, designate Gateway1 as preferred for the same PeerA nid.

All sends from PeerB go to the prioritized nid of PeerA through the preferred gateway

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, designate Gateway1 should be listed as preferred for the same PeerA nid.

All sends from PeerB go to the prioritized nid of PeerA through the preferred gateway

specified nid of PeerA on tcp:

Code Block

lnetctl udsp add --dst 192.168.122.103@tcp

lnetctl udsp add --dst 192.168.122.103@tcp --rte 192.168.122.102@tcp

lnetctl udsp show
udsp:
    - idx: 0
      src: NA
      dst: 192.168.122.103@tcp
      rte: NA
      action:
          priority: 0
    - idx: 1
      src: NA
      dst: 192.168.122.103@tcp
      rte: 192.168.122.102@tcp

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerA "send_count" and "recv_count" should increase only for the prioritised nid. On PeerB, Gateway1 should be listed as preferred for the specified nid of PeerA on tcp:

.102@tcp

PeerB:

Code Block
lnetctl peer show -v 4 >>>> - nid: 192.168.122.103@tcp udsp info: net priority: -1 nid priority: 0 Preferred gateway NIDs: NID-0: 192.168.122.102@tcp <<<<

...

UDSP

Behaviour

On PeerB, designate Gateway1 as preferred for tcp remote network

The preferred gateway for tcp is used for all traffic

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp:

Code Block
lnetctl udsp add --dst tcp --rte 192.168.122.102@tcp lnetctl udsp show udsp: - idx: 0 src: NA dst: tcp rte: 192.168.122.102@tcp

Use "lnetctl stats show" on Gateway1 and Gateway2. On every send from PeerB to PeerA, "route_count" should increase only on Gateway1. On PeerB, Gateway1 should be listed as preferred for every nid on tcp:

PeerB:

Code Block

lnetctl peer show -v 4

>>>>
        - nid: 192.168.122.103@tcp
          udsp info:
              net priority: -1
              nid priority: -1
              Preferred gateway NIDs:
                  NID-0: 192.168.122.102@tcp
<<<<<
>>>>>
        - nid: 192.168.122.154@tcp
          udsp info:
              net priority: -1
              nid priority: -1
              Preferred gateway NIDs:
                  NID-0: 192.168.122.102@tcp

<<<<

Space shortcuts

Page tree

Versions Compared

Old Version 5

New Version 6

Key

Local Source Net

Prioritize Destination

Local Destination NID

Local Destination Net

Local Source Net

Prioritize Destination

Local Destination NID

Local Destination Net

NID Pairing

Local Destination NID Pairing

Remote Peer Config

NID Pairing

Local Destination NID Pairing

Remote Peer Config

Gateway Preference For Source

Preferred Gateway For Source Net

Gateway Preference For Source

Preferred Gateway For Source Net

Preferred Gateway For Source NID