• Setup: configure single network, 3 NIDs on the network 
• Execute UDSP "Show" command 
• Verify that the resulting YAML file doesn't contain any policy description

test_udsp_basic_02 

it seems the tc only add rule to 1 network instead of NID, also not lower the priority

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives two of the interfaces highest priority
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected
• Add UDSP rule that lowers the priority of one of the NIDs with highest priority back to default
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected

test_udsp_basic_03

tc only delete rule on network, doesn't delete the rule on NID

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Add UDSP rule that gives the highest priority to one of the NIDs on the network with the highest priority
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected (for network and NID)
• Delete UDSP rule that gives the highest priority to one of the NIDs
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected (for network)
• Delete UDSP rule that gives the highest priority to one of the networks
• Execute UDSP "Show" command
• Verify that the resulting YAML file doesn't contain any policy description

test_udsp_basic_04

tc doesn't delete rule

• Setup: configure single network, 3 NIDs on the network locally, 3 NIDs on the remote peer
• Add UDSP rule that gives priority to one of the remote peer's NIDs
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected
• Delete UDSP rule that gives the highest priority to the remote peer's NID
• Execute UDSP "Show" command
• Verify that the resulting YAML file doesn't contain any policy description

test_udsp_basic_05

tc doesn't delete rule

• Setup: configure single network, 3 NIDs on the network locally, 3 NIDs on the local peer
• Add UDSP rule that gives priority to a pair of the local NID and peer's NID
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected
• Delete UDSP rule that gives the highest priority to the NID pair
• Execute UDSP "Show" command
• Verify that the resulting YAML file doesn't contain any policy description

test_udsp_basic_06

tc doesn't delete rule

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router and remote peer NID as preferred
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected
• Delete UDSP rule that gives the highest priority to the router/remote NID pair
• Execute UDSP "Show" command
• Verify that the resulting YAML file doesn't contain any policy description

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives two of the interfaces highest priority
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected
• Execute UDSP "Delete" command providing policy index that doesn't exist
• Verify that the command failed with an error

• Setup: configure single network, 3 NIDs on the network 
• Verify that attempts to add UDSP rule for source or dst network fail: "0tcp", "0o2ib", "tcpp", "t", "tcp100000000000000000000000000000000000000000000000001"

• Setup: configure single network, 3 NIDs on the network 
• Verify that attempts to add UDSP rule for source or dst NID fail: "@tcp", "1@tcp", "1.1.1@tcp", "1.1.1.1.1@tcp"

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives one of the local interfaces highest priority
• Start traffic TODO: need to specify direction?
• Stop traffic
• Verify that the NID with the highest priority was used

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives two of the local interfaces highest priority
• Start traffic
• Stop traffic
• Verify that two NIDs with the highest priority were used 

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives two of the interfaces highest priority
• Start traffic
• Stop traffic
• Verify that two NIDs with the highest priority were used
• Add UDSP that lowers the priority of one of the NIDs with highest priority back to default
• Start traffic
• Stop traffic
• Verify that the remaining highest-priority NID was used

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives two of the interfaces highest priority
• Start traffic
• Stop traffic
• Verify that two NIDs with the highest priority were used
• Add UDSP that lowers the priority of both of the NIDs with the highest priority back to default
• Start traffic
• Stop traffic
• Verify that all NIDs were used

• Setup: configure single network, 3 NIDs on the network, 3 NIDs on the local peer
• Add UDSP rule that gives the highest priority to one of the local peer NIDs
• Start traffic
• Stop traffic
• Verify that local peer NID with the highest priority was used

• Setup: configure single network, 3 NIDs on the network, 3 NIDs on the local peer
• Add UDSP rule that gives one of the local NIDs highest priority
• Add UDSP rule that pairs the highest priority local NID with a certain peer NID
• Start traffic
• Stop traffic
• Verify that the paired peer NID was used

• Setup: configure single network, 3 NIDs on the network, 3 NIDs on the local peer
• Add UDSP rule that creates 3 NID pairs such that one of the peer NIDs is not in any pair
• Start traffic
• Stop traffic
• Verify that the peer NID that is not part of any pair was not used (less used?)
• Delete UDSP rule
• Start traffic
• Stop traffic
• Verify that all NIDs are used 

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives highest priority to a NID that doesn't exist
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected

pass - run manually

set udsp on eth0:2, lctl ping and net show the send_count +1 on that interface;

then del that interface; lctl ping again, traffic goes to other interface;

add the eth0:2 back again, lctl ping, traffic to the prioritized interface again

• Setup: configure single network, 3 NIDs on the network 
• Add UDSP rule that gives one of the interfaces highest priority
• Start traffic
• Verify that the NID with the highest priority was used
• Bring the highest priority NID down
• Verify that remaining NIDs are used
• Stop traffic

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Start traffic
• Stop traffic
• Verify that NIDs on the network with the highest priority were used

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Start traffic
• Stop traffic
• Verify that NIDs on the network with the highest priority were used
• Add UDSP rule that reverses networks priorities
• Start traffic
• Stop traffic
• Verify that NIDs on the new network that has the highest priority were used

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Add UDSP rule that gives the highest priority to one of the NIDs on the network with the highest priority
• Start traffic
• Stop traffic
• Verify that the NID with highest priority on the network with highest priority was used

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Add UDSP rule that gives the highest priority to one of the NIDs on the network with the lowest priority
• Start traffic
• Stop traffic
• Verify that the NIDs on the network with highest priority were used

• Setup: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• On the peer add UDSP rule that gives the highest priority to the other network
• Start traffic
• Stop traffic
• Verify that the local network with highest priority was used

• Setup: configure two networks, two NIDs on each network 
• Add UDSP rule that gives highest priority to a network that doesn't exist
• Execute UDSP "Show" command
• Verify that the resulting YAML file contains policy description as expected

• Setup: configure two networks, two NIDs on each network 
• Add UDSP rule that gives one of the networks highest priority
• Start traffic
• Verify that the network with the highest priority was used
• Bring the highest priority network down
• Verify that the remaining network is used
• Stop traffic

the test itself passed, but the unconfigure code(at the end of the script) return error

error: "LUTFError:\n  node-name: mds_HOST\n  msg: lustre_rmmod failed\n  arg:\

          \ null\n  file name: /usr/lib64/lustre/tests/lutf/python/tests-infra/lnet.py\n\

          \  line number: 611\n  function: unconfigure\n"

if comment out the unconfigure_lnet(), test can pass

lutf>>> R()

Tests:

  - name: lutf-udsp

    description: auster lutf

    SubTests:

      - status: PASS

        duration: 8

        return_code: 0

        name: udsp_routed_net_01

    duration: 8

    status: PASS

    submission: Fri Apr 22 17:07:56 UTC 2022

• Configure single local network with two NIDs on the network  (RemoteLayout1)
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router and remote peer NIDs as preferred
• Add UDSP rule that gives the same remote peer NID highest priority
• Start traffic
• Stop traffic
• Verify that the preferred remote peer NID and router NID were used

• Configure single local network with two NIDs on the network (RemoteLayout1)
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Assign higher route priority to the route using gateway 1
• Add UDSP rule that designates a pair of a router (gateway 2) and remote peer NIDs as preferred
• Start traffic
• Stop traffic
• Verify that the higher priority route was used (gateway 1)

fail - manually run

fail to set up the env as layout2, I can make the top path passed, but the bottom one wont work

• Configure two local networks (net1, net2) with two NIDs on each network (RemoteLayout2)
• Configure remote peer to have access to the remote network with two NIDs
• On peerA add UDSP rule that designates gateway 1 as preferred for all NIDs on peerB
• Start traffic (peerA to peerB)
• Stop traffic
• Verify that if net1 was used then the paired router was used 

pass with only non-exist router since following error(manually run)

while udsp doesn't support add --src and --rte at the same time

[root@trevis-88vm4 lutf]# lnetctl udsp add --src 10.240.43.83@tcp --rte 10.240.43.80@tcp

add:

    - udsp:

          errno: -1

          descr: "The combination of src, dst and rte is not supported"

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router that doesn't exist and a local network as preferred
• Start traffic
• Stop traffic
• Verify that both routers were used

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router and a local network that doesn't exist as preferred
• Start traffic
• Stop traffic
• Verify that both routers were used

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router that doesn't exist and a remote peer NID as preferred
• Add UDSP rule that gives the same remote peer NID highest priority
• Start traffic
• Stop traffic
• Verify that the all peer NIDs and both routers were used

pass - manually run

UDSP rule

lnetctl udsp add --dst tcp1 --rte 10.240.43.80@tcp

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a pair of a router and a peerB NID as preferred
• Start traffic
• Verify that the preferred router was used
• Bring down the preferred router
• Verify that the remaining router was used
• Stop traffic

pass - manually run

add UDSP rule to interface A

simulate the error by "lctl net_drop_add -s A -d B -i 2"

do lctl ping B from A

check lnetctl net show -v 3 and found send_count is increased on A as well as other interfaces,  the same time, health stats dropped number increased too

delete the rule by "lctl net_drop_del" and do lctl ping again

send_count shows only A increased by 1

*health value didn't been noticed decrease, but developer checked lnet debug log and it showed the changing

• MR Node with Multiple interfaces
• Add UDSP rule that gives one of the local interfaces highest priority
• Send a PING
• Verify that the NID with the highest priority was used
• Send a PING
• Simulate an <error>
• PING msg should be queued on resend queue
• PING msg will be resent on a different interface
• Failed interface's health value will be decremented
• Failed interface will be placed on the recovery queue
• Eventually health score should recover so then only the preferred interface will be used (sensitivity > 0)

pass - manually run

add UDSP on  tcp1

then set that nid A in tcp1 in net_drop, the traffic will go either the good interface on tcp1 or failed. we got failed ping every other time. 

• MR Node with Multiple interfaces: configure two networks, two NIDs on each network
• Add UDSP rule that gives one of the networks highest priority
• Start traffic
• Stop traffic
• Verify that NIDs on the network with the highest priority were used
• Start traffic
• Simulate an <error> (drop on one of the NIDs on the highest priority network)
• Failed interface's health value will be decremented
• Failed interface will be placed on the recovery queue
• Verify that the remaining interface on the failed network is used
• Eventually health score should recover so then only both interface on the preferred network will be used (sensitivity > 0)

• Configure single local network with two NIDs on the network 
• Configure two routers/gateways each providing access to a remote network
• Configure remote peer to have access to the remote network with two NIDs
• Add UDSP rule that designates a router as preferred for all remote peer NIDs 
• Start traffic
• Stop traffic
• Verify that the preferred router was used
• Add a PUT drop rule on the router to drop traffic on one of the interfaces.
• Verify that traffic goes to the other interfaces of the preferred router. There shouldn't be any drop in traffic.
• Add a PUT drop rule on the router to drop traffic on all interfaces.
• Verify that traffic goes through the non-preferred router until one of the interfaces of the preferred router recovers.

pass - manually run

(disable by lnetctl set discovery 0)

• Setup: configure single network, 2 NIDs on the network, peer with MR disabled 
• Add UDSP rule that gives one of the interfaces highest priority
• Start traffic
• Stop traffic
• Verify that the NID with the highest priority was used 

pass - manually run

(interop with a 2.10.8 node which doesn't have MR)

• Setup: configure single network, 2 NIDs on the network, peer incapable of MR ("down-rev")
• Add UDSP rule that gives one of the interfaces highest priority
• Start traffic
• Stop traffic
• Verify that the NID with the highest priority was used 

• Setup: configure single network, 2 NIDs on the network, 2 NIDs on the local peer with MR disabled
• Add UDSP rule that gives the highest priority to one of the local peer NIDs
• Start traffic
• Stop traffic
• Verify that local peer NID with the highest priority was used

• Setup: configure single network, 2 NIDs on the network, 2 NIDs on the local peer incapable of MR ("down-rev")
• Add UDSP rule that gives the highest priority to one of the local peer NIDs
• Start traffic
• Stop traffic
• Verify that local peer NID with the highest priority was used