Overview

The Unit Test Plan (UTP) will follow the same section breakdown as the Requirements in the Scope & Requirement Document [1].

The following types of tests shall be included where it makes sense.

In-Range UT - these are the test cases which cover normal operations.
Out-of-Range UT - these are the test cases which cover out of range scenarios:
1. border cases
2. race conditions
3. unexpected events
  1. EX: Tearing down an active Network Interface
Error UT
1. Error parameters
2. Error Conditions
  1. network goes down unexpectedly
  2. Wire gets disconnected, etc

Performance Testing cases will be a separate section in this document.

Unit Test Plan

Configuration tests should be done through the DLC direct interface, as well as the YAML interface.

Basic API

This section lists unit test cases that verify basic functionality, i.e. adding, deleting and showing UDSP rules. These tests focus on UDSP API and don't generate any traffic. The tests are structured as follows:

Configuration
Dump UDSP rules and verify result (optional)
Add or Delete UDSP (command line or YAML)
Dump UDSP rules and verify result
Go to 3 (optional).
Clean up.

Configuration

UDSP rules can be created at any time, but are applied only if the node they are configured for is considered to be "reachable". Run discovery command on peers as part of configuration.

Expected Behavior

Basic UT

Primary Requirement ID	Secondary Requirement ID	Unit Test Description
cfg-095	cfg-160	Setup: configure single network, 3 NIDs on the network Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
cfg-100	cfg-115, cfg-125	Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected
cfg-125		Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Add UDSP rule that lowers the priority of one of the NIDs with highest priority back to default Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected
cfg-110		Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Add UDSP rule that gives the highest priority to one of the NIDs on the network with the highest priority Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected (for network and NID) Delete UDSP rule that gives the highest priority to one of the NIDs Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected (for network) Delete UDSP rule that gives the highest priority to one of the networks Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
cfg-115		Setup: configure single network, 3 NIDs on the network locally, 3 NIDs on the remote peer Add UDSP rule that gives priority to one of the remote peer's NIDs Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Delete UDSP rule that gives the highest priority to the remote peer's NID Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
cfg-120		Setup: configure single network, 3 NIDs on the network locally, 3 NIDs on the local peer Add UDSP rule that gives priority to a pair of the local NID and peer's NID Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Delete UDSP rule that gives the highest priority to the NID pair Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
		Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and remote peer NID as preferred Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Delete UDSP rule that gives the highest priority to the router/remote NID pair Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
		Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and local network as preferred Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Delete UDSP rule that gives the highest priority to the router/local network pair Execute UDSP "Show" command Verify that the resulting YAML file doesn't contain any policy description
TODO: add N:1, 1:N, N:N mapping tests
TODO: update requirements to cover router-related rules

Error UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
cfg-105		basic-cfg-err-001	Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected Execute UDSP "Delete" command providing policy index that doesn't exist Verify that the command failed with an error

Issues

Unit Test ID	Description	Status
basic-cfg-err-001	"add" command with bad syntax yields "Success" description in the result along with errno -22	fix ready
basic-cfg-err-001	"del" command with "idx" parameter value that doesn't match any existing policy yields no error	ignore
basic-cfg-err-001	"del" command with bad syntax ("idxx" instead of "idx") causes info for the "add" command to be printed	fix ready
basic-cfg-err-001	"add" command that duplicates existing policy yields "cannot add udsp: Unknown error -114"	fix ready
basic-cfg-err-001	"add" command that duplicates existing policy except assigning a (different) priority yields a duplicated policy with the specified priority	fix ready

Single Network Configuration

Expected Behavior

In-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives one of the interfaces highest priority Start traffic TODO: need to specify direction? Stop traffic Verify that the NID with the highest priority was used
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Start traffic Stop traffic Verify that two NIDs with the highest priority were used
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Start traffic Stop traffic Verify that two NIDs with the highest priority were used Add UDSP that lowers the priority of one of the NIDs with highest priority back to default Start traffic Stop traffic Verify that the remaining highest-priority NID was used
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives two of the interfaces highest priority Start traffic Stop traffic Verify that two NIDs with the highest priority were used Add UDSP that lowers the priority of both of the NIDs with the highest priority back to default Start traffic Stop traffic Verify that all NIDs were used
			Setup: configure single network, 3 NIDs on the network, 3 NIDs on the local peer Add UDSP rule that gives the highest priority to one of the local peer NIDs Start traffic Stop traffic Verify that local peer NID with the highest priority was used
		singlenet-006	Setup: configure single network, 3 NIDs on the network, 3 NIDs on the local peer Add UDSP rule that creates 3 NID pairs such that one of the peer NIDs is not in any pair Start traffic Stop traffic Verify that the peer NID that is not part of any pair was not used (less used?) Delete UDSP rule Start traffic Stop traffic Verify that all NIDs are used
			Setup: configure single network, 3 NIDs on the network Start traffic Verify that all NIDs were used Add UDSP rule that gives one of the interfaces highest priority Verify that the NID with the highest priority was used Stop traffic
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives one of the interfaces highest priority On the peer, add UDSP rule that gives one of the peer interfaces highest priority Start traffic Stop traffic Verify that the NIDs with the highest priority were used on both sides, except for responses TODO: can this be verified?

Out-of-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives highest priority to a NID that doesn't exist Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected
			Setup: configure single network, 3 NIDs on the network Add UDSP rule that gives one of the interfaces highest priority Start traffic Verify that the NID with the highest priority was used Bring the highest priority NID down Verify that remaining NIDs are used Stop traffic

Issues

Unit Test ID	Description	Status
singlenet-006	NID pair UDSP doesn't appear to be taking effect	under investigation
singlenet-006	Deleting NID-pair UDSP that specifies multiple source NIDs causes kernel crash	under investigation

Multiple Network Configuration

Expected Behavior

In-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Start traffic Stop traffic Verify that NIDs on the network with the highest priority were used
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Start traffic Stop traffic Verify that NIDs on the network with the highest priority were used Add UDSP rule that reverses networks priorities Start traffic Stop traffic Verify that NIDs on the new network that has the highest priority were used
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Add UDSP rule that gives the highest priority to one of the NIDs on the network with the highest priority Start traffic Stop traffic Verify that the NID with highest priority on the network with highest priority was used
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Add UDSP rule that gives the highest priority to one of the NIDs on the network with the lowest priority Start traffic Stop traffic Verify that the NIDs on the network with highest priority were used
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority On the peer add UDSP rule that gives the highest priority to the other network Start traffic Stop traffic Verify that both networks were used

Out-of-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives highest priority to a network that doesn't exist Execute UDSP "Show" command Verify that the resulting YAML file contains policy description as expected
			Setup: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Start traffic Verify that the network with the highest priority was used Bring the highest priority network down Verify that the remaining network is used Stop traffic

Issues

Unit Test ID	Description	Status
LU-13193	"Oops" when adding a destination nid UDSP	DNS

Routed Network Configuration

Expected Behavior

In-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and remote peer NIDs as preferred Add UDSP rule that gives the same remote peer NID highest priority Start traffic Stop traffic Verify that the preferred remote peer NID and router NID were used
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and local network as preferred Start traffic Stop traffic Verify that the preferred router was used
			Configure two local networks (net1, net2) with two NIDs on each network Configure two routers/gateways per local network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and local network (net1) as preferred Start traffic Stop traffic Verify that if net1 was used then the paired router was used

Out-of-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router that doesn't exist and a local network as preferred Start traffic Stop traffic Verify that both routers were used
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and a local network that doesn't exist as preferred Start traffic Stop traffic Verify that both routers were used
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router that doesn't exist and a remote peer NID as preferred Add UDSP rule that gives the same remote peer NID highest priority Start traffic Stop traffic Verify that the all peer NIDs and both routers were used
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and local network as preferred Start traffic Verify that the preferred router was used Bring down the preferred router Verify that the remaining router was used Stop traffic

Interoperability with LNet Health feature

Tests cases listed in this section are intended to verify that UDSP and LNet Health features work properly when both are enabled. The expected behaviour is that UDSP decisions come second to the health-based decisions, such that, for example, given healthier alternatives, NID with a lower health score won't be chosen even if it is preferred according to a UDSP rule. More detail on how to simulate events that affect the health score can be found here: LNet Health Test Plan [3].

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			MR Node with Multiple interfaces Add UDSP rule that gives one of the local interfaces highest priority Send a PING Verify that the NID with the highest priority was used Send a PING Simulate an <error> PING msg should be queued on resend queue PING msg will be resent on a different interface Failed interface's health value will be decremented Failed interface will be placed on the recovery queue Eventually health score should recover so then only the preferred interface will be used (sensitivity > 0)
			MR Node with Multiple interfaces: configure two networks, two NIDs on each network Add UDSP rule that gives one of the networks highest priority Start traffic Stop traffic Verify that NIDs on the network with the highest priority were used Start traffic Simulate an <error> (drop on one of the NIDs on the highest priority network) Failed interface's health value will be decremented Failed interface will be placed on the recovery queue Verify that the remaining interface on the failed network is used Eventually health score should recover so then only both interface on the preferred network will be used (sensitivity > 0)
			Configure single local network with two NIDs on the network Configure two routers/gateways each providing access to a remote network Configure remote peer to have access to the remote network with two NIDs Add UDSP rule that designates a pair of a router and local network as preferred Start traffic Stop traffic Verify that the preferred router was used Add a PUT drop rule on the router to drop traffic on one of the interfaces. Verify that traffic goes to the other interfaces of the preferred router. There shouldn't be any drop in traffic. Add a PUT drop rule on the router to drop traffic on all interfaces. Verify that traffic goes through the non-preferred router until one of the interfaces of the preferred router recovers.

Compatibility with Non-MR Nodes

Expected Behavior

In-Range UT

Primary Requirement ID	Secondary Requirement ID	Unit Test ID	Unit Test Description
			Setup: configure single network, 2 NIDs on the network, peer with MR disabled Add UDSP rule that gives one of the interfaces highest priority Start traffic Stop traffic Verify that the NID with the highest priority was used
			Setup: configure single network, 2 NIDs on the network, peer incapable of MR ("down-rev") Add UDSP rule that gives one of the interfaces highest priority Start traffic Stop traffic Verify that the NID with the highest priority was used
			Setup: configure single network, 2 NIDs on the network, 2 NIDs on the local peer with MR disabled Add UDSP rule that gives the highest priority to one of the local peer NIDs Start traffic Stop traffic Verify that local peer NID with the highest priority was used
			Setup: configure single network, 2 NIDs on the network, 2 NIDs on the local peer incapable of MR ("down-rev") Add UDSP rule that gives the highest priority to one of the local peer NIDs Start traffic Stop traffic Verify that local peer NID with the highest priority was used

Functional Requirements

Performance Requirements

Glossary

Term	Meaning
LNet
UDSP	User Defined Selection Policy
UTP	Unit Test Plan
DLC	Dynamic LNet Config (e.g. let, lnetctl scripts)
NI	Network Interface
RPC	Remote Procedure Call
FS	File System
o2ib	Infiniband Network
tcp	Ethernet based Network
NUMA	Non-Uniform Memory Access
RR	Round Robin
CPT	CPU Partition
CB	Channel Bonding
NID	Network Identifier

References

#	Title
1	UDSP Scope and Requirements
2	Multi-Rail Scope and Requirements Document
3	LNet Health Test Plan

Space shortcuts

Page tree

Overview

Unit Test Plan

Basic API

Configuration

Expected Behavior

Basic UT

Error UT

Issues

Single Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Issues

Multiple Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Issues

Routed Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Interoperability with LNet Health feature

Compatibility with Non-MR Nodes

Expected Behavior

In-Range UT

Functional Requirements

Performance Requirements

Glossary

References

Space shortcuts

Page tree

UDSP Unit Test Plan

Overview

Unit Test Plan

Basic API

Configuration

Expected Behavior

Basic UT

Error UT

Issues

Single Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Issues

Multiple Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Issues

Routed Network Configuration

Expected Behavior

In-Range UT

Out-of-Range UT

Interoperability with LNet Health feature

Compatibility with Non-MR Nodes

Expected Behavior

In-Range UT

Functional Requirements

Performance Requirements

Glossary

References