...
UDSPs are configured from lnetctl via either command line or YAML config files and then passed to the kernel. Policies are applied to all local networks and remote peers then stored in the kernel. During the selection process the policies are examined as part of the selection algorithm.
UDSP Rules Types
Outlined below are the UDSP rule types
- Network rules
- NID rules
- NID Pair rules
- Router rules
Network Rules
These rules define the relative priority of the networks against each other. 0 is the highest priority. Networks with higher priorities will be selected during the selection algorithm, unless the network has no healthy interfaces. If there exists an interface on another network which can be used and its healthier than any which are available on the current network, then that one will be used. Health will always trump all other criteria.
Syntax
| Code Block |
|---|
lnetctl policy add --src *@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src *@o2ib1 --priority 0 |
NID Rules
These rules define the relative priority of individual NIDs. 0 is the highest priority. Once a network is selected the NID with the highest priority is preferred. Note that NID priority is prioritized below health. For example, if there are two NIDs, NID-A and NID-B. NID-A has higher priority but lower health value, NID-B will still be selected. In that sense the policies are there as a hint to guide the selection algorithm.
Syntax
| Code Block |
|---|
lnetctl policy add --src <ip>@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src 10.10.10.2@o2ib1 --priority 1 |
...
Conceptual Overview
UDSPs are used to finely control traffic. In order to achieve this in the most optimal way possible, the policies can not be examined on the fast path, with every message being sent. The policies shall be instantiated on LNet constructs. LNet constructs are: Local Nets, Local NIs, Peer Nets and Peer NIs. Once a policy is instantiated on an LNet construct, meaning specific fields in the LNet construct structure are filled, then these fields are examined in the selection algorithm.
Currently, the policies control the preference of some constructs over others during the selection algorithm.
UDSP Structure
A UDSP consists of two parts:
- The matching criteria
- The matching criteria is used to match an LNet construct against the policy
- Policy action
- The policy action is the action taken on the LNet construct when the policy is matched.
UDSP Rule Types
Network Rules
These rules define the relative priority of pathsthe networks against each other. 0 is the highest priority. Once a destination NID is selected the source NID with the highest priority is selected to send from.
Syntax
| Code Block |
|---|
lnetctl policy add --src <ip>@<net type> --dst <ip>@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src 10.10.10.2@o2ib1 --dst 10.10.10.4@o2ib1 --priority 1 |
Router Rules
Router Rules define which set of routers to use. When defining a network there could be paths which are more optimal than others. To have more control over the path traffic takes, admins configure interfaces on different networks, and split up the router pools among the networks. However, this results in complex configuration, which is hard to maintain and error prone. It is much more desirable to configure all interfaces on the same network, and then define which routers to use when sending to a remote peer. Router Rules allow this functionality
Syntax
...
Networks with higher priorities will be selected during the selection algorithm, unless the network has no healthy interfaces. If there exists an interface on another network which can be used and is healthier than any which are available on the current network, then that one will be used. Health will always trump all other criteria.
Matching Criteria
In order to match a network rule the network type and and expression representing the network number must be provided. Example:
| Code Block |
|---|
tcp1 # match tcp1 exactly
tcp[1-3] # match tcp1, tcp2 and tcp3
tcp* # match any tcp network |
The policy can apply to local or remote network, depending on the specification in the command.
Matching Action
When a network matches the policy the action is applied on the network LNet construct. The only action available is setting the selection priority of the network. When the selection algorithm is iterating through available networks, the one with the highest selection priority is selected.
NID Rules
These rules define the relative priority of individual NIDs. 0 is the highest priority. Once a network is selected the NID with the highest priority is preferred. Note that NID priority is prioritized below health. For example, if there are two NIDs, NID-A and NID-B. NID-A has higher priority but lower health value, NID-B will still be selected. In that sense the policies are there as a hint to guide the selection algorithm.
Matching Criteria
A NID expression is used to match the policy against local or remote NIDs. Example
| Code Block |
|---|
10.10.10.2@tcp1 # match the exact nid
10.10.10.[2-3]@tcp1 # match 10.10.10.2@tcp1 and 10.10.10.3@tcp1
10.10.10.[2-3]@tcp* # match 10.10.10.2 and 10.10.10.3 on any tcp network |
The policy can apply to local or remote NIDs, depending on the specification in the command.
Matching Action
When a NID matches the policy the action specified in the rule is applied to the NI LNet construct. The only action available is setting the selection priority of the NID. When the selection algorithm is iterating through available NIs, the one with the highest selection priority is selected.
NID Pair Rules
These rules define prefered paths. Once a local NI is selected, as this is the first step in the selection algorithm, the peer NI which has the local NI on its preferred list is selected..The end result of this strategy is an association between a local NI and a peer NI (or a group of them)
Matching Criteria
A NID pair rule takes two expressions describing the src and destination NIDs which should be preferred. The matching rules for each of the supplied NID expressions is the same as the NID Rules above.
Matching Action
The remote NIDs available in the system are examined. Each remote NID which matches the destination NID expression in the policy, will have a set of local NIDs, which match the source NID expression in the policy, added to its preferred local NID list. If no local NID matches the source NID expression in the policy, then the action is a no-op.
Router Rules
Router Rules define which set of routers to use when sending messages to a destination NID(s). When defining a network there could be paths which are more optimal than others. To have more control over the path traffic takes, admins configure interfaces on different networks, and split up the router pools among the networks. However, this results in complex configuration, which is hard to maintain and error prone. It is much more desirable to configure all interfaces on the same network, and then define which routers to use when sending to a remote peer. Router Rules allow this functionality
Matching Criteria
A Router rule takes two expressions describing the destination NIDs and the router NIDs which should be preferred when sending to the matching destination NIDs. The matching rules for each of the supplied NID expressions is the same as the NID Rules above.
Matching Action
The remote NIDs available in the system are examined. Each remote NID which matches the destination NID expression in the policy, will have a set of router NIDs, which match the router NID expression in the policy, added to its preferred router NID list. If no router NID matches the router NID expression in the policy, then the action is a no-op.
UDSP application and Selection Priority
UDSP Rule Interactions
UDSP Rules Types
Outlined below are the UDSP rule types
- Network rules
- NID rules
- NID Pair rules
- Router rules
User Interface
Command Line Syntax
Below is the command like syntax for managing UDSPs
...