...

UDSPs are configured from lnetctl via either command line or YAML config files and then passed to the kernel. Policies are applied to all local networks and remote peers then stored in the kernel. During the selection process the policies are examined as part of the selection algorithm.

List of Terms

Conceptual Overview

UDSPs are used to finely control traffic. In order to achieve this in the most optimal way possible, the policies can not be examined on the fast path, with every message being sent. The policies shall be instantiated on LNet constructs. LNet constructs are: Local Nets, Local NIs, Peer Nets and Peer NIs. Once a policy is instantiated on an LNet construct, meaning specific fields in the LNet construct structure are filled, then these fields are examined in the selection algorithm.

...

There is one particular rule which all UDSPs adhere to: Health trumps UDSPs. The selection algorithm always selects the healthiest interface to send from or to send to regardless if there is another network and/or interface which has a higher user assigned Selection Priority. The following diagram gives a simplified overview of the selection algorithm. The HLD linked above is the best place to get more details, as this HLD concentrates on the UDSP design and does not rehash all the details of the selection algorithm. 

The diagram shows the integration of the UDSP provided actions in the logical flow of the selection algorithm.

User Interface

Command Line Syntax

Below is the command like syntax for managing UDSPs

Order of Operations

It is important to define the order of rule operations, when there are multiple rules that apply to the same construct.

The order is defined by the selection algorithm logical flow:

1. iterate over all the networks that a peer can be reached on and select the best local network
   1. The remote network with the highest priority is examined
      1. Network Rule
   2. The local network with the highest priority is selected
      1. Netword Rule
   3. The local NI with the highest priority is selected
      1. NID Rule
2. If the peer is a remote peer and has no local networks,
   1. then select the remote peer network with the highest priority
      1. Network Rule
   2. Select the highest priority remote peer_ni on the network selected
      1. NID Rule
   3. Now that the peer's network and NI are decided select the router in round robin from the peer NI's preferred router list
      1. Router Rule
3. Otherwise for local peers, select the peer_ni from the peer.
   1. highest priority peer NI is selected
      1. NID Rule
   2. Select the peer NI which has the local NI selected on its preferred list.
      1. NID Pair Rule

User Interface

Overview of Operations

There are three main operations which can be carried out on UDSPs either from the command line or YAML configuration: add, delete, show.

Add

The UI allows adding a new rule. With the use of the idx optional parameter, the admin can specifiy where in the rule chain the new rule should be added. By default the rule is appended to the list. Any other value will result in inserting the rule in that position.

When a new UDSP is added the entire UDSP set is re-evaluated. This means all Nets, NIs and peer NIs in the systems are traversed and the rules re-applied. This is an expensive operation, but given that UDSP management should be a rare operation, it shouldn't be a problem.

Delete

The UI allows deleting an existing UDSP using its index. The index can be shown using the show command. When a UDSP is deleted the entire UDSP set are re-evaluated. The Nets, NIs and peer NIs are traversed and the rules re-applied..

Show

The UI allows showing existng UDSPs. The format of the YAML output is as follows:

udsp:
    - idx: <unsigned int>
	  src: <ip>@<net type>
	  dst: <ip>@<net type>
      rte: <ip>@<net type>
	  action:
		- priority: <unsigned int>

Command Line Syntax

Below is the command like syntax for managing UDSPs

# Adding a local network udsp
# if multiple local networks are available, each one can have a priority. 
# The one with the highest priority is 

# Adding a local network udsp
# if multiple local networks are available, each one can have a priority. 
# The one with the highest priority is preferred
lnetctl policy add --src <net type><net number expression>
                   --<action type> <action context sensitive value> 
                   --idx <value>
	--src: is defined in ip2nets syntax. '<net type><expr>' syntax indicates the network.
		   <net type> is one of o2ib, tcp, gni, etc
           <expr> is an ip2nets expression describing a network number.
	--<action type>: 'priority' is the only implemented action type
	--<action context sensitive value>: is a value specific to the action type.
					  For 'priority' it's a value for [0 - 255], with 0 as the highest
                      priority
	--idx: The index of where to insert the rule. If it's larger than the policy list size it's
		   appended to the end of the list. If not specified the default behaviour is to append
		   to the end of the list

# Adding a local NID udsp
# After a local network is chosen, if there are multiple NIs in the network the
# one with highest priority is preferred.
lnetct policy add --src <Address descriptor>@<net type><expr>
                  --<action type> <action context sensitive value>
				  --idx <value>
	--src: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.
	--<action type>: 'priority' is the only implemented action type
	--<action context sensitive value>: is a value specific to the action type.
					  For 'priority' it's a value for [0 - 255], with 0 as the highest
                      priority
	--idx: The index of where to insert the rule. If it's larger than the policy list it's
		   appended to the end of the list. If not specified the default behaviour is to append
		   to the end of the list

# Adding a remote NID udsp
# select the peer NID with the highest priority.
lnetct policy add --dst <Address descriptor>@<net type><expr>
                  --<action type> <action context sensitive value>
				  --idx <value>
	--dst: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.
	--<action type>: 'priority' is the only implemented action type
	--<action context sensitive value>: is a value specific to the action type.
					  For 'priority' it's a value for [0 - 255], with 0 as the highest
                      priority
	--idx: The index of where to insert the rule. If it's larger than the policy list it's
		   appended to the end of the list. If not specified the default behaviour is to append
		   to the end of the list

# Adding a NID pair udsp
# When this rule is instantiated the local NIDs which match the rule are added on a list
# on the peer NIs matching the rule. When selecting the peer NI, the one with the 
# local NID being used on its list is preferred.
lnetct policy add --src <Address descriptor>@<net type><expr>
                  --dst <Address descriptor>@<net type><expr>
				  --idx <value>
	--src: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.
	--dst: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.
           Destination NIDs can be local or remote.
	--idx: The index of where to insert the rule. If it's larger than the policy list it's
		   appended to the end of the list. If not specified the default behaviour is to append
		   to the end of the list

# Adding a Router udsp
# similar to the NID pair udsp. The router NIDs matching the rule are added on a list
# on the peer NIs matching the rule. When sending to a remote peer, the router which
# has its nid on the peer NI list is preferred.
lnetct policy add --dst <Address descriptor>@<net type><expr>
                  --rte <Address descriptor>@<net type><expr>
				  --idx <value>
	--dst: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.--idx <value>
	--rtedst: the address descriptor defined in ip2nets syntax as described in the manual
		   <net type><expr> is similar to what has been described previously.
	--idxrte: Thethe indexaddress ofdescriptor wheredefined toin insertip2nets thesyntax rule.as If it's larger thandescribed in the policy list it'smanual
		   appended<net to the end of the list. If not specified the default behaviour is to append
		  type><expr> is similar to thewhat endhas ofbeen the list


# show all policies in the system.
# the policies are dumped in YAML form.
# Each policy is assigned an index.
# The index is part ofdescribed previously.
	--idx: The index of where to insert the rule. If it's larger than the policy YAMLlist block
lnetctl policy show

#it's
		   appended to deletethe aend policyof the indexlist. mustIf benot specified.
# Thethe normaldefault behaviour then is to append
		  first showto the listend of the policieslist


# grabshow theall index and use it policies in the delete command.
lnetctl policy del --idx <value>
system.
# the generally,policies theare syntaxdumped isin asYAML follows
 lnetctlform.
# Each policy <addis |assigned del | show>
  --src: ip2nets syntax specifying the local NID to match
  --dst: ip2nets syntax specifying the remote NID to match
  --rte: ip2nets syntax specifying the router NID to match
  --priority: Priority to apply to rule matches
  --idx: Index of where to insert the rule. By default it appends to
		 the end of the rule list 

Possible Future Actions

As of the time of this writing only "priority" action shall be implemented. However, it is feasible in the future to implement different actions to be taken when a rule matches. 

Drop

We can implement a "drop" rule. Any message destined to a specific NID or are received from a NID are dropped. Such a policy can be used to drop messages from untrusted peers. This would be a more general solution than the one provided here 

Another possible action is a silent mirror action. All message destined or received from a specific peer are mirrored to a different sink. The intention of this is not for backup, but for debugging traffic, similar in functionality to wireshark. The advantage of this is we can monitor and interpret traffic between specific peers. to to implement "redirect" action, which redirects traffic to another destination. Yet another example is "lawful intercept" or "mirror" action, which mirrors messages to a different destination. This might be useful for keeping a standby server updated with all information going to the primary server. A lawful intercept action allows personnel authorized by a Law Enforcement Agency (LEA) to intercept file operations from targeted clients and send the file operations to an LI Mediation Device.

...

udsp:
    - idx: <unsigned int>
	  src: <ip>@<net type>
	  dst: <ip>@<net type>
      rte: <ip>@<net type>
	  action:
		- priority: <unsigned int>

Overview of Operations

There are three main operations which can be carried out on UDSPs either from the command line or YAML configuration: add, delete, show.

Add

The UI allows adding a new rule. With the use of the idx optional parameter, the admin can specifiy where in the rule chain the new rule should be added. By default the rule is appended to the list. Any other value will result in inserting the rule in that position.

When a new UDSP is added the entire UDSP set is re-evaluated. This means all Nets, NIs and peer NIs in the systems are traversed and the rules re-applied. This is an expensive operation, but given that UDSP management should be a rare operation, it shouldn't be a problem.

Delete

The UI allows deleting an existing UDSP using its index. The index can be shown using the show command. When a UDSP is deleted the entire UDSP set are re-evaluated. The Nets, NIs and peer NIs are traversed and the rules re-applied..

Show

...

an index.
# The index is part of the policy YAML block
lnetctl policy show

# to delete a policy the index must be specified.
# The normal behaviour then is to first show the list of policies
# grab the index and use it in the delete command.
lnetctl policy del --idx <value>

# generally, the syntax is as follows
 lnetctl policy <add | del | show>
  --src: ip2nets syntax specifying the local NID to match
  --dst: ip2nets syntax specifying the remote NID to match
  --rte: ip2nets syntax specifying the router NID to match
  --priority: Priority to apply to rule matches
  --idx: Index of where to insert the rule. By default it appends to
		 the end of the rule list 

Possible Future Actions

As of the time of this writing only "priority" action shall be implemented. However, it is feasible in the future to implement different actions to be taken when a rule matches. 

Drop

We can implement a "drop" rule. Any message destined to a specific NID or are received from a NID are dropped. Such a policy can be used to drop messages from untrusted peers. This would be a more general solution than the one provided here 

Mirror

All message destined or received from a specific peer are mirrored to a different sink. The intention of this is not for backup, but for debugging traffic, similar in functionality to wireshark. Lustre clients are stateful so using such a feature wouldn't work for keeping a backup server uptodate. However, it would be useful for monitoring traffic. There is an extension to wireshark which is available to interpret dumps captured by utilities such as tcpdump or ibdump. However, such utilities capture all traffic, which makes interpreting traffic extremely hard. Mirroring would allow us to isolate and capture traffic between specific peers. This policy can be applied on a server or a router. 

Redirect

Redirect messages to from a specific NID (or group of NIDs) to a different server. This can be applied on routers for example or even servers, to intercept traffic.

Anchor
YAMLSyntax YAMLSyntax

YAML Syntax

udsp:
    - idx: <unsigned int>
	  src: <ip>@<net type>
	  dst: <ip>@<net type>
      rte: <ip>@<net type>
	  action:
		- priority: <unsigned int>

Design

All policies are stored in kernel space. All logic to add, delete and match policies will be implemented in kernel space. This complicates the kernel space processing. Arguably, policy maintenance logic is not core to LNet functionality. What is core is the ability to select source and destination networks and NIDs in accordance with user definitions. However, the kernel is able to manage policies much easier and with less potential race conditions than user space.

Design Principles

UDSPs are comprised of two parts:

1. The matching rule
2. The rule action

The matching rule is what's used to match a NID or a network. The action is what's applied when the rule is matched.

A rule can be uniquely identified using an internal ID which is assigned by the LNet module when a rule is added and returned to the user space when the UDSPs are shown.

UDSP Storage

UDSPs shall be defined by administrators either via LNet command line utility, lnetctl, or via YAML configuration file.  lnetctl parses the UDSP and stores it in an intermediary format, which will be flattened and passed down to the kernel LNet module. LNet shall store these UDSPs on a policy list. Once policies are added to LNet they will be applied on existing networks, NIDs and routers. The advantage of this approach is that UDSPs are not strictly tied to the internal constructs, IE networks, NIDs or routers, but can be applied whenever the internal constructs are created and if the internal constructs are deleted then they remain and can be automatically applied at a future time.

This makes configuration easy since a set of UDSPs can be defined, like "all IB networks priority 1", "all Gemini networks priority 2", etc, and when a network is added, it automatically inherits these rules.

Peers are normally not created explicitly by the administrators. The ULP requests to send a message to a peer or the node receives an unsolicited message from a peer which results in creating a peer construct in LNet. It is feasible, especially for router policies, to have a UDSP which associates a set of clients with in a specific range with a set of optimal routers. Having the policies stored and matched in kernel aids in fulfilling this requirement.

UDSP Application

Performance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "flattened" to the constructs they impact. For example, a Network Rule is added as follows: o2ib priority 0. This rule gives priority for using o2ib network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every send to see if it we get any matches.

...

It is important to define the order of rule operations, when there are multiple rules that apply to the same construct.

The order is defined by the selection algorithm logical flow:

...

1. The remote network with the highest priority is examined
   1. Network Rule
2. The local network with the highest priority is selected
   1. Netword Rule
3. The local NI with the highest priority is selected
   1. NID Rule

...

1. then select the remote peer network with the highest priority
   1. Network Rule
2. Select the highest priority remote peer_ni on the network selected
   1. NID Rule
3. Now that the peer's network and NI are decided select the router in round robin from the peer NI's preferred router list
   1. Router Rule

UDSPs add/del/show commands are parsed in user space and passed down in structure form to the LNet kernel module. All policies are stored in kernel space. All logic to add, delete and match policies will be implemented in kernel space. This complicates the kernel space processing. Arguably, policy maintenance logic is not core to LNet functionality. What is core is the ability to select source and destination networks and NIDs in accordance with user definitions. However, the kernel is able to manage policies much easier and with less potential race conditions than user space.

Design Principles

UDSPs are comprised of two parts:

1. The UDSP matching criteria
2. The UDSP action

The matching criteria is what's used to match an LNet construct. The action is what's applied on the LNet construct when the rule is matched.

A rule can be uniquely identified using an internal ID which is assigned by the LNet module when a rule is added and returned to the user space when the UDSPs are shown.

UDSP Storage

UDSPs shall be defined by administrators either via LNet command line utility, lnetctl, or via YAML configuration file.  lnetctl parses the UDSP and stores it in an intermediary format, which will be marshalled and passed down to the kernel LNet module. LNet shall store these UDSPs on a policy list. Once policies are added to LNet they will be applied on existing networks, NIDs and routers. The advantage of this approach is that UDSPs are not strictly tied to the LNet constructs, IE networks, NIDs or routers, but can be applied whenever the LNet constructs are created and if the LNet constructs are deleted then they remain and can be automatically applied at a future time.

This makes configuration easy since a set of UDSPs can be defined, like "all IB networks priority 1", "all Gemini networks priority 2", etc, and when a network is added, it automatically inherits these rules.

Peers are normally not created explicitly by the administrators. The ULP requests to send a message to a peer or the node receives an unsolicited message from a peer which results in creating a peer construct in LNet. It is feasible, especially for router policies, to have a UDSP which associates a set of clients with in a specific range with a set of optimal routers. Having the policies stored and matched in kernel aids in fulfilling this requirement.

UDSP Instantiation

Performance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "instantiated" on the LNet constructs they impact. For example, a Network Rule is added as follows: lnetctl policy add --src o2ib0 --priority 0. This rule gives priority for using o2ib0 network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib0 network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every send to see if it we get any matches.

...

1. NID Rule

...

Kernel Design

Anchor
InKernelStructures InKernelStructures

In Kernel Structures

...