...

UDSPs are configured from lnetctl via either command line or YAML config files and then passed to the kernel. Policies are applied to all local networks and remote peers then stored in the kernel. During the selection process the policies are examined as part of the selection algorithm. Policies will be the top most priority in the selection process since it is user defined. The rest of the selection criteria will be applied on the subset of interfaces which match the policies.

UDSP Rules Types

Outlined below are the UDSP rule types

...

These rules define the relative priority of the networks against each other. 0 is the highest priority. Networks with higher priorities will be selected during the selection algorithm, unless the network has no healthy interfaces. If there exists an interface on another network which can be used and its healthier than any which are available on the current network, then that one will be used. Health will always trump all other criteria.

Syntax

lnetctl policy add --src *@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src *@o2ib1 --priority 0

NID Rules

These rules define the relative priority of individual NIDs. 0 is the highest priority. Once a network is selected the NID with the highest priority is preferred. Note that NID priority is prioritized below health. For example, if there are two NIDs, NID-A and NID-B. NID-A has higher priority but lower health value, NID-B will still be selected. In that sense the policies are there as a hint to guide the selection algorithm.

Syntax

NID Pair Rules

lnetctl policy add --src <ip>@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src 10.10.10.2@o2ib1 --priority 1 

NID Pair Rules

These These rules define the relative priority of paths. 0 is the highest priority. Once a destination NID is selected the source NID with the highest priority is selected to send from.

Syntax

Net Pair Rules

lnetctl policy add --src <ip>@<net type> --dst <ip>@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src 10.10.10.2@o2ib1 --dst 10.10.10.4@o2ib1 --priority 1 

Net Pair Rules

Net Pair Rules is a generalization of the NID Pair Rules. It attempts to give Net Pair Rules is a generalization of the NID Pair Rules. It attempts to give a priority for all NIDs on two different networks. This can be done by using ip2nets format while defining the NID Pair Rules. For example *@tcp → *@o2ibused to control traffic heading to a remote network

Syntax

lnetctl policy add --src *@<net type> --dst *@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --src *@o2ib1 --dst *@o2ib2 --priority 0

Router Rules

Router Rules define which set of routers to use. When defining a network there could be paths which are more optimal than others. To have more control over the path traffic takes, admins configure interfaces on different networks, and split up the router pools among the networks. However, this results in complex configuration, which is hard to maintain and error prone. It is much more desirable to configure all interfaces on the same network, and then define which routers to use when sending to a remote peer. Router Rules alow this functionalityand then define which routers to use when sending to a remote peer. Router Rules allow this functionality

Syntax

lnetctl policy add --dst <ip>@<net type> --rte <ip>@<net type> --<action-type> <context dependent value>
ex:
lnetctl policy add --dst 10.10.10.2@o2ib3 --rte 10.10.10.[5-8]@o2ib --priority 0

User Interface

As illustrated in the example above all policies can be specified using the following syntax:

lnetctl policy <add | del | show>
  --src: ip2nets syntax specifying the local NID to match
  --dst: ip2nets syntax specifying the remote NID to match
  --rte: ip2nets syntax specifying the router NID to match
  --priority: Priority to apply to rule matches

As of the time of this writing only "priority" action shall be implemented. However, it is feasible in the future to implement different actions to be taken when a rule matches. For example, we can implement a "redirect" action, which redirects traffic to another destination. Yet another example is an "lawful intercept" or "mirror" action, which mirrors messages to a different destination, this might be useful for keeping a standby server updated with all information going to the primary server. An lawful intercept action allows personnel authorized by a Law Enforcement Agency (LEA) to intercept file operations from targeted clients and send the file operations to an LI Mediation Device.

Design Approach

There are two ways to implement this feature; in kernel space or in user space.

Kernel Space Design

All policies are stored in kernel space. All logic to add, delete and match policies will be implmented in user space. This complicates the kernel space processing. Arguably, policy maintenance logic is not core to LNet functionality. What is core is the ability to select source and destination networks and NIDs in accordance with user definitions.

User space Design

In this approach all the policy parsing and storage happens in user space. The kernel API is kept very simple. It'll just provide a way to set actions on LNet constructs, IE networks, NIDs and routers. A user space utility, lnetctl, will parse policies and store them in a persistent file. Whenver lnetctl is used to add a net or a peer explicitly, then it will read the policy file and apply policy actions on construct being created in the kernel.

However, since the kernel has no concept of policies, if peers are created internally, as they usually are, then these policies can not be applied. One method to get around this issue to implement uevents. Whenever a peer is created a uevent is fired to userspace. This uevent results in the policy file being read and if there is a match against one of the policies in the file, then the peer is updated appropriately. There is a period of time between the creation of the peer and the handling of the uevent in userspace where the policy is not applied and therefore traffic patterns might not conform to user specified behavior until the peer is updated.

Design Principles

Rule Storage

...

Performance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "flattened" to the constructs they impact. For example, a Network Rule is added as follows: o2ib priority 0. This rule gives priority for using o2ib network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every network send to see if it matches or notany networks.

Rule Structure

Selection policy rules are comprised of two parts:

...

/* This is a common structure which describes an expression */
struct lnet_match_expr {
    __u32   lme_start;
    __u32   lme_end;
    __u32   lme_incr;
    char    lme_r_expr[0];
};

struct lnet_selection_descriptor {
    enum selection_type lsd_type;
    char                *lsd_pattern1;
    char                *lsd_pattern2;

    union {
        __u32           lsda_priority;
    } lsd_action_u;
};

/*
 * lustre_lnet_add_selection
 *   Delete the peer NIDs. If all peer NIDs of a peer are deleted
 *   then the peer is deleted
 *
 *   selection - describes the selection policy rule
 *   seq_no - sequence number of the command
 *   err_rc - YAML structure of the resultant return code
 */
int lustre_lnet_add_selection(struct selection_descriptor *selection, int seq_no, struct cYAML **er_rc);

...