Page History

...

As of the time of this writing only "priority" action shall be implemented. However, it is feasible in the future to implement different actions to be taken when a rule matches. For example, we can implement a "redirect" action, which redirects traffic to another destination. Yet another example is an "lawful intercept" or "mirror" action, which mirrors messages to a different destination, this . This might be useful for keeping a standby server updated with all information going to the primary server. An A lawful intercept action allows personnel authorized by a Law Enforcement Agency (LEA) to intercept file operations from targeted clients and send the file operations to an LI Mediation Device.

...

All policies are stored in kernel space. All logic to add, delete and match policies will be implemented in kernel space. This complicates the kernel space processing. Arguably, policy maintenance logic is not core to LNet functionality. What is core is the ability to select source and destination networks and NIDs in accordance with user definitions. However, the kernel is able to manage policies much easier and with less potential race conditions than user space.

Design Principles

Rule Storage

UDSPs are comprised of two parts:

1. The matching rule
2. The rule action

The matching rule is what's used to match a NID or a network. The action is what's applied when the rule is matched.

A rule can be uniquely identified using an internal ID which is assigned by the LNet module when a rule is added and returned to the user space when the UDSPs are shown.

UDSP Storage

UDSPs Rules shall be defined by administrators either via LNet command line utility, lnetctl, or via YAML configuration file.  lnetctl parses the UDSP and stores it in an intermediary format, which will be flattened and passed down to the kernel LNet module. LNet shall store these rules UDSPs on a policy list. Once policies are added to LNet they will be applied on existing networks, NIDs and routers. The advantage of this approach is that rules UDSPs are not strictly tied to the internal constructs, IE networks, NIDs or routers, but can be applied whenever the internal constructs are created and if the internal constructs are deleted then they remain and can be automatically applied at a future time.

This makes configuration easy since a set of rules UDSPs can be defined, like "all IB networks priority 1", "all Gemini networks priority 2", etc, and when a network is added, it automatically inherits these rules.

Peers are normally not created explicitly by the administrators. The ULP requests to send a message to a peer or the node receives an unsolicited message from a peer which results in creating a peer construct in LNet. It is feasible, especially for router policies, to have a UDSP which associates a set of clients with in a specific range with a set of optimal routers. Having the policies stored and matching occur matched in kernel aids in fulfilling this requirement.

...

UDSP Application

Performance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "flattened" to the constructs they impact. For example, a Network Rule is added as follows: o2ib priority 0. This rule gives priority for using o2ib network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every send to see if it we get any matches.

...

/* lnet structure will keep a list of UDSPs */
struct lnet {
	...
	list_head ln_udsp_list;
	...
}

/* each NID range is defined as net_id and an ip range */
struct lnet_ud_nid_descr {
	__u32 ud_net_id;
	list_head ud_ip_range;
}

/* UDSP action types */
enum lnet_udsp_action_type udsp_action_type {
	EN_LNET_UDSP_ACTION_PRIORITY = 0,
	EN_LNET_UDSP_ACTION_NONE = 1,
}

 /*
 * a UDSP rule can have up to three user defined NID descriptors
 * 		- src: defines the local NID range for the rule
 * 		- dst: defines the peer NID range for the rule
 * 		- rte: defines the router NID range for the rule
 *
 * An action union defines the action to take when the rule
 * is matched
 */ 
struct lnet_udsp {
	list_head udsp_on_list;
	__u32 idx;
	lnet_ud_nid_descr *udsp_src;
	lnet_ud_nid_describe *udsp_dst;
	lnet_ud_nid_descr *udsp_rte;
	enum lnet_udsp_action_type udsp_action_type;
	union udsp_action {
		__u32 udsp_priority;
	};
}

/* The rules are flattened in the LNet structures as shown below */
struct lnet_net {
...
	/* defines the relative priority of this net compared to others in the system */
	__u32 net_priority;
...
}

struct lnet_ni {
...
	/* defines the relative priority of this NI compared to other NIs in the net */
	__u32 ni_priority;
...
}

struct lnet_peer_ni {
...
	/* defines the relative peer_ni priority compared to other peer_nis in the peer */
	__u32 lpni_priority;

	/* defines the list of local NID(s) (>=1) which should be used as the source */
	union lpni_pref {
		lnet_nid_t nid;
		lnet_nid_t *nids;
	}

	/* defines the list of router NID(s) to be used when sending to this peer NI */
	lnet_nid_t *lpni_rte_nids;
...
}

/* UDSPs will be passed to the kernel via IOCTL */
#define IOC_LIBCFS_ADD_UDSP _IOWR(IOC_LIBCFS_TYPE, 106, IOCTL_CONFIG_SIZE)

/* UDSP will be grabbed from the kernel via IOCTL
#define IOC_LIBCFS_GET_UDSP _IOWR(IOC_LIBCFS_TYPE, 106, IOCTL_CONFIG_SIZE)

...

Userspace Structures

Marshaled Structures

Kernel Structure

Defined here.

Selection policy rules are comprised of two parts:

1. The matching rule
2. The rule action

The matching rule is what's used to match a NID or a network. The action is what's applied when the rule is matched.

/* each NID range is defined as net_id and an ip range */
struct lnet_ud_nid_descr {
	__u32 ud_net_id;
	list_head ud_ip_range;
}

/* UDSP action types */
enum lnet_udsp_action_type {
	EN_LNET_UDSP_ACTION_PRIORITY = 0,
	EN_LNET_UDSP_ACTION_NONE = 1,
}

 /*
 * a UDSP rule can have up to three user defined NID descriptors
 * 		- src: defines the local NID range for the rule
 * 		- dst: defines the peer NID range for the rule
 * 		- rte: defines the router NID range for the rule
 *
 * An action union defines the action to take when the rule
 * is matched
 */ 
struct lnet_udsp {
	list_head udsp_on_list;
	__u32 idx;
	lnet_ud_nid_descr *udsp_src;
	lnet_ud_nid_describe *udsp_dst;
	lnet_ud_nid_descr *udsp_rte;
	enum lnet_udsp_action_type udsp_action_type;
	union udsp_action {
		__u32 udsp_priority;
	};
}

Marshaled Structures

struct cfs_range_expr {
	struct list_head re_link;
	__u32 re_lo;
	__u32 re_hi;
	__u32 re_stride;
};
	
struct lnet_ioctl_udsp {
	__u32 iou_idx;
	enum lnet_udsp_action_type iou_action_type
	union action iou_action {
		__u32 priority;
	}
	__u32 iou_src_dot_expr_count;
	__u32 iou_dst_dot_expr_count;
	__u32 iou_rte_dot_expr_count;
	char iou_bulk[0];
};

The address is expressed as a list of cfs_range_expr. These need to be marshalled. For IP address there are 4 of these structures. Other type of addresses can have a different number. As an example, gemini will only have one. The corresponding iou_[src|dst|rte]_dot_expr_count is set to the number of expressions describing the address. Each expression is then flattened in the structure. They have to be flattened in the order defined: SRC, DST, RTE.

The kernel will recieve the marshalled data and will form its internal structures. The functions to marshal and de-marshal should be straight forward. Note that user space and kernel space use the same structures. These structure will be defined in a common location. For this reason the functions to marshal and de-marshal will be shared.

Kernel Structure

Defined hereA rule can be uniquely identified using the matching rule or an internal ID which assigned by the LNet module when a rule is added and returned to the user space when they are returned as a result of a show command.

Structures

/* This is a common structure which describes an expression */
struct lnet_match_expr {

};

struct lnet_selection_descriptor {
    enum selection_type lsd_type;
    char                *lsd_pattern1;
    char                *lsd_pattern2;

    union {
        __u32           lsda_priority;
    } lsd_action_u;
};

/*
 * lustre_lnet_add_selection
 *   Delete the peer NIDs. If all peer NIDs of a peer are deleted
 *   then the peer is deleted
 *
 *   selection - describes the selection policy rule
 *   seq_no - sequence number of the command
 *   err_rc - YAML structure of the resultant return code
 */
int lustre_lnet_add_selection(struct selection_descriptor *selection, int seq_no, struct cYAML **er_rc);

...