...
As of the time of this writing only "priority" action shall be implemented. However, it is feasible in the future to implement different actions to be taken when a rule matches. For example, we can implement a "redirect" action, which redirects traffic to another destination. Yet another example is an "lawful intercept" or "mirror" action, which mirrors messages to a different destination, this might be useful for keeping a standby server updated with all information going to the primary server. An lawful intercept action allows personnel authorized by a Law Enforcement Agency (LEA) to intercept file operations from targeted clients and send the file operations to an LI Mediation Device.
Design Approach Overview
There are two ways to implement this feature; in kernel space or in user space.
...
All policies are stored in kernel space. All logic to add, delete and match policies will be implemented in user kernel space. This complicates the kernel space processing. Arguably, policy maintenance logic is not core to LNet functionality. What is core is the ability to select source and destination networks and NIDs in accordance with user definitions.
...
Design
...
Principles
Rule Storage
Rules shall be defined through user space and passed to the LNet module. LNet shall store these rules on a policy list. Once policies are added to LNet they will be applied on existing networks, NIDs and routers. The advantage of this approach is that rules are not strictly tied to the internal In this approach all the policy parsing and storage happens in user space. The kernel API is kept very simple. It'll just provide a way to set actions on LNet constructs, IE networks, NIDs and or routers. A user space utility, lnetctl, will parse policies and store them in a persistent file. Whenever lnetctl is used to add a net or a peer explicitly, then it will read the policy file and apply policy actions on construct being created in the kernel.
However, since the kernel has no concept of policies, if peers are created internally, as they usually are, then these policies can not be applied. One method to get around this issue to implement uevents. Whenever a peer is created a uevent is fired to user space. This uevent results in the policy file being read and if there is a match against one of the policies in the file, then the peer is updated appropriately. There is a period of time between the creation of the peer and the handling of the uevent in user space where the policy is not applied and therefore traffic patterns might not conform to user specified definition until the peer is updated.
Design Principles
Rule Storage
Rules shall be defined through user space and passed to the LNet module. LNet shall store these rules on a policy list. Each rule type will have its own list. Once policies are added to LNet they will be applied on existing networks, NIDs and routers. The advantage of this approach is that rules are not strictly tied to the internal constructs, IE networks, NIDs or routers, but can be applied whenever the internal constructs are created and if the internal constructs are deleted then they remain and can be automatically applied at a future time.
This makes configuration easy since a set of rules can be defined, like "all IB networks priority 1", "all Gemini networks priority 2", etc, and when a network is added, it automatically inherits these rules.
Rule Application
, but can be applied whenever the internal constructs are created and if the internal constructs are deleted then they remain and can be automatically applied at a future time.
This makes configuration easy since a set of rules can be defined, like "all IB networks priority 1", "all Gemini networks priority 2", etc, and when a network is added, it automatically inherits these rules.
Peers are normally not created explicitly by the admin. The ULP requests to send a message to a peer or the node receives an unsolicited message from a peer which results in creating a peer construct in LNet. It is feasible, especially for router policies, where we want to associate a specific set of routers to clients within a user defined range. Having the policies and matching occur in kernel aids in fulfilling this requirement.
Rule Application
Performance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "flattened" to the constructs they impact. For example, a Network Rule is added as follows: o2ib priority 0. This rule gives priority for using o2ib network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every send to see if it we get any matches.
In Kernel Structures
| Code Block |
|---|
/* lnet structure will keep a list of UDSPs */
struct lnet {
...
list_head ln_udsp_list;
...
}
/* each NID range is defined as net_id and an ip range */
struct lnet_ud_nid_descr {
__u32 ud_net_id;
list_head ud_ip_range;
}
/*
* a UDSP rule can have up to three user defined NID descriptors
* - src: defines the local NID range for the rule
* - dst: defines the peer NID range for the rule
* - rte: defines the router NID range for the rule
*
* An action union defines the action to take when the rule
* is matched
*/
struct lnet_udsp {
list_head udsp_on_list;
lnet_ud_nid_descr *udsp_src;
lnet_ud_nid_descr *udsp_dst;
lnet_ud_nid_descr *udsp_rte;
union udsp_action {
__u32 udsp_priority;
};
}
/* The rules are flattened in the LNet structures as shown below */
struct lnet_net {
...
/* defines the relative priority of this net compared to others in the system */
__u32 priority;
...
}
struct lnet_ni {
...
/* defines the relative priority of this NI compared to other NIs in the net */
__u32 priority;
...
}
struct lnet_peer_ni {
...
/* defines the relative peer_ni priority compared to other peer_nis in the peer */
__u32 priority;
/* defines the list of local NID(s) (>=1) which should be used as the source */
union lpni_pref {
lnet_nid_t nid;
lnet_nid_t *nids;
}
/* defines the list of router NID(s) to be used when sending to this peer NI */
lnet_nid_t *rte_nids;
...
}
/* UDSPs will be passed to the kernel via IOCTL */
#define IOC_LIBCFS_ADD_UDSP _IOWR(IOC_LIBCFS_TYPE, 106, IOCTL_CONFIG_SIZE)
/* UDSP will be grabbed from the kernel via IOCTL
#define IOC_LIBCFS_GET_UDSP _IOWR(IOC_LIBCFS_TYPE, 106, IOCTL_CONFIG_SIZE) |
Kernel IOCTL Handling
| Code Block |
|---|
/* api-ni.c will be modified to handle adding a UDSP */
int
LNetCtl(unsigned int cmd, void *arg)
{
...
case IOC_LIBCFS_ADD_UDSP: {
struct lnet_ioctl_config_udsp *config_udsp = arg;
mutex_lock(&the_lnet.ln_api_mutex);
/*
* add and do initial flattening of the UDSP into
* internal structures
*/
rc = lnet_add_and_flatten_udsp(config_udsp);
mutex_unlock(&the_lnet.ln_api_mutex);
return rc;
}
case IOC_LIBCFS_GET_UDSP: {
struct lnet_ioctl_config_udsp *get_udsp = arg;
mutex_lock(&the_lnet.ln_api_mutex);
/*
* get the udsp at index provided. Return -ENOENT if
* no more UDSPs to get
*/
rc = lnet_add_udsp(get_udsp, get_udsp->idx);
mutex_unlock(&the_lnet.ln_api_mutex);
return rc
}
...
} |
Kernel Selection Algorithm Modifications
User space Design
In this approach all the policy parsing and storage happens in user space. The kernel API is kept very simple. It'll just provide a way to set actions on LNet constructs, IE networks, NIDs and routers. A user space utility, lnetctl, will parse policies and store them in a persistent file. Whenever lnetctl is used to add a net or a peer explicitly, then it will read the policy file and apply policy actions on constructs being created in the kernel.
However, since the kernel has no concept of policies, if peers are created internally, as they usually are, then these policies can not be applied. One method to get around this issue is to implement uevents. Whenever a peer is created a uevent is fired to user space. This uevent results in the policy file being read and if there is a match against one of the policies in the file, then the peer is updated appropriately. There is a period of time between the creation of the peer and the handling of the uevent in user space where the policy is not applied and therefore traffic patterns might not conform to user specified definition until the peer is updated.
It is possible to keep peer creation in incomplete state until the peer is updated from user space. However, this will create a dependency for the kernel in user space. Cases where user space processes crash or misbehave could impact the kernelPerformance needs to be taken into account with this feature. It is not feasible to traverse the policy lists on every send operation. This will add unnecessary overhead. When rules are applied they have to be "flattened" to the constructs they impact. For example, a Network Rule is added as follows: o2ib priority 0. This rule gives priority for using o2ib network for sending. A priority field in the network will be added. This will be set to 0 for the o2ib network. As we traverse the networks in the selection algorithm, which is part of the current code, the priority field will be compared. This is a more optimal approach than examining the policies on every send to see if it matches any networks.
Rule Structure
Selection policy rules are comprised of two parts:
...