Scope
Problem Statement
Multi-Rail (MR) and Health add the capability of using all the interfaces available. Instead of configuring each interface on a separate LNet, it is now possible to configure multiple interfaces on the same LNet. MR code can now iterate over all interfaces and select an interface to send a message based on one of the following criteria:
- Healthiest interface
- NUMA closeness
- Most available credits
- Round Robin
However, there is no way for the user to override this selection criteria. This becomes particularly useful when desiring to redirect traffic over specific paths depending on source and/or destination NIDs
Use Cases
There are two uses cases outlined on - LU-9121Getting issue details... STATUS
- The ability to select routers based on destination peer NIDs
- Assigning different interfaces to separate targets (MDT, OST) running on the same node
Other possible use cases
- Assigning priority to different networks. For example if you have an OPA and a IB network and you want to always use the IB network as long as it's alive.
- Preferring a specific path to avoid a bottleneck in the fabric
Work Overview
There are four primary tasks in this project
- Reuse and improve the existing policy infrastructure, currently used for creating runtime errors
- Currently there exists some infrastructure which was added to enable error injection policies to be added. This infrastructure can be enhanced to handle all policy based code. Both UDSP and Fault Injection features can use it.
- Implement the kernel side feature on top of this infrastructure, based on IOCTL commands
- Once we have the infrastructure, we can add the different UDSP policies
- Implement user space controls to drive the kernel side LNet feature
- Each UDSP policy will have a corresponding user space management interface.
- Write a Unit Test Plan
- This should be a live document modified as the design progresses.
- I'm pondering making it part of the High-Level Design. Basically each portion of the design will have a Test Plan section to be filled out
- Create a set of LUTF scripts to test this feature.
- It is critical to create the test scripts along side the development, because based on previous experience, the more test scripts are delayed the more likely they will not get done.
The High Level Design document will detail each of these tasks.
Requirements
These requirements are from the original MR Requirements document.
ID | Class | Version | Status | Description |
---|---|---|---|---|
cfg-095 | REQUIRED | 1.0 | ACCEPTED | DLC shall provide APIs to configure User Defined Selection Policy (UDSP) |
cfg-100 | REQUIRED | 1.0 | ACCEPTED | UDSP shall be comprised of a set of rules. |
cfg-105 | REQUIRED | 1.0 | ACCEPTED | Only one UDSP shall be added/removed/modified per configuration operation |
cfg-110 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define network priorities |
cfg-115 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define interface priorities |
cfg-120 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define one local NID to one remote NID mapping (1:1). |
cfg-125 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define mapping priority. |
cfg-130 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define many local NIDs to many remote NIDs mapping (N:N). |
cfg-135 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define many local NIDs to a one remote NID mapping (N:1). |
cfg-140 | REQUIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define one local NID to many remote NIDs mapping (1:N). |
cfg-145 | DESIRED | 1.0 | ACCEPTED | UDSP shall allow rules which define the number of messages that should be sent using one rule. This allows fine grained control over traffic distribution. |
cfg-150 | REQUIRED | 1.0 | ACCEPTED | UDSP rules shall provide the option to define relative rule priority |
cfg-155 | REQUIRED | 1.0 | ACCEPTED | If UDSP rule priority is not defined it defaults to highest priority |
cfg-160 | REQUIRED | 1.0 | ACCEPTED | lnetctl utility shall provide a command line front end interface to configure UDSP by calling the DLC APIs mentioned in the above requirements |
cfg-165 | REQUIRED | 1.0 | ACCEPTED | lnetctl utility shall accept and parse YAML configuration files specifying UDSP configuration |
cfg-200 | REQUIRED | 1.0 | UNDER REVIEW | UDSP shall allow rules which define many peer NIDs to many router NIDs mapping (N:N). |
cfg-205 | REQUIRED | 1.0 | UNDER REVIEW | UDSP shall allow rules which define mapping local networks to router NIDs (N:N). |
1 Comment
Amir Shehata
Currently UDSPs are used to enhance the selection algorithm to select between peer_nis of the same peer (as one of the peer). However, the selection algorithm can be very similar when selecting routers. Currently routers are simply round robined over (other criteria is used such as credits available, etc). However, this selection of routers can theoretically be merged with the peer_ni selection algorithm. This will allow policies to impact the selection of routes as well.
This comes in as a requirement from Aurelient Degremont at CEA.
We need to put some thought on the best way to define the policies to make them usable across multi-rail interfaces as well as routers.