This word, or the terms "REQUIRED" or "MUST", mean that the definition is an absolute requirement of the specification.

This phrase, or the phrase "MUST NOT", mean that the definition is an absolute prohibition of the specification.

This phrase, or the phrase "NOT RECOMMENDED" mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Unique ID of the requirement; comprised of:

• Three letter acronym of the requirement category, as defined above
• A number which starts at 005 and incremented to allow for the addition of extra requirements between already existing requirements

Version number of the requirement

• Draft version will be in the format of 0.X
  
  • Where X >= 0
    • Ex: 0.1
• Accepted version will be in the format of Y.00
  • Where Y >= 0

Current status of the requirement. Will be one of the following:

• IN-PROGRESS: Still being developed and discussed
• ACCEPTED: Has been agreed on and signed off

A timeout that occurs due to a problem on the peer NI. A message has been posted to the network and completed successfully, but an expected acknowledgement is not received.

Not receiving an LNET ACK/REPLY is categorized as a remote timeout.

Some LND protocol messages expect an acknowledgement, not receiving those constitute a remote timeout.

A health range value shall be configured. The NI health is a value between 0 and 1000. Each NI health value is initialized to 1000, fully healthy, on startup and is decremented on failure to transmit on/to that NI.

The health range value is used to provide a range in which to select the NI. Any NI within the configured health value range will be considered healthy. Values greater than 1000 turns off selection based on health. A value of 0 selects only healthy interfaces.

The LND transmit timeouts shall be provided by the LNet layer.

• Currently the LND transmit timeout is an LND module parameter that defaults to 50 seconds. This is too long. For LNet Health we'll use a trickle down approach, where the ULP provides LNet with the timeout when sending a PUT or a GET, if not provided a configurable default is used. LNet then divides that timeout by the number of retries and provides that to the LND layer. If the message times out at the LND layer, then it is reported up via lnet_finalize() and a re-transmit is issued.
• Currently an o2iblnd connection is not destroyed until all work requests are completed, which is not time bound.
  
  • There is a ticket open to use the drain qp API available
  • This is a separate work item from this project.

The LND shall detect when a message times out before receiving an expected acknowledgment message:

• EX: an IBLND_MSG_PUT_ACK is not received for IBLND_MSG_PUT_REQ

This will be termed "remote timeout"

Note that for IMMEDIATE o2iblnd messages there is no expected ACK/NACK and therefore the transmit complete is the only indication that the message transmit was successful. This feature will not add any further hand shaking for this category of messages.

The health value is a positive number between 0 - 1000.

This range is chosen to allow enough granularity for decrementing and incrementing the health value.

LNet shall determine the NI to select based on the following ordered criteria:

1. NI health
2. NUMA closeness
3. NI available credits
4. Round Robin

When LNet fails to send on a local NI or to a remote NI, it shall place that NI on a recovery queue. The NI shall be pinged or used for a ping periodically to determine if it has recovered. 1000 minus current health value pings must pass sequentially in order for an interface to be considered fully healthy.

EX: if the NI's health value is 900, then 100 pings using that NI must be successful in order for the NI to be considered fully healthy. Each successful send will increment the NI's health value by 1.

On a local, remote and network timeout LNet shall reselect a pair of local and peer NI to resend the message.

• Another option  is to be more granular when selecting the interfaces depending on the timeout that occurred. If local timeout then only re-select a different local NI. If remote timeout then re-select a peer NI and for a network timeout then select a new pair of local and peer NIs.
• This has the disadvantage of a more complicated implementation.
• There isn't a clear advantage of making the selection more granular.
• Since the health value of the NI in question is decremented based on the error encountered, then the selection algorithm will favor the NI with poor health less.

For the routers LNet shall re-transmit a message over any of its local NIs.

Routers are a special case since non-MR peers expect the same source NID of the final destination, but doesn't care about the router NIDs. The router NIDs are not passed up to ptlrpc or other ULPs.

LNet shall not attempt to resend a message on the following failure types:

1. Shutdown in progress
2. Out of memory
3. Discovery errors out with one of the errors on this list.
4. An MD bind failure
   1. -EINVAL
   2. -HOSTUNREACH

5. Invalid information given

6. Internal failure

The assumption is that any resend will encounter the same failure again. Let the upper layers deal with the failure.

LNet shall re-transmit messages no more than the retry count specified by the user.

LNet shall stop re-transmitting when one of the following criterion is satisfied

1. Message is sent successfully
2. Retry count is reached
3. Transaction timeout expires.

LNet shall timeout a message and send a failure event to the ULP if a message is not re-transmitted successfully

LNet shall calculate the message timeout based on the ULP provided timeout if one is provided or the configured transaction timeout otherwise.

message timeout = transaction timeout / number of retries.

LNet shall allow the callers of LNetGet() or LNetPut() to specify a different transaction timeout other than the one configured in the system.

• EX: lnetctl ping can specify a shorter timeout than ptlrpc

LNet shall activate the transaction timeout only after a PUT which requires an ACK or a GET which requires a REPLY is successfully passed to the LND (IE lnd_send() returns successfully)

For PUT which requires no ACK no timeout will be activated.