...

• address is wrong
• Route can not be determined
• Connection can not be established
• Connection was rejected due to incompatible parameters

Desired Behavior

1. the remote interface health is updated
2. Failure statistics incremented
3. A resend is issued on a different remote interface if there is one available.
4. if no other remote interface is present then the send fails.

Implementation Specifics 

In all these cases a different peer_ni should be tried if one exists. lnet_select_pathway() already takes src_nid as a parameter. When resending due to one of these failures src_nid will be set to the src_nid in the message that is being resent.

...

Roughly, LNet would be analogous to the IP layer and ptlrpc is analogous to the TCP layer.

Reasons for timeout

The discussion here refers to the LND Transmit timeout.

Timeouts could occur due to several reasons:

1. The message is on the sender's queue and is not posted within the timeout
   1. This indicates that the local interface is too busy and is unable to process the messages on its queue.
2. The message is posted but the transmit is never completed
   1. An actual culprit can not be determined in this scenario. It could be a sender issue, a receiver issue or a network issue.
3. The message is posted, the transmit is completed, but the remote never acknowledges.
   1. In the IBLND, there are explicit acknowledgements in most cases when the message is received and forwarded to the LNet layer. Look below for more details.
   2. If an LND message is in waiting state and it didn't receive the expected response, then this indicates an issue at the remote's LND, either at the lower protocol, IB/TCP, or the notification at the LND layer is not being processed in a timely fashion.

Each of these scenarios can be handled differently

Desired Behavior

The desired behavior is listed for each of the above scenarios:

Scenario 1

1. Connection is closed
2. the local interface health is updated
3. Failure statistics incremented
4. A resend is issued on a different local interface if there is one available.
5. if no other local interface is present, or all are in failed mode, then the send fails.

Scenario 2

1. Connection is closed
2. the local and remote interface health is updated
3. Failure statistics incremented on both local and remote
4. A resend is issued on a different path all together if there is one available.
5. if no other path is present  then the send fails.

Scenario 3

1. Connection is closed
2. the remote interface health is updated
3. Failure statistics incremented
4. A resend is issued on a different remote interface if there is one available.
5. if no other remote interface is present then the send fails.

Note, that the behavior outlined is consistent with the explcit error cases identified in previous section. Only Scenario 2, diverges as a different path is selected all together, but still the same code structure is used.

Implementation Specifics

O2IBLND

Overview

There are two types of events to account for:

...