...
As shown in the above diagram whenever a tx is queued to be sent or is posted but haven't received confirmation yet, the tx_deadline is still active. The scheduler thread checks the active connections for any transmits which has passed their deadline, and then it closes those connections and notifies LNet via lnet_notify().
The tx timeout can occur also when sending ACKs and NACKs which are invisible to the LNet layeris cancelled when in the call kiblnd_tx_done(). This function checks 3 flags: tx_sending, tx_waiting and tx_queued. If all of them are 0 then the tx is closed as completed. The key flag to note is tx_waiting. That flag indicates that the tx is waiting for a reply. It is set to 1 in kiblnd_send, when sending the PUT_REQ or GET_REQ. It is also set when sending the PUT_ACK. All of these messages expect a reply back. When the expected reply is received then tx_waiting is set to 0 and kiblnd_tx_done() is called, which eventually cancels the tx_timeout, by basically removing the tx from the queues being checked for the timeout.
The notification in the LNet layer that the connection has been closed can be used by MR to attempt to resend the message on a different peer_ni.
...
| Gliffy Diagram | ||||
|---|---|---|---|---|
|
After the completion of an o2iblnd tx ib_post_send(), a completion event is added to the completion queue. This triggers kiblnd_complete to be called. If this is an IBLND_WID_TX then kiblnd_tx_complete() is called, which calls kiblnd_tx_done() if the tx is not sending, waiting or queued. In this case the tx_timeout is closed.
In summary, the tx_timeout serves to ensure that messages which do not require an explicit response from the peer are completed on the tx event added by M|OFED to the completion queue. And it also serves to ensure that any messages which require an explicit reply to be completed receive that reply within the tx_timoutTBD: when the passive (peer 2 in the above diagarm) send the GET_DONE, it goes through the same path of calling kiblnd_queue_tx() which ends up setting the tx_timeout on the DONE message. But I don't see how we avoid the timeout on this DONE message, since it also has a deadline. Same with the PUT scenario.
TX Timeouts in the presence of LNet Routers
...