Active Sending

A node which actively sends a message can be a client or a server. The below flow diagram describes the message processing

Textual Description

• Lustre requests a message send to a destination NID
• Node looks at the net of the NID and discovers that it is a remote net
• Node looks up the remote net and finds all the list of routes to that remote net
  • If remote net is not found then destination NID is not reachable
  • If there are multiple routes then one is selected based on priority, number of hops or in round robin
• Node starts the message sending process
  • Node looks at the number of credits available for the gateway peer
    • If there a enough credits available (determined by peer_credits) then the message is sent to the LND for processing
    • If there aren't enough credits available, then the message is queued internally until another active send message is finalized and a credit is returned. Messages are then popped out in FIFO order
  • The LND processing is  specific to the LND type. For the purpose of this write up we'll talk about o2iblnd
    • o2iblnd will deal with each of a PUT/GET/ACK/REPLY in slightly different way. For example an LNet PUT can be broken down into PUT_REQ/PUT_ACK|NACK/PUT_DONE. For more details you can take a look at: O2IBLND Detailed Discussion#LNDTXTimeout
    • o2iblnd has its own set of connection credits. If the credits on the connection are depleted then the message is queued until there are credits available, then the message is posted
    • o2iblnd will then place the transmit in the active transmit state and wait for the transmit to complete as reported by the OFED layer.
      • Each message has a deadline, based on the lnet_transaction_timeout  or the lnd timeout  configuration parameters. If the deadline is reached while the message is queued or the in the transmit state, then the transmit expires and the message is finalized.
      • Assuming everything works then the message is sent to the router

Router Message Processing

• The router receives the message and looks at the destination NID. If the NID is not its own, then it knows it needs to route that.
  • The router pulls a buffer depending on the size of the message. If there are no available buffers then the message is queued until a buffer becomes available.
  • Once a buffer becomes available then the message is received into that buffer
  • Message is then forwarded to the destination NID.
    • The send processing described above is followed.