Overview
LNet is a virtual networking layer which allows Lustre nodes to communicate with each other.
System Level Overview
System Diagram
- lnetctl: User space utility used to configure and query LNet kernel module
- DLC Library: User space library which communicates with LNet kernel module primarily via IOCTL
- LNet IOCTL: Module which handles the IOCTLs and calls appropriate callbacks in the LNet kernel module
- PTLRPC: Kernel module which implements an RPC protocol. It's the primary user of LNet.
- LNet: Kernel module which implements the Lustre Networking communication protocol
- o2iblnd: Verbs driver. It executes RDMA operations via verbs
- socklnd: TCP/IP driver. It sends/receives TCP messages
- gnilnd: Cray/HPE driver not maintained by us
Block Level Overview
LNet Block Level Diagram
The diagram above represents the different functional blocks in LNet. A quick overview will help in understanding the code
- When LNet starts up it reads various module parameters and configures itself based on these values.
- Further configuration can be added dynamically via
lnetctlutility. - The main APIs to request LNet to send messages are
LNetPut()andLNetGet().- When a message is sent a peer block is created to track messages to and from that peer.
- When a message is received a peer block is created to track messages to and from that peer.
- When sending messages LNet has to select the local and remote interfaces (IE the path the message will traverse to reach its destination). It does so through the selection algorithm.
- In that process it selects the local network interfaces and remote network interfaces for the destination peer.
- Each peer has its own set of credits used to rate limit messages to it. LNet checks and manages these credits before sending the message.
- When a message is sent a credit is consumed.
- When a message is received a credit is returned.
- If the destination peer is not on the same network as the node, then lookup a route to the final destination. If no route is present then the message can not be sent.
- If a node is acting as a router, then it can receive messages to which it is not the final destination. It then can forward these messages to the final destination.
- When a received message is to be forwarded then a router buffer is used to receive the message data. Router buffers have their own credits.
- A fault injection module can be activated for testing. That module will simulate message send/receive failures.