LNet calls ksock_startup() on every lnet_ni created either dynamically when called via lnetctl or via module parameters on initial startup.
A ksock_net block is created and assigned to the lnet_ni.ni_data field. On all APIs call from LNet to the socklnd this field is used to pull up the ksock_net.
Of particular interest is the ksock_net.ksock_interface. This is an array of LNET_INTERFACES_NUM length. This is so because of the legacy tcp bonding feature. There could be multiple interfaces assigned to one ksock_net. However, since Multi-Rail feature manages the multiple network interfaces per network, there is no need to continue supporting tcp bonding.
Once a ksock_net block is created it's added on the global network list, ksocknal_data.ksnd_nets.
This list is traversed when adding a new network. If the interface being added is already being used by one of the configured networks, then we do not need to create a set of scheduler threads. However, if it's a new interface then we'll increase the number of scheduler threads, as long as we stay below the maximum number of configured scheduler interfaces. This is so we can aid in processing the transmits and receives on the new interface.
Scheduler threads are created per CPT and they are intended to serve transmit and receive operations.
When LNet requests a message to be sent by calling ksocknal_send() ksocklnd will create a peer if one doesn't exist. ksocklnd identifies a peer via its source and destination NIDs. This lends itself to how Multi-Rail works. At the LNet level then, messages which are going over the local Net to the same peer, can traverse multiple peer_nis at the ksocklnd level.
When a transmit is initially launched then a peer is created if none exist. The following steps take place:
ksock_peer_ni block is created and initializeksock_route block is created and initialized with the IP address and portNotes on routes and their use:
These restrictions defined by the code means there could exist only one route between one specific local interface and one peer.
With legacy TCP bonding there could exist multiple routes between each interface stored in the ksock_net.ksock_interface array and the peer_ni.
Therefore the use of ksock_route is purely to serve the legacy TCP bonding implementation, which has been superseded by the LNet Multi-Rail feature.
Once the routes are created TCP sockets must be created with the remote peer. This is referred to in the code as "connecting routes". The process is triggered by ksocknal_launch_all_connections_locked().
The route is placed on the ksocknal_data.ksnd_connd_routes queue. One of the connd threads then picks that up and starts the actual connection procedure by calling ksocknal_connect().
The number of sockets to create are configurable via the typed_conns module parameter. If This is set to 1 then three sockets will be created:
A connection is created per socket. This connection will be added to the ksock_peer_ni list of connections.
A hello message is sent by the active side of the connection. This hello message contains the list of IP addresses stored in the ksnd_data.ksock_interfaces to which we don't have routes yet. When the passive side receives the hello message it sends its own hello as a response. The active side will receive that hello which contains the list of the remote's peer IP addresses. It will then create additional routes to these interfaces, which we would create on demand when sending messages are sent.
Connection creation management is unduly complex due to TCP bonding. In fact the purpose of the hello message appears to be primarily for passing around the IP addresses of the peer.
Since TCP bonding is now deprecated, this code can be removed, simplifying the over all design of the socklnd code.