Page History

...

Network Interface Management

...

Scheduler threads are created per CPT and they are intended to serve transmit and receive operations.

Peer and Connection Management

Peer and Route Management

When LNet requests a message to be sent by calling ksocknal_send() ksocklnd  will create a peer if one doesn't exist. ksocklnd identifies a peer via its source and destination NIDs. This lends itself to how Multi-Rail works. At the LNet level then, messages which are going over the local Net to the same peer, can traverse multiple peer_nis  at the ksocklnd  level.

When a transmit is initially launched then a peer is created if none exist. The following steps take place:

• A ksock_peer_ni block is created and initialize
• A ksock_route block is created and initialized with the IP address and port
  • The IP address of the peer is derived from the NID of the peer as provided by LNet.
• The route is associated with the peer

Notes on routes and their use:

• One route block per peer IP address can be created.
• The IP address used to create the route is derived from the peer's NID as provided by LNet.
• Since the IP address is used to create the route and there could only be one route to a specific IP address and the IP address is derived from the peer.
• The peer is uniquely identified by the local NI (bound to only one interface) and destination IP address defined the peer's NID.

These restrictions defined by the code means there could exist only one route between one specific local interface and one peer.

With legacy TCP bonding there could exist multiple routes between each interface stored in the ksock_net.ksock_interface array and the peer_ni.

Therefore the use of ksock_route is purely to serve the legacy TCP bonding implementation, which has been superseded by the LNet Multi-Rail feature.

Connection Management

Once the routes are created TCP sockets must be created with the remote peer. This is referred to in the code as "connecting routes". The process is triggered by ksocknal_launch_all_connections_locked().

The route is placed on the ksocknal_data.ksnd_connd_routes queue. One of the connd threads then picks that up and starts the actual connection procedure by calling ksocknal_connect().

The number of sockets to create are configurable via the typed_conns module parameter. If This is set to 1 then three sockets will be created:

• Control socket
• Bulk in socket
• Bulk out socket

A connection is created per socket. This connection will be added to the ksock_peer_ni list of connections.

A hello message is sent by the active side of the connection. This hello message contains the list of IP addresses stored in the ksnd_data.ksock_interfaces to which we don't have routes yet. When the passive side receives the hello message it sends its own hello as a response. The active side will receive that hello which contains the list of the remote's peer IP addresses. It will then create additional routes to these interfaces, which we would create on demand when sending messages are sent.

Connection creation management is unduly complex due to TCP bonding. In fact the purpose of the hello message appears to be primarily for passing around the IP addresses of the peer.

Since TCP bonding is now deprecated, this code can be removed, simplifying the over all design of the socklnd code.

Sending Messages