Original Pre-Health Requirements

Router Requirements

A router is a node which has the routing feature turned on using lnetctl set routing 1 or the equivalent modprobe configuration.

  1. Track the last time stamp any message was received on a local NI
  2. if the NI hasn't received any traffic for a period of router_ping_timeout + MAX(live_router_check_interval, dead_router_check_interval) then it's marked down
    1. This is done so that other nodes using the gateway can mark the route down, given that avoid_asym_router_failure is set to 1.
  3. Do not send messages to a peer NI which is marked down.
  4. Set the peer status to up when messages are received
  5. Query a peer NI, which is marked down, a minimum of once per second when there is traffic destined to it. If the query result determines that the peer NI is reachable, the peer NI state is set to UP. Messages can then be send to that peer NI.

Gateway Requirements

A gateway in this context is the peer NI created when adding a route on a node. For example: lnetctl route add --net tcp --gateway <gateway-NID>. Dealing with that peer-NI is somewhat of a special case.

  1. Mark the gateway peer NI as down when the LND fails to send a message
    1. Note although the LND notifications happen for all peer NIs it is only pertinent on routers or for gateways.
  2. Mark the gateway peer NI as up when we receive an unsolicited message or when we receive a REPLY for a PING sent from the router checker.
  3. Mark the route as down if one of the gateway's interfaces, identified by the gateway peer NI, are down, provided the avoid_asym_router_failure is set to 1.

Peer Requirements

  1. Do not check for peer aliveness when sending a message to a peer.
  2. Pick a route which has its gateway peer NI marked as up.

Implementation Details

The routing infrastructure currently performs the following functionality

  1. Keep track of the last time the peer was alive, lpni_last_alive
  2. Keep track the last time the peer was notified that its state has changed, lpni_timestamp
    1. The peer can change state under the following conditions:
      1. The LND notifies that the peer is down when it fails to send a message to the peer.
        1. As an example in o2iblnd:
          1. kiblnd_peer_connect_failed() and kiblnd_disconnect_conn() call kiblnd_peer_notify() which calls lnet_notify() to set the peer to dead if there was an error
      2. A message is received in lnet_parse()
        1. In this case the peer stat is set to alive only for gateway peer NIs
      3. When the router checker ping is responded to or it fails.
      4. If the router checker ping times out.
  3. This step only concerns routers: Only send the message if the peer is alive, determined as outlined above.
  4. On the router if the NI hasn't received any traffic for a period of router_ping_timeout + MAX(live_router_check_interval, dead_router_check_interval) then it's marked down.
    1. This is done in order for the peers using the router to mark the peer down when the avoid_asym_router_failure is set to 1, which it is by default.

LNet Multi-Rail

Multi-Rail introduced the concept of a peer and a peer NI. A peer can have multiple peer NIs. This changes the semantics of route configuration. Currently a route can be configured as:

lnetctl route add --net <remote net> --gateway <gateway-peer-NID>

The gateway-peer-NID refers to a specific interface on the router. However with MR enabled on the router, multiple interfaces can be configured on the same network. Therefore, the configuration semantics should be as follows:

lnetctl route add --net <remote net> --gateway <gateway-primary-NID>

When a route is entered the primary NID specified in the gateway parameter should be immediately discovered. The discovery process will determine all the interfaces available on the router. There could be multiple interfaces on the same network.

A route should only be marked down if all the interfaces on the primary NID's network are down.

Nodes on different networks will use different primary NIDs to refer to the same router. IE a primary NID is only a representation of the router on the peer with the route configured.

LNet Resiliency

The LNet Health/Resiliency feature has added the following features:

  1. Track a local NI or a peer NI health.
  2. Re-send messages
  3. Recover local NIs or peer NIs by pinging them every second.

The original route code which implements the requirements outlined above are no longer inline with the new mechanisms implemented. There needs to be an effort taken to bring the router code more inline with the new features implemented.

Some details were documented here: Routing and MR integration