Page History

• Configure 3 NIs with equadistant NUMA distance
• Send three or more messages
• Dump statistics on each NI to verify that each NI was used to send messages

• Configure 3 NIs closer to different NUMA nodes
• dump the NI statistics
• Verify that each NI has the correct device CPT

• Configure 3 NIs with different NUMA distances
• Send messages
• Confirm through statistics that messages are being sent over the nearest NI (NUMA wise)

• Configure 2 NIs with different NUMA distances
• Send messages
• Confirm through statistics that messages are being sent over the nearest NI (NUMA wise)
• add another NI which is close NUMA wise than the current nearest
• confirm through statistics that messages are not being sent over the newly added NI

• Configure 3 NIs, one EDR, one FDR and one QDR
• set the NUMA range to a large value so all NIs are considered through RR
• start traffic
• monitor statistics on each NI.
• Confirm that EDR is preferred until it becomes saturated, then FDR is selected then QDR

• Configure 3 NIs
• set the NUMA range to a large value so all NIs are considered through RR
• start traffic
• monitor statistics on each NIs to confirm all are being used.
• Remove one of the NIs
• Confirm that that NI is no longer used for new messages
• Confirm that the other 2 NIs are being used.
• No messages should be dropped.

• Configure 3 NIs
• Configure a peer with 3 NIDs
• Send messages to the peer
• Confirm through statistics that peer NIDs are being used based on their available credits.

• Configure 3 NIs which are not equadistant all on the same network
• configure a peer with 3 NIDs all on the same network
• start traffic
• Confirm closest NUMA NI is being used
• Confirm peer NIDs are being used
• set NUMA range to a large value
• Confirm all NIs are being used
• Confirm no change in traffic pattern to the peers

• Configure NIs A, B and C
• Configure the peer with the same NIDs
• Send 1 message which requires a response from NI A
• Confirm that responses are being sent to the same NI

• Configure NIs A, B and C
• Configure the peer with the same NIDs
• Send 1 message which requires a response from NI A
• bring down NI A
• confirm that response is sent to one of the other configured NIDs

• Configure an MR system
• Start traffic
• monitor traffic is being sent to all configured peers
• bring down one of the peer NIDs
• monitory traffic is no longer sent to that peer NID
• no messages should be dropped

• Configure an MR system
• Start traffic
• monitor traffic is being sent to all configured peers
• bring down one of the peer NIDs
• monitory traffic is no longer sent to that peer NID
• bring up the peer NID again
• monitor traffic is being sent to it again

When a peer NID is removed then added or if a new peer NID is added, then it's peer_ni->lpni_seq number will start off at 0.

In the case of credits being == then the newly added peer will always be picked until it's lpni_seq number catches up with the other peer NIs sequence numbers. See code below.

		} else if (lpni->lpni_txcredits == best_lpni_credits) {
			/*
			 * The best peer found so far and the current peer
			 * have the same number of available credits let's
			 * make sure to select between them using Round
			 * Robin
			 */
			if (best_lpni) {
				if (best_lpni->lpni_seq <= lpni->lpni_seq)
					continue;
			}

This behavior will manifest itself in low bandwidth environment. In high bandwidth environment it is likely that the credits in the selection algorithm will be different and the peer_NI will be picked according to credits.

Another scenario to consider is when the lpni_seq number wraps. In low bandwidth environment this could cause the peer NI which wrapped to be picked until it catches up with the other peer NIs sequence numbers.

This in itself might not be significant enough, but does raise the question of the benefit of having a seq number to start with. Does it give much of a functional advantage, or having the credits criteria enough.

The same issue is present with local NI sequence numbers.

• Configure an MR system
• Start traffic
• monitor traffic is being sent to all configured peers
• bring down one of the peer NIDs
• monitory traffic is no longer sent to that peer NID
• bring down all peer NIDs
• message should fail.

• Configure an MR system
• Start traffic
• monitor traffic is being sent to all configured peers over all NIs
• bring down the local NIs one by one
• note traffic is migrated to the NIs still up, until no NIs are left then messages are dropped

• Configure an MR system
• Start traffic
• monitor traffic is being sent to all configured peers over all NIs
• bring down the local NIs one by one
• note traffic is migrated to the NIs still up, until no NIs are left then messages are dropped
• bring up the NIs again and confirm that NIs are being reused.

• Configure two networks tcp and o2ib
• Configure nodes to have multiple interfaces on each of the networks
• start traffic over the o2ib network
• o2ib should be used
• bring down the o2ib network
• traffic should migrage to the tcp network.
• no traffic should be dropped.

• Configure an MR system
• bring down an NI
• confirm that the show info shows the NI as down

• TODO: how do we test device failure?

• Configure an MR system
• Configure peers via DLC
• Run traffic
• Delete one of the peer_nis we're sending to via DLC
• Traffic going over that peer_ni should continue but no more traffic should use that NI

• Configure an MR system
• Configure peers via DLC
• Run traffic
• Delete one of the peer_nis we're sending to via DLC
• Bring that peer_ni back
• Note traffic stops and starts on that peer with no traffic loss
• Repeat the deletion and reconfiguration of the peer_ni

• Configure an MR system
• Configure peers via DLC
• Run traffic
• Delete the entire peer
• The peer should be recreated on the next message, but it won't be MR capable.

• dump per NI statistics
  • transmitted
  • received
  • dropped
  • timeouts
  • state

• configure multiple NIs
• run traffic
• dump stats on all NIs

• configure multiple NIs
• run traffic
• dump stats on all NIs
• Filter on specific NID

• dump LNet level statistics

• configure multiple peers
• start traffic
• dump per peer statistics

• configure multiple NIs
• toggle their state from ACTIVE to DOWN
• confirm that state change is being printed to console.

• configure an MR system
• start traffic
• bring down an NI
• confirm that messages indicating that another NI/peer is being used is printed.

• Configure an MR system
• run traffic
• stop traffic
• dump NI statistics
• dump peer statistics
• dump LNet level statistics
• zero out stats
• dump all statistics above to confirm they've been zeroed out.