Purpose

LNet is tracking the state of the interfaces it is configured to use by subscribing to kernel notification events. Upon receiving an event, LNet decides whether to mark the corresponding interface as available.

Interface being unavailable due to such event is designated as "fatal down" state and can be checked via "lnetctl net show -v 4".

The steps listed below are the recommended procedure to be used when this functionality needs to be verified. Some of these steps need to be executed manually or require access to the switch to which the network interfaces are connected.

Setup

Use two nodes: one where link/NI state is going to be manipulated (node A) and another for checking the ability to communicate (node B), configured such that they can connect.

Host machine with a dual-port network card rather than a VM is preferred.

LND-specific tests are labelled as such. Tests not labelled as LND-specific shold be repeated for every applicable LND.

"Connect/Disconnect interface" step in the procedure means both "ifup/ifdown" and physical cable pull/plug to be tried separately. These actions are not equivalent. 

If it is not possible to manipulate IB cable directly, "ibportstate" command can be used instead, for example: "ibportstate 2 1 -C mlx4_0 -P 1 enable". 


Test procedure

#DescriptionResultNotes

1

Initial state of Node A : interfaces (IA1, IA2) are up, IPs (IPA1, IPA2) are assigned, pingable from Node B. 

  1. Load LNet module
  2. Verify that "lnetctl net show" indicates that NIs (NIA1, NIA2) are "up"
  3. Disconnect IA1. 
  4. Use "lnetctl net show" on A to verify that NIA1 is "down". 
  5. "lnetctl ping" from B to NIA1 should fail
  6. "lnetctl ping" from A to B should succeed
  7. "lnetctl ping" from B to NIA2 should succeed
  8. Connect IA1
  9. Use "lnetctl net show" on A to verify that NIA1 is "up". 
  10. "lnetctl ping" from B to NIA1 should succeed
  11. Shutdown LNet


2.socklnd

Initial state of Node A : interfaces (IA1, IA2) are down, IPs (IPA1, IPA2) are assigned, but not pingable

  1. Load LNet module
  2. LNet should fail to load if interface it is (statically) assigned to use is "down"
  3. LNet should fail to add tcp interface dynamically if it is down.
  4. Shutdown LNet


2.o2iblnd

Initial state of Node A : interfaces (IA1, IA2) are down, IPs (IPA1, IPA2) are assigned, but not pingable

  1. Load LNet module
  2. LNet should be able to load if interface it is (statically) assigned to use is "down"
  3. LNet should be able to add ib interface dynamically if it is down
  4. Verify that "lnetctl net show" indicates that NIs (NIA1, NIA2) are "down"
  5. "lnetctl ping" from B to NIA1 and NIA2 should fail
  6. Connect IA1
  7. Use "lnetctl net show" on A to verify that NIA1 is "up". NIA2 should stay "down".
  8. "lnetctl ping" from B to NIA1 should succeed, but fail to NIA2
  9. Connect IA2
  10. Verify that "lnetctl net show" indicates that both NIs (NIA1, NIA2) are "up"
  11. "lnetctl ping" from B to NIA1 and NIA2 should succeed
  12. Shutdown LNet


3

Configure (LACP) bonding on Node A such that IA1, IA2 are slaves of IA0, with IP IPA0 assigned and pingable from Node B. Initially IA1 and IA2 are both up.

  1. Load LNet module
  2. Verify that "lnetctl net show" indicates that NIA0 is "up"
  3. Disconnect IA1
  4. Use "lnetctl net show" on A to verify that NIA0 is still "up".
  5. "lnetctl ping" from B to NIA0 should succeed
  6.  Disconnect IA2
  7. Verify that "lnetctl net show" indicates that NIA0 is "down"
  8. "lnetctl ping" from B to NIA0 should fail
  9. Connect IA1
  10. Verify that "lnetctl net show" indicates that NIA0 is "up"
  11. "lnetctl ping" from B to NIA0 should succeed
  12. Shutdown LNet


4

Initial state of Node A : interfaces (IA1, IA2) are up, IPs (IPA1, IPA2) are assigned, pingable from Node B. 

  1. Load LNet module
  2. Verify that "lnetctl net show" indicates that NIs (NIA1, NIA2) are "up"
  3. Delete IPA1 (e.g. using "ip addr del").
  4. Verify that "lnetctl net show" indicates that NIA1 is "down", NIA2 is still "up"
  5. "lnetctl ping" from B to NIA2 should succeed, but fail to NIA1
  6. Add IPA1 back
  7. Verify that "lnetctl net show" indicates that both NIA1 and NIA2 are "up"
  8. "lnetctl ping" from B to NIA1, NIA2 should succeed
  9. Shutdown LNet





  • No labels