| Test # | Tag | Procedure | Script | Result |
|---|
| 1 | Discovery triggered on route add | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
|
| pass |
| 2 | Discovery triggered on interval | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
|
| pass |
| 3 | Router tcp1 down due to no traffic | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
- Since there is no traffic on tcp1 RouterA tcp1 should be down
- verify via: lnetctl net show -v
|
| pass |
| 4 | Router tcp1 comes up when peerB is brought up | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
- Since there is no traffic on tcp1 RouterA tcp1 should be down
- verify via: lnetctl net show -v
- Bring up Peer B and add network on tcp1
- Add router to tcp on peer B
- Observe that a discovery occurs from peerB → RouterA
- Observe that a RouterA tcp1 is now up
|
| pass |
| 5 | Add route without router there | - Bring up Peer A and add network on tcp0
- Add route to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the peer is down
- Bring up Router A with two interfaces: tcp0, tcp1
- After router_interval_timeout a discovery should verify that router A is up
- lnetctl route show -v # shows that router is down because no routerA tcp1 network should be down
- lnetctl peer show -v # shows the peer is up
- Bring up PeerB and add network on tcp1
- lnetctl route show -v # shows that router is up
|
| pass |
| 6 | traffic should trigger an attempt at router discovery | - Bring up Peer A and add network on tcp0
- Add route to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the router is down
- Bring up Router A with two interfaces: tcp0, tcp1
- Bring up PeerB and add network on tcp1
- Before the router_interval_timeout expires do a:
- lnetctl discover Router@tcp
- This should trigger a discovery of router A
- lnetctl peer show -v # shows the peer is up and multi-rail
- lnetctl route show -v # shows the route up
|
| pass |
| 7 | Ping should not trigger discovery of router | - Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the router is down
- Bring up Router A with two interfaces: tcp0, tcp1
- Bring up PeerB and add network on tcp1
- Before the router_interval_timeout expires do a:
- lnetctl ping PeerB@tcp1
- This should NOT trigger a discovery of router A
- ping should fail
- lnetctl peer show -v # shows the peer is down
- lnetctl route show -v # shows the route down
|
| pass |
| 8 | Multi-interface router even traffic distribution | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
|
| pass |
| 9 | Multi-interface router with one bad interface | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
- Enable health (sensitivity, retries)
- Add a PUT drop rule on the router to drop traffic on one of the interfaces in tcp0
- Observe that traffic goes to the other interfaces. There shouldn't be any drop in traffic.
- As long as the interface has less than optimal health, it should never be used for routing.
|
| pass |
| 10 | Multi-interface router with a bad interface that recovers | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
- Enable health (sensitivity, retries)
- Add a PUT drop rule on the router to drop traffic on one of the interfaces in tcp0
- Observe that traffic goes to the other interfaces. There shouldn't be any drop in traffic.
- As long as the interface has less than optimal health, it should never be used for routing.
- Remove the PUT drop rule from the router
- Eventually that interface should be healthy again
- Traffic should resume using that interface
|
|
|
| 11 | Multi-Router/Multi-interface setup | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
|
|
|
| 12 | Multi-Router/Multi-interface setup with failed gateway | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
- Add a drop rule on Router A that impacts all of its interfaces
- Observe that traffic is diverted to Router B with no drop in traffic.
|
|
|
| 13 | Multi-Router/Multi-interface setup with router recovery | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
- Add a drop rule on Router A that impacts all of its interfaces
- Observe that traffic is diverted to Router B with no failure.
- Remove the rule from Router A
- Observe that traffic starts going through Router A again. There should be no drop in traffic
|
|
|
| 14 | router sensitivity < 100 | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- modify the router_sensitivity to 50%
- Add a drop rule on router A
- Observe that traffic doesn't completely stop to Router A until its health goes to 50% of the optimal value.
|
|
|
| 15 | Extra Health Testing | - Run through the health test cases above while there exists a multi-rail router.
|
|
|
