| Test # | Tag | Procedure | Script | Result |
|---|
| 1 | Discovery triggered on route add | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
|
| pass |
| 2 | Discovery triggered on interval | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
|
| pass |
| 3 | Router tcp1 down due to no traffic | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
- Since there is no traffic on tcp1 RouterA tcp1 should be down
- verify via: lnetctl net show -v
|
| pass |
| 4 | Router tcp1 comes up when peerB is brought up | - Bring up Router A with two interfaces
- Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs from peer A→ Router A
- Keep the two nodes up for 4 minutes
- Every router_interval_timeout a discovery should occur from peerA→ RouterA
- Since there is no traffic on tcp1 RouterA tcp1 should be down
- verify via: lnetctl net show -v
- Bring up Peer B and add network on tcp1
- Add router to tcp on peer B
- Observe that a discovery occurs from peerB → RouterA
- Observe that a RouterA tcp1 is now up
|
| pass |
| 5 | Add route without router there | - Bring up Peer A and add network on tcp0
- Add route to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the peer is down
- Bring up Router A with two interfaces: tcp0, tcp1
- After router_interval_timeout a discovery should verify that router A is up
- lnetctl route show -v # shows that router is down because no routerA tcp1 network should be down
- lnetctl peer show -v # shows the peer is up
- Bring up PeerB and add network on tcp1
- lnetctl route show -v # shows that router is up
|
| pass |
| 6 | traffic should trigger an attempt at router discovery | - Bring up Peer A and add network on tcp0
- Add route to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the router is down
- Bring up Router A with two interfaces: tcp0, tcp1
- Bring up PeerB and add network on tcp1
- Before the router_interval_timeout expires do a:
- lnetctl discover Router@tcp
- This should trigger a discovery of router A
- lnetctl peer show -v # shows the peer is up and multi-rail
- lnetctl route show -v # shows the route up
|
| pass |
| 7 | Ping should not trigger discovery of router | - Bring up Peer A and add network on tcp0
- Add router to tcp1 on peerA
- Observe that a discovery occurs but no response since router is not up
- lnetctl route show -v # shows that router is down
- lnetctl peer show -v # shows the router is down
- Bring up Router A with two interfaces: tcp0, tcp1
- Bring up PeerB and add network on tcp1
- Before the router_interval_timeout expires do a:
- lnetctl ping PeerB@tcp1
- This should NOT trigger a discovery of router A
- ping should fail
- lnetctl peer show -v # shows the peer is down
- lnetctl route show -v # shows the route down
|
| pass |
| 8 | Multi-interface router even traffic distribution | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
|
| pass |
| 9 | Multi-interface router with one bad interface | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
- Enable health (sensitivity, retries)
- Add a PUT drop rule on the router to drop traffic on one of the interfaces in tcp0
- Observe that traffic goes to the other interfaces. There shouldn't be any drop in traffic.
- As long as the interface has less than optimal health, it should never be used for routing.
|
| pass |
| 10 | Multi-interface router with a bad interface that recovers | - Bring up Router A with 4 interfaces. 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic using selftest
- Observe that traffic is distributed on all router interfaces evenly
- Enable health (sensitivity, retries)
- Add a PUT drop rule on the router to drop traffic on one of the interfaces in tcp0
- Observe that traffic goes to the other interfaces. There shouldn't be any drop in traffic.
- As long as the interface has less than optimal health, it should never be used for routing.
- Remove the PUT drop rule from the router
- Eventually that interface should be healthy again
- Traffic should resume using that interface
|
| pass In an idle system the bad peer interface will be pinged once every second causing its sequence number to go up. So when it comes back online it will not be used until the sequence numbers equalize. This will be the case if the system is busy, but the issue will be reversed. |
| 11 | Multi-Router/Multi-interface setup | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
|
| pass |
| 12 | Multi-Router/Multi-interface setup with failed gateway | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
- Shutdown router A
- Observe that traffic is diverted to Router B with no drop in traffic.
|
| pass |
| 13 | Multi-Router/Multi-interface setup with router recovery | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up router B with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- Run traffic
- Observe that traffic is distributed evenly on the interfaces of router A and B
- Add a drop rule on Router A that impacts all of its interfacesShutdown router A
- Observe that traffic is diverted to Router B with no failure.
- Remove the rule from Startup Router A
- Observe that traffic starts going through Router A again. There should be no drop in traffic
|
| Problem found. Possibly with discovery. 1. bring up two routers with 4 interfaces 2 on each network 2. bring down one of the routers 3. bring it up again but with only 2 of its interfaces on 1 network 4. Client goes berserk, keeps trying to discover it. toggles between state: 0x139 and 39 |
| 14 | router sensitivity < 100 | - Bring up router A with 4 interfaces 2 on tcp0 and 2 on tcp1
- Bring up Peer A with interface on tcp0
- Bring up Peer B with interface on tcp1
- modify the router_sensitivity to 50%
- Add a drop rule on router A
- Observe that traffic doesn't completely stop to Router A until its health goes to 50% of the optimal value.
|
|
|
| 15 | Extra Health Testing | - Run through the health test cases above while there exists a multi-rail router.
|
|
|
