Selection Algorithm Scenarios

Test #	Tag	Procedure	Result
1	SRC_SPEC_LOCAL_MR_DST	MR Node MR Peer Send a ping REPLY for PING should always come on the same interface that PING was sent on. Check the TRACE in the logs to verify Repeat the test. A different local NI should be used for each new PING.	pass
2	SRC_SPEC_LOCAL_MR_DST	MR Node MR Peer Initiate discovery Node → PING → Peer Node ← PUSH ← Peer Node should respond with an ACK to the same interface as the one it received the PUSH on Check the TRACE in the logs to verify Repeat the test Peer's local_ni when sending the PUSH should be different.	pass
3	SRC_SPEC_ROUTER_MR_DST	MR Node NMR Router MR Peer Send a ping REPLY for PING should always come on the same interface that PING was sent on. Check the TRACE in the logs to verify Router should be used Repeat the test. A different local NI should be used for each new PING.	pass
4	SRC_SPEC_ROUTER_MR_DST	MR Node NMR Router MR Peer Initiate discovery Node → PING → Peer Node ← PUSH ← Peer Node should respond with an ACK to the same interface as the one it received the PUSH on Check the TRACE in the logs to verify Router should be used Repeat the test. Peer's local_ni when sending the PUSH should be different.	pass
5	SRC_SPEC_ROUTER_MR_DST	MR Node MR Router MR Peer Send a ping REPLY for PING should always come on the same interface that PING was sent on. Check the TRACE in the logs to verify Repeat sending Router interfaces should be used in round robin, while the peer destination should remain constant. Repeat the test. A different local NI should be used for each new PING.	pass
6	SRC_SPEC_ROUTER_MR_DST	MR Node MR Router MR Peer Initiate discovery Node → PING → Peer Node ← PUSH ← Peer Node should respond with an ACK to the same interface as the one it received the PUSH on Check the TRACE in the logs to verify Router interfaces should be used in round robin, while the peer destination should remain constant. Repeat the test. Peer's local_ni when sending the PUSH should be different.	pass
7	SRC_SPEC_LOCAL_NMR_DST	Same as 1 and 2 Except that repeating the test will not result in a different local_ni being used.	pass
8	SRC_SPEC_ROUTER_NMR_DST	Same as 3 - 6 Except that repeating the test will not result in different local NIs being used.	pass
9	SRC_ANY_LOCAL_MR_DST	MR Node MR Peer Send multiple PINGs PING REPLYs should come on the same interface Every PING will select a new local/remote NIs	pass
10	SRC_ANY_ROUTER_MR_DST	MR Node NMR Router MR Peer Send Multiple PINGs Node will cycle over local_NIs Node will use the same destination NID as final destination Node will use the NMR Router	pass
11	SRC_ANY_ROUTER_MR_DST	MR Node MR Router MR Peer Send Multiple PINGs Node will cycle over local_NIs Node will use the same destination NID as final destination Node will use the different interfaces of the MR Router MR Router will cycle over the interfaces of the Final destination.	pass
12	SRC_ANY_LOCAL_NMR_DST	MR Node NMR Peer Send multiple PINGs Node will use same source/dst NID for all PINGs	pass
13	SRC_ANY_ROUTER_NMR_DST	MR Node NMR Router NMR Peer Send multiple PINGs Node will use the same source/dst NIDs for all PINGs Node will use the router interface	pass
14	SRC_ANY_ROUTER_NMR_DST	MR Node MR Router NMR Peer Send multiple PINGs Node will use the same source/dst NIDs for all PINGs Node will cycle through the Router's interfaces	pass

Error Scenarios

Synchronous Errors

Test #

Tag

Procedure

Script

Result

1

Immediate Failure

Send a PING
simulate an immediate LND failure (EX: NOMEM)
Message should not be resent

lnetctl discover <nid>

lctl net_drop_add with "-e local_error"

lnetctl discover <nid>

pass

Asynchronous Errors

Test #

Tag

Procedure

Script

Result

1

LNET_MSG_STATUS_LOCAL_INTERRUPT

LNET_MSG_STATUS_LOCAL_DROPPED

LNET_MSG_STATUS_LOCAL_ABORTED

LNET_MSG_STATUS_LOCAL_NO_ROUTE

LNET_MSG_STATUS_LOCAL_TIMEOUT

MR Node with Multiple interfaces
Send a PING
Simulate an <error>
PING msg should be queued on resend queue
PING msg will be resent on a different interface
Failed interfaces' health value will be decremented
Failed interface will be placed on the recovery queue

2Sensitivity == 0

Same setup as 1
NI is not placed on the recovery queue

3Sensitivity > 0

Same setup as 1
NI is placed on the recovery queue
Monitor network activity as NI is pinged until health is back to maximum

4

Sensitivity > 0

Buggy interface

Examples:

lctl net_drop_add -s 10.9.10.3@tcp -d 10.9.10.4@tcp -m GET -i 20 -e local_dropped

Key messages in debug log:

(lib-msg.c:762:lnet_health_check()) 10.9.10.3@tcp->10.9.10.4@tcp:GET:LOCAL_DROPPED - queuing for resend

(lib-msg.c:508:lnet_handle_local_failure()) ni 10.9.10.3@tcp added to recovery queue. Health = 950

(lib-move.c:2928:lnet_recover_local_nis()) attempting to recover local ni: 10.9.10.3@tcp

pass

2

Sensitivity == 0

Same setup as 1
NI is not placed on the recovery queue

pass

3

Sensitivity > 0

Same setup as 1
NI is placed on the recovery queue
Monitor network activity as NI is pinged

ever 1

until health is back to maximum

pass

4

Sensitivity > 0

Buggy interface

Same setup as 1
NI is placed on recovery queue
NI is pinged ever 1 second
Simulate ping failure ever other ping
NI's health should be decremented on failure
NI should remain on the recovery queue

5

Retry count == 0

Same setup as 1
Message will not be retried and the message will be finalized

immediately6Retry count > 0

immediately

pass

6

Retry count > 0

Same setup as 1
Message will be transmitted for a maximum of retry count or until the message expires

Key messages in debug log:

(lib-move.c:1715:lnet_handle_send()) TRACE: 10.9.10.3@tcp(10.9.10.3@tcp:<?>) -> 10.9.10.4@tcp(10.9.10.4@tcp:10.9.10.4@tcp) : GET try# 0

(lib-move.c:1715:lnet_handle_send()) TRACE: 10.9.10.3@tcp(10.9.10.3@tcp:<?>) -> 10.9.10.4@tcp(10.9.10.4@tcp:10.9.10.4@tcp) : GET try# 1

pass

Same setup as 1

Message will be transmitted for a maximum of retry count or until the message expires


7	REPLY timeout	Same setup as 1 Except Use LNet selftest Simulate a local timeout Re-transmit No REPLY received Message is finalized and TIMEOUT event is propagated.	pass
8	ACK timeout	Same setup as 7 except simulate ACK timeout	pass
9	LNET_MSG_STATUS_LOCAL_ERROR	Same setup as 1 Message is finalized immediately (not resent) Local NI is placed on the recovery queue Same procedure to recover the local NI	pass
10	LNET_MSG_STATUS_REMOTE_DROPPED	Same setup as 1 Message is queued for resend depending on retry_count peer_ni is placed on the recovery queue (not if sensitivity == 0) peer_ni is pinged every 1

second

second

pass

11

LNET_MSG_STATUS_REMOTE_ERROR

LNET_MSG_STATUS_REMOTE_TIMEOUT

LNET_MSG_STATUS_NETWORK_TIMEOUT

Same setup as 1
Message is not resent
peer_ni recovery happens as outlined in previous cases

pass

Random Failures

Test #	Tag	Procedure	Script	Result
1	self test	MR Node NMR Peer Self-test Randomize local NI failure Randomize Remote NI failure	ip link set eth1 down	pass
2	self test	MR Node MR Peer Self-test Randomize local NI failure Randomize Remote NI failure		pass
3	self test	MR Node MR Router NMR Peer Self-test Randomize local NI failure Randomize Remote NI failure
4	self test	MR Node MR Router MR Peer Self-test Randomize local NI failure Randomize Remote NI failure
5	self test	MR Node NMR Router NMR Peer Self-test Randomize local NI failure Randomize Remote NI failure
6	self test	MR Node NMR Router MR Peer Self-test Randomize local NI failure Randomize Remote NI failure

...

Test #

Tag

Procedure

Script

Result

1

lnet_transaction_timeout

Set lnet_transaction_timeout to a value < retry_count via lnetctl and YAML
- This should lead to a failure to set
Set lnet_transaction_timeout to a value > retr_count via lnetctl and YAML
- lnet_lnd_timeout value should == lnet_transaction_timeout / retry_count
Show value via "lnetctl global show"

lnetctl set transaction_timeout <value>

pass

2

lnet_retry_count

Set the lnet_retry_count to a value > lnet_transaction_timeout via lnetctl and YAML
- This should lead to a failure to set
Set the lnet_retry_count to a value < lnet_transaction_timeout via lnetctl and YAML
- lnet_lnd_timeout value should == lnet_transaction_timeout / retry_count
Show value via "lnetctl global show"

lnetctl set retry_count <value>

pass

3

lnet_health_sensitivity

Set the lnet_health sensitivity from lnetctl and from YAMLShow value via "lnetctl global show"YAML
Show value via "lnetctl global show"

lnetctl set health_sensitivity <value>

Jira

server	Whamcloud Community Jira
serverId	8bba2dd1-4333-3006-bfcd-f35d4ebbd2ad
key	LU-11530

4

NI statistics

verify LNet health statistics
- lnetctl net show -v 3

pass

5

Peer NI statistics

verify LNet health statistics for peer NIs
- lnetctl peer show -v 3

pass

6

NI Health value

verify setting the local NI health statistics
- lnetctl net set --nid <nid> --health <value>
Redo from YAML

Jira

server	Whamcloud Community Jira
serverId	8bba2dd1-4333-3006-bfcd-f35d4ebbd2ad
key	LU-11529

7

Peer NI Health value

verify setting the local NI health statistics
- lnetctl peer set --nid <nid> --health <value>
Redo from YAML

Jira

server	Whamcloud Community Jira
serverId	8bba2dd1-4333-3006-bfcd-f35d4ebbd2ad
key	LU-11529

Testing Tools

The drop policy has been modified to drop outgoing messages with specific errors. This can be done via the following commands. Unfortunately, for details on these commands you'll need to look at the code. A combination of these commands on the different nodes should cover approximately 75% of the health code paths.

...

Space shortcuts

Page tree

Versions Compared

Old Version 17

New Version Current

Key

Selection Algorithm Scenarios

Error Scenarios

Synchronous Errors

Asynchronous Errors

Random Failures

Testing Tools

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 17

New Version Current

Key

Selection Algorithm Scenarios

Error Scenarios

Synchronous Errors

Asynchronous Errors

Random Failures

Testing Tools