Versions Compared

Old Version 4

changes.mady.by.user Amir Shehata

Saved on

compared with

New Version 5

changes.mady.by.user Amir Shehata

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

In order to properly test this feature fine grained control over the LND behavior is required. Hooks The drop/delay message policies will be added in the socklnd and o2iblnd to generate errors on demand. It will listen to IOCTL commands which will put it in "debug mode". While in "debug mode", it'll be tightly coupled with the userspace utility which initiated the debug mode. Different levels of debug will necessiate the LND to propagate events up to userspace and wait for commands back. The userspace utility can then instruct the LND to drop messages, return errors, etc.modified to simulate the various errors that could occur when sending a message. This is described in the sections below.

Selection Algorithm Scenarios

...

Test #TagProcedureScriptResult
1

LNET_MSG_STATUS_LOCAL_INTERRUPT

LNET_MSG_STATUS_LOCAL_DROPPED

LNET_MSG_STATUS_LOCAL_ABORTED

LNET_MSG_STATUS_LOCAL_NO_ROUTE

LNET_MSG_STATUS_LOCAL_TIMEOUT

  • MR Node with Multiple interfaces
  • Send a PING
  • Simulate an <error>
  • PING msg should be queued on resend queue
  • PING msg will be resent on a different interface
  • Failed interfaces' health value will be decremented
  • Failed interface will be placed on the recovery queue


2Sensitivity == 0
  • Same setup as 1
  • NI is not placed on the recovery queue


3Sensitivity > 0
  • Same setup as 1
  • NI is placed on the recovery queue
  • Monitor network activity as NI is pinged until health is back to maximum


4

Sensitivity > 0

Buggy interface

  • Same setup as 1
  • NI is placed on recovery queue
  • NI is pinged ever 1 second
  • Simulate ping failure ever other ping
  • NI's health should be decremented on failure
  • NI should remain on the recovery queue


5Retry count == 0
  • Same setup as 1
  • Message will not be retried and the message will be finalized immediately


6Retry count > 0
  • Same setup as 1
  • Message will be transmitted for a maximum of retry count or until the message expires


7REPLY timeout
  • Same setup as 1
  • Except Use LNet selftest
  • Simulate a local timeout
  • Re-transmit
  • No REPLY received
  • Message is finalized and TIMEOUT event is propagated.


8ACK timeout
  • Same setup as 7 except simulate ACK timeout


9LNET_MSG_STATUS_LOCAL_ERROR
  • Same setup as 1
  • Message is finalized immediately (not resent)
  • Local NI is placed on the recovery queue
  • Same procedure to recover the local NI


10LNET_MSG_STATUS_REMOTE_DROPPED
  • Same setup as 1
  • Message is queued for resend depending on retry_count
  • peer_ni is placed on the recovery queue (not if sensitivity == 0)
  • peer_ni is pinged every 1 second


11

LNET_MSG_STATUS_REMOTE_ERROR

LNET_MSG_STATUS_REMOTE_TIMEOUT

LNET_MSG_STATUS_NETWORK_TIMEOUT

  • Same setup as 1
  • Message is not resent
  • peer_ni recovery happens as outlined in previous cases


Random Failures

Test #TagProcedureScriptResult
1self test
  • MR Node
  • NMR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


2self test
  • MR Node
  • MR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


3self test
  • MR Node 
  • MR Router
  • NMR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


4self test
  • MR Node 
  • MR Router
  • MR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


5self test
  • MR Node 
  • NMR Router
  • NMR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


6self test
  • MR Node 
  • NMR Router
  • MR Peer
  • Self-test
  • Randomize local NI failure
  • Randomize Remote NI failure


User Interface

Test #TagProcedureScriptResult
1lnet_transaction_timeout
  • Set lnet_transaction_timeout to a value < retry_count via lnetctl and YAML
    • This should lead to a failure to set
  • Set lnet_transaction_timeout to a value > retr_count via lnetctl and YAML
    • lnet_lnd_timeout value should == lnet_transaction_timeout / retry_count
  • Show value via "lnetctl global show"


2lnet_retry_count
  • Set the lnet_retry_count to a value > lnet_transaction_timeout via lnetctl and YAML
    • This should lead to a failure to set
  • Set the lnet_retry_count to a value < lnet_transaction_timeout via lnetctl and YAML
    • lnet_lnd_timeout value should == lnet_transaction_timeout / retry_count
  • Show value via "lnetctl global show"


3lnet_health_sensitivity
  • Set the lnet_health sensitivity from lnetctl and from YAML
  • Show value via "lnetctl global show"


4NI statistics
  • verify LNet health statistics
    • lnetctl net show -v 3


5Peer NI statistics
  • verify LNet health statistics for peer NIs
    • lnetctl peer show -v 3


6NI Health value
  • verify setting the local NI health statistics
    • lnetctl net set --nid <nid> --health <value>
  • Redo from YAML


7Peer NI Health value
  • verify setting the local NI health statistics
    • lnetctl peer set --nid <nid> --health <value>
  • Redo from YAML


Testing Tools

The drop policy has been modified to drop outgoing messages with specific errors. This can be done via the following commands. Unfortunately, for details on these commands you'll need to look at the code. A combination of these commands on the different nodes should cover approximately 75% of the health code paths.

Code Block
lctl net_drop_add -s *@tcp -d *@tcp -m ACK -i 20
lctl net_drop_add -s *@tcp -d *@tcp -m REPLY -i 20
lctl net_drop_add -s *@tcp -d *@tcp -m GET -i 1 -e random
lctl net_drop_add -s *@tcp -d *@tcp -m PUT -i 20 -e random
lctl net_drop_del -s *@tcp -d *@tcp

The -e parameter can take the following arguments

Code Block
local_interrupt 		# will result in a resend
local_dropped			# will result in a resend
local_aborted			# will result in a resend
local_no_route			# will result in a resend
local_error				# will not result in a resend
local_timeout			# will result in a resend
remote_error			# will not result in a resend
remote_dropped			# will result in a resend
remote_timeout			# will not result in a resend
network_timeout			# will not result in a resend
random
drop_send 				# will drop the message and never call lnet_finalize()

The -e can be repeated multiple times to specify different errors. The error simulated will be selected randomly from the ones defined. random can be given to select any error to simulate at random.

Simulation Details

The error simulation occurs immediately before putting the message on the wire. If the drop rule policy is defined and is matched, then the message is not sent and error is simulated.

Types of Error Simulation Testing

TypeDescription
Drop with error

This is the newly added error simulation. And it is designed to simulate different health failures. This can be used to exercise the following scenarios

  1. re-transmit message scenair
  2. Immediate failure and message finalization
  3. Local NI recovery
  4. Peer NI recovery
  5. Finalizing active message
Drop Received messagesThis an existing rule and it can be used to drop received GET/PUT messages. This will result in no ACK/REPLY being sent to the message initiator and will exercise the response timeout code.

Configuration

LNet Health is off by default. To turn it on, two configuration parameters need to be set

  1. Retry counter. This will indicate how many times a message should be resent before it succeeds or times out. It default to 0, which means no message re-transmission will occur.
    1. lnetctl set retry_count <value>
  2. Health sensitivity. This is a value by which to decrement the health of the interface. When an interface health value goes below the optimal value, it gets placed on a recovery queue and will be pinged every second until its health recovers. This value by default is 0, which means that a peer or local NI will never go into recovery state.
    1. lnetctl set health_sensitivity <value>
  3. Transaction timeout. This is the timeout value to wait before a response expires or before a message on the active list expires.
    1. lnetctl set transaction_timeout <value>