...
For example, inconsistent MTU setting in the system can often be found responsible for bulk transfer failures
System Logs
Collection
A proper "sosreport" or "showall" package should contain the system/kernel logs. Console logs may also be of interest.
Sometimes custom sosreport generations from clients may actually be missing the logs. It is worthwhile to check that the logs are included in the package.
Make sure to collect the logs from all nodes participating in the scenario which is being investigated. For example, if the issue is that a particular client is failing to mount Lustre FS, then collect the logs from the client and all of the servers it can use.
Bits for Analysis
Startup
When LNet starts and is able to initialize an NI successfully, it logs something like
LNet: Added LNI 10.1.0.101@o2ib
This can be used to check if NI is coming up using the expected interface and IP.
Errors
Scanning for LNet and Lustre errors in the logs may be useful to quickly establish the problem area.
Other errors which may be important if they are logged in the same time period are anything from the Linux networking services, e.g. any notifications of network interfaces status change.
Mellanox Driver Dumps
If Mellanox HW is involved, look for messages from the MLNX driver in the kernel logs. Basically, scan for occurrences of "mlx5_core" or "mlnx" ("5" may be a different number depending on the driver version).
Some output from Mellanox driver is expected at system startup and may be useful to verify the driver version, for example. However, later the driver is supposed to be silent if everything is normal and anything it dumps in or around the problem period may contain information which is key.
Lnetctl Outputs
Here are some important lnetctl outputs which should be collected manually if they are not included with showall/sosreport automatically
- lnetctl global show
- lnetctl stats show
- lnetctl net show -v 4
- lnetctl peer show -v 4 (may produce large amount of data on large systems with many peers)
- lnetctl routing show
Scenarios
Connectivity
LNet Routing
...