This document intends to provide guidelines for troubleshooting common issues involving LNet as seen in the field, with the idea that non-developers should be able to diagnose most of such problems, and, if solution is not achievable on the spot, provide developers with adequate detail for timely investigation.
From Lustre POV LNet is the layer providing the abstraction of physical network. LNet module is an "umbrella" module over a number of LNDs (Lustre Network Drivers) dealing with specific network types.
Because Lustre is a distributed FS, lots of errors in the system may have the markers of a "network error", and it may be not always trivial to determine whether LNet is actually at fault.
It is important to know how the customer is using the network with Lustre.
It should be made clear how Lustre and LNet modules are configured.
Separately, configuration outside of Lustre/LNet should be checked also:
It is important to know whether all nodes of the same type are similar and are configured the same. Check the following across the nodes of the same type:
For example, inconsistent MTU setting in the system can often be found responsible for bulk transfer failures
A proper "sosreport" or "showall" package should contain the system/kernel logs. Console logs may also be of interest.
Sometimes custom sosreport generations from clients may actually be missing the logs. It is worthwhile to check that the logs are included in the package.
Make sure to collect the logs from all nodes participating in the scenario which is being investigated. For example, if the issue is that a particular client is failing to mount Lustre FS, then collect the logs from the client and all of the servers it can use.
When LNet starts and is able to initialize an NI successfully, it logs something like
LNet: Added LNI 10.1.0.101@o2ib
This can be used to check if NI is coming up using the expected interface and IP.
Scanning for LNet and Lustre errors in the logs may be useful to quickly establish the problem area.
Other errors which may be important if they are logged in the same time period are anything from the Linux networking services, e.g. any notifications of network interfaces status change.
If Mellanox HW is involved, look for messages from the MLNX driver in the kernel logs. Basically, scan for occurrences of "mlx5_core" or "mlnx" ("5" may be a different number depending on the driver version).
Some output from Mellanox driver is expected at system startup and may be useful to verify the driver version, for example. However, later the driver is supposed to be silent if everything is normal and anything it dumps in or around the problem period may contain information which is key.
Here are some important lnetctl outputs which should be collected manually if they are not included with showall/sosreport automatically
There's currently no way to trigger a rest for any of the counters shown by lnetctl. For live debugging it is important to take "before" and "after" snapshots and/or use "watch" linux command to track changes of a particular stat.