| Table of Contents |
|---|
How do I build lustre on centos 7.x?
Refer to Walk-thru- Build Lustre MASTER on RHEL 7.3/CentOS 7.3 from Intel Git
Also refer to Building Lustre/LNet Centos/RHEL 7.x for some quirks when building.
How can I load LNet
Load the module
| Code Block |
|---|
modprbe lnet |
if using standard /etc/modprob.d/lustre.conf for module parameters, then:
| Code Block |
|---|
# load all the module parameters
lnetctl lnet configure --all |
if dynamically configuring then
| Code Block |
|---|
# don't configure via module parameters
lnetctl lnet configure |
How can I configure networks dynamically?
Ensure that you have loaded the lnet module and configured it as shown above
| Code Block |
|---|
lnetctl net add --net <net-name> --if <interface name>
# Examples
lnetctl net add --net o2ib --if ib0
lnetctl net add --net tcp --if eth0 |
How can I configure different tunables per network interface?
Sometimes nodes can have different types of interfaces, for example, MLX and OPA. It is desired to configure both of those with different tunables. To do that, we must use the YAML configuration.
| Code Block |
|---|
|
How can I configure routing?
An LNet router routes LNet messages from one LNet network to another. For example from o2ib1 to tcp2 or from o2ib1 to o2ib2. This is especially useful when you have a cluster with nodes divided on different types of fabric, like OPA and MLX.
- Configure LNet and its interfaces as shown above
- Configure a router
Code Block # enable routing lnetctl set routing 1
- Configure route entries on each of the nodes
Code Block lnetctl route add --net <destination net> --gateway <gateway nid> #Examples lnetctl route add --net o2ib1 --gateway 10.10.10.2@o2ib # The above says: # any messages destined to o2ib1 should be forwarded to 10.10.10.2@o2ib # o2ib has to be a reachable network lnetctl route add --net tcp --gateway 10.10.10.3@2o2ib --hop 2 --priority 1 # If there are multiple routes sometimes it's useful to define the priority between these routes # hop should define the number of hops to the gateway. # Unfortunately, due to legacy reasons hop and priority perform the same function # it would have been better to only have one to reduce confusion. # routes with less number of hops or with higher priority are selected first # if routes have the same number of hops and priority they are visited in round-robin.
What tunables impact routing?
A router can be configured with a set number of buffers. These buffers are used to receive messages to be forwarded
There are three categories of buffers:
- Tiny buffers, which have no pages for 0 payload messages
- Small buffers, which consist of 1 PAGE
- Large buffers, which consist of 256 PAGEs
The number of buffers allocated can be controlled by the following module parameters.
| Code Block |
|---|
tiny_router_buffers # 512 min for each CPT
small_router_buffers # 4096 min for each CPT
large_router_buffers # 256 min for each CPT |
They can also be set dynamically via lnetctl
| Code Block |
|---|
|
Other parameters of interest
| Code Block |
|---|
check_routers_before_use # Assume routers are down and ping them before use
avoid_asym_router_failure # Avoid asymmetrical router failures (0 to disable)
dead_router_check_interval # Seconds between dead router health checks (<= 0 to disable)
live_router_check_interval # Seconds between live router health checks (<= 0 to disable)
router_ping_timeout # Seconds to wait for the reply to a router health query |
What is a router pinger?
What is LNet Multi-Rail?
How can I configure multiple network interfaces per network?
How can I statically configure Multi-Rail?
How can I dynamically configure Multi-Rail?
How should I configure a system to use LNet Multi-Rail?
What's the best OPA LND tunables to use?
What's the best HFI tunables to use with Luster?
| Code Block |
|---|
options hfi1 krcvqs=8 piothreshold=0 sge_copy_mode=2 wss_threshold=70 |