...
Also refer to Building Lustre/LNet Centos/RHEL 7.x for some quirks when building.
How
...
do I load LNet
Load the module
| Code Block |
|---|
modprbe lnet |
...
| Code Block |
|---|
# don't configure via module parameters lnetctl lnet configure |
How
...
do I configure networks dynamically?
Ensure that you have loaded the lnet module and configured it as shown above
| Code Block |
|---|
lnetctl net add --net <net-name> --if <interface name> # Examples lnetctl net add --net o2ib --if ib0 lnetctl net add --net tcp --if eth0 |
How
...
do I configure different tunables per network interface?
Sometimes nodes can have different types of interfaces, for example, MLX and OPA. It is desired to configure both of those with different tunables. To do that, we must use the YAML configuration.
...
| Code Block |
|---|
#> cat networkConfig.yaml
net:
- net type: o2ib
local NI(s):
- interfaces:
0: ib0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
lnd tunables:
peercredits_hiw: 4
map_on_demand: 0
concurrent_sends: 8
fmr_pool_size: 512
fmr_flush_trigger: 384
fmr_cache: 1
conns_per_peer: 1
- net type: o2ib1
local NI(s):
- interfaces:
0: ib2
tunables:
peer_timeout: 180
peer_credits: 128
peer_buffer_credits: 0
credits: 1024
lnd tunables:
peercredits_hiw: 4
map_on_demand: 32
concurrent_sends: 256
fmr_pool_size: 2048
fmr_flush_trigger: 512
fmr_cache: 1
conns_per_peer: 4
ntx: 2048
#> lnetctl import < networkConfig.yaml |
How
...
do I configure routing?
An LNet router routes LNet messages from one LNet network to another. For example from o2ib1 to tcp2 or from o2ib1 to o2ib2. This is especially useful when you have a cluster with nodes divided on different types of fabric, like OPA and MLX.
...
Refer to: http://wiki.lustre.org/Multi-Rail_LNet for the Requirements, HLD and LUG presentations.
How
...
do I configure multiple network interfaces per network?
Via command line:
| Code Block |
|---|
lnetctl net add --net <network> --if <list of comma separated interfaces> # Example lnetctl net add --net o2ib --if ib0,ib1 |
...
| Code Block |
|---|
net:
- net type: o2ib
local NI(s):
- interfaces:
0: ib0
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
lnd tunables:
peercredits_hiw: 4
map_on_demand: 0
concurrent_sends: 8
fmr_pool_size: 512
fmr_flush_trigger: 384
fmr_cache: 1
conns_per_peer: 1
- interfaces:
0: ib1
tunables:
peer_timeout: 180
peer_credits: 8
peer_buffer_credits: 0
credits: 256
lnd tunables:
peercredits_hiw: 4
map_on_demand: 0
concurrent_sends: 8
fmr_pool_size: 512
fmr_flush_trigger: 384
fmr_cache: 1
conns_per_peer: 1 |
How
...
do I statically configure Multi-Rail?
There are two steps to configuring multi-rail:
...
For more information on exact configuration examples, refer to: https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lnetmr
How
...
do I dynamically configure Multi-Rail?
Configuring the peers manually is error prone process. It's best if a node is able to discover its peers dynamically. The Dynamic Discovery feature allows a node to discover the interfaces of its peers the first time it communicates with it.
...
Refer to: Multi-Rail (MR) Routing
How
...
do I test LNet performance?
lnet_selftest is available for performance testing. Refer to: https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#lnetselftest
For a sample lnet_selftest script: self-test template script
Is there a way to functionaly test LNet?
We're currently working on a functional test tool, LNet Unit Test Framework. The documents will be made available soon.
What's the best OPA LND tunables to use?
...
Can you tell me more about how to configure LNet and QoS?
Refer to: Lustre QoS
How can I find out the service level of LNet traffic?
How can I change the service level of LNet traffic?
How does SL2VL mapping work in openSM?
...