Video Presentation
| Widget Connector | ||||||
|---|---|---|---|---|---|---|
|
Slide Deck
Video Transcript
Slide 1 - Introduction
Hello, my name is Amir Shehata. I'm the lead Luster Networking Engineer at DDN Storage.
Today I'd like talk to you about the Multi Rail and Health Lustre networking capabilities Features in the EXAScaler product [ Luster Networking Layer ].
Slide 2 - Agenda
I'll start off by explaining what Multi-Rail is all about and what it offers us
...
From there I'll highlight the benefits and give some ideas on best configuration practices.
So Let's get started
Slide 3 - What is Multi-Rail and Health
In this presentation, when I speak about Multi-Rail, I'm referring to two related features: Multi-Rail and Health.
...
To explain this further, let's say, we have two interfaces in the o2ib network, IB0 and IB1. If LNet fails to send on IB0, then it will decrement the health of that interface and retry on the IB1 interface. It will then keep monitoring the IB0 interface until it's sure it is healthy again. In the mean LNet will keep using IB1, the healthier interface.
Slide 4 - Without MR
Now let's look at an example. The DGX-2, NVIDIA's AI box, can have up to 8 different interfaces. Without MR each one of these interfaces will need to be configured in its own LNet network. And since the EXAScaler servers do not have as many interfaces, we will have to alias these interfaces and connect the aliases to the different networks. Complicated configuration for sure.
Slide 5 - With MR
Once we throw MR into the mix, the configuration becomes very simply. Only one LNet network, and all the interfaces of all the nodes, clients and servers, are connected to that network. Of course this is only one potential configuration.
Slide 6 - Dual
...
Fabric
The key point I'm trying to make is now we can match the underlying fabric. If we're dealing with two fabrics, OPA vs IB as an example, or even if the underlying fabric physical network has two segments, then Multi-Rail allows us to match the underlying fabric configuration. This makes configuration much more intuitive and less complicated.
Slide 7 - MR Benefits
Now that we have a basic understanding of what LNet Multi-Rail brings to the table, we can see how it simplifies the network configuration.
...
And as I mentioned before even if we have two different LNet networks each with one interfaces, we get the same performance scaling, because LNet can use both of them simultaneously to communicate with peers.
Slide 8 - Interface Selection
With large client machines like the DGX-2, it's not enough to select an interface to use at random. There are restrictions imposed on us by the HW.
...
With these criteria LNet can make intelligent decisions about which interface to use.
Slide 9 - Performance
Here is a real life example of the benefits afforded to us by Multi-Rail. This graph shows a comparison performance test done between a GPU Direct workload and a CPU workload. The test was performed on a DGX-2 and 2 AI 400 servers. As you can see LNet Multi-Rail benefits both workloads, however due to the GPU's shorter latency, it's able to achieve higher performance, almost saturating the AI-400 servers bandwidth.
Slide 10 - Configuring MR
Configuring Multi-Rail is dead simple. In fact no work is needed to configure it. It's on by default. All we need to do is group the interfaces in the same LNet network.
...
You can if you want turn off Multi-Rail, although I wouldn't recommend it. You can do that by turning off the automatic interface discovery feature, as I show on the slide. By disabling discovery the node will not know about the peer's Multi-Rail capability. This can be useful for handling backwards compatibility with peers which are not Multi-Rail capable.
Slide 11 - Health Benefits
Now let's switch to health.
...
For subsequent messages the healthiest interface is used.
Slide 12 - When to use Health
Health is most beneficial when you have multiple interfaces or networks which you can fail to. But if a node has only one interface, the benefits of the health feature is not as pronounced. There is only one interface to use, no matter how unhealthy it becomes.
Slide 13: Health Paremters
There are three parameters which control the health feature.
...
To turn health off, set the health_sensitivity and retry_count to 0. The interface's health will not be decremented and messages will not be re-sent on failure.
Slide 14 - Conclusion
In conclusion, we saw how the LNet Multi-Rail feature simplifies network configuration and increases the performance of the node by aggregating the interfaces' bandwidth.
...