A Preliminary Exploration of the LUTF

Slide Deck

Download from here.

Video Transcript

Slide 01

Hello, this is Amir Shehata with another quick tip on the LUTF. In this video we will take a 5-minute tour over how the LUTF directory is structured and how to configure it.

Slide 02

I have written a step by step walk-thru on how to build and deploy the LUTF. Please go to the link provided here or in the video transcript on the wiki for more details.

Slide 03

Once you have the Lustre tests rpm installed along with the other lustre rpms, you can see the LUTF installed in the standard lustre tests directory:

 /usr/lib64/lustre/tests/lutf

Slide 04

The LUTF directory structure is fairly straight forward.

The top level directory contains the LUTF binary and a set of other generated shared libraries and python files.

These include the C/Python wrappers which the LUTF python scripts use to call the C APIs.

Slide 05

Let's look at an example. You can see here two files of interest: _lnetconfig.so and lnetconfig.py.

The .py file is generated by SWIG and enables the LUTF python scripts to call the liblnetconfig API

The .so file is a shared library which includes the glue code that implements the liblnetconfig API calls

Slide 06

Configuration scripts are under the python/config  directory. These configuration scripts setup all the different elements required to run the LUTF.

Various infrastructure scripts, which implement the remote script execution feature as well as other LUTF features, are under the infra directory.

Lustre and LNet specific infrastructure scripts, such as scripts to clean the test setup and configure LNet are located in the tests-infra directory

Finally all the various tests suites are located under the tests directory.

At the time of this recording, the shown test suites are available.

Slide 7

The LUTF requires a single YAML configuration file which specifies its environment. Let's briefly explore the configuration parameters.

If you remember I mentioned that the LUTF can run in master or agent mode. Some parameters are relevant to the agent and not the master.

Although all parameters can be specified. On startup only the parameters which matter are examined.

Slide 8

First off, let's look at the shell parameter. It defines how the LUTF will run. There are three modes it can run in:

  • First, the interactive mode. In this mode you can access the python shell. You can do everything you would normally do from a regular python shell from there.
  • Second, the daemon mode. As the name suggests the LUTF will run as a daemon process. However, its shell can still be access by TELNETTing to it.
  • Finally, the batch mode. Now this mode is unique to the master and it basically runs all the specified tests and then shuts down.

Slide 9

The agent parameter tells the LUTF if it should run as a master or an agent

node-name specifies a symbolic name for the node. It can be any alpha-numeric value.

The LUTF agent will need to know the master's IP address or hostname which it will use to connect to the master.  A symbolic name is also provided to associate with the master.

The LUTF will require a port to listen on for incoming connections as well as a port to use for the telnet server

Slide 10

This is what the LUTF YAML configuration looks like. A couple of other parameters relevant to the master node are:

  • the suite. This defines a suite name to run when in batch mode. If nothing is specified the LUTF will run all available suites
  • and the script. This defines the script name to run and it's only valid if a suite is provided.

Both the suite and the script names can contain wild cards which define a group of suites or scripts.

You can follow the shown link for more details on the configuration parameters.

Before I leave this topic, I just want to remind you that the YAML configuration file will be automatically generated for you, as I'll show you later on.

Slide 11

We're almost ready to run the LUTF, but first just a quick note on a couple of important naming conventions.

All test suites under the python/tests  directory must start with the literal: suite_ 

And all the tests under a test suite must start with the literal: test_ 

Otherwise the LUTF will not pick them up and you won't be able to run them.

In the next tutorial we will run the LUTF and see what it offers us.

LUTF YAML Configuration

lutf:
   shell: interactive # [interactive|batch|daemon]
   agent: false # [true|false] run as an agent or master
   telnet-port: 49304 # a port to telnet to, to access agent
   master-address: 192.14.4.3 # IP address of master or hostname
   master-port: 43943 # port master is listening on
   node-name: ost2_HOST # name to associate with the node.
   master-name: mds_HOST # name of master to be given to agents
   script: test_dlc_01.py # script name to run in batch mode. Suite name needs to be specified as well.
   suite: suite_dlc # suite name to run. If not present all suites are run in batch mode
   lutf-path: /usr/lib64/lustre/tests/lutf/ # path to lutf directory
   py-path: "/usr/lib64/:/usr/lib64/tests" # ':' separated lists of paths for extra python modules