View Source

Target release	Release name or number
Epic
Document status
Document owner	Amir Shehata
Designer	Amir Shehata
Developers	Amir Shehata, Sonia Sharma
QA	Lead tester

Goals

Test LNet via the Dynamic LNet Configuration interface
Provide an easy to use and expandable infrastructure to create and add test scripts
Provide a method to easily parse results and understand the health of the LNet Code.

Background and strategic fit

Currently there exists very little testing for LNet. With the expected LNet features to be added in the future, it is important to be able to test LNet as thoroughly as possible, as well as be able to regression test existing functionality. The LUTF fulfills that need.

Overview

The LUTF provides a master/slave work flow. The LUTF Master is deployed on one node and the Agents are deployed on the test nodes. The master controls the agent by directing them to execute python tests scripts. The master manages the interactions between the different LUTF agents.

Each agent collects the result from the test script and transfers it to the master. The master parses and summarizes the results in a YAML file.

The LUTF provides infrastructure for the test scripts to pull information about the different nodes on which the agent runs. This way the script writer only concentrates on writing testing logic, rather than writing code to run the test scripts or collect results. It also provides an abstraction layer, so that the scripts don't need to have prior knowlege of the nodes the tests will run on.

Automating the LUTF

To increase the chances of people using the LUTF, its interface must be user friendly. Here are the key aspects that must be streamlined

Deploying the LUTF on a system, which includes installing any dependences:
1. The LUTF shall be included in Lustre's test rpm. So installing that RPM will automatically install the LUTF
2. The LUTF dependencies. The LUTF shall provide a script which is simple to execute to install all dependencies
3. The LUTF shall officially support CentOS only.
Starting the LUTF can be done in two ways, a fully automated way or a more manual way. For the manual way:
1. The LUTF shall accept a YAML config on the master which identifies specific configuration parameters. Everything has a default value, so if no YAML config or incomplete configuration is provided then the defaults will be used.
  1. The master address
  2. The master port: the port the listens to agents attempting to connect
  3. The daemon flag to indicate if the master should be started in daemon mode.
  4. python script path: Is the path to find test scripts
2. The LUTF shall accept a YAML config on the agent which identifies specific configuration parameters. Almost all parameters have a default value, except for master address, so if no YAML config or incomplete configuration is provided then the defaults will be used.
  1. The master address
  2. The master port
  3. The daemon flag
  4. The python script path
Starting the LUTF automatically, requires identifying the master, the agent and the test scripts to execute. The test scripts can be a directory or a specific test script.
1. The LUTF shall accept a YAML script which identifies the IP addresses of the master, agents and the test scripts to execute
2. The LUTF shall use that information to deploy the LUTF on the indicated nodes
  1. The LUTF shall assume that the Lustre test rpm has been installed on these nodes
3. The LUTF shall start an instance on the master and the agents and execute the scripts

Examples of the LUTF usage

It is assumed that python 2.7.5 or greater is already installed on the target nodes.

Manual Method

# on master and agents
#> python lutf-deploy.py

# on master
#> python lutf-start.py master-config.yaml
#LUTF-PROMPT> ## Python prompt that can be used to control the agents and execute scripts

# on agent
#> python lutf-start.py agent-config.yaml

Automated Method

# on master
#> python lutf-launch.py lutf-config.yaml

Requirements

Interface Requirements

ID	Status	Class	Description
			The LUTF shall provide an integrated python interpreter
			The LUTF shall support python 2.7.5
			The LUTF shall use SWIG to create the interface between the python front end and the C back end.
			The LUTF shall allow the execution of python scripts within the LUTF environment
			The LUTF shall provide an API to run scripts on the test nodes on which the Agent runs
			The LUTF shall provide a synchronization mechanism to enable a script running on master to synchronize between the different Agents. EX: A test could require configuring nodes, then running traffic, then changing configuration on one of the nodes, then running traffic again. Such a test would require synchronization
			The LUTF shall provide infrastructure to create and mount a Luster file system.
			The LUTF shall provide a mechanism to collect test results from the Agents
			The LUTF shall provide the results of a test run in YAML format
			The LUTF shall provide infrastructure to run a suite of tests and collect results on all the test runs
			The LUTF shall provide a set of parameters, which the AT can use to provide information on the available test nodes
			The LUTF shall provide infrastructure to deploy and run the LUTF master and LUTF Agents on the test nodes
			The LUTF shall be integrated in the existing Lustre tests
			The LUTF shall be packaged with the Lustre tests
			The LUTF shall be deployed in /usr/lib64/lustre/tests/ director
			The LUTF shall integrate with auster. auster will execute the LUTF scripts via the standard command line arguments. The provided configuration file sets up the necessary environment variable and will then execute an lutf.sh script which will invoke the LUTF master. The LUTF master will start up the agents and run the scripts specified.
			The LUTF shall provide infrastructure to run the available test suites.
			The LUTF YAML format shall be provided to Maloo to be parsed and displayed.
			The LUTF shall require the all test nodes be setup with passwordless ssh.
			The LUTF shall be packaged in the lustre test rpm
			The LUTF shall provide a method to generate test documentation for the test scripts provided.

LUTF Master Requirements

ID	Status	Class	Description
			The LUTF master shall provide an API to query the management IP address of the test nodes on which the Agent runs
			The LUTF master shall provide an API to query all available interfaces on the tests nodes on which the Agent runs
			The LUTF master shall monitor the heartbeat of each Agent and report if an agent died.
			The LUTF master shall collect core information if a node crashes. TODO: Will this overlap with Maloo. If Maloo already provides this functionality then there is no need to duplicate it.

LUTF Agent Requirements

ID	Status	Class	Description
			The LUTF agent shall execute scripts passed in by the agent
			The LUTF agent shall report back the results to the master in YAML formate
			The LUTF shall capture all console output and include that in the results file.