Introduction

Currently there is no dedicated functional test tool in Lustre test suites for LNet testing. Lustre Unit Test Framework (LUTF) fills that gap to provide a means for testing existing LNet features as well as new features that would be added in future. It facilitates an easy way of adding new test cases/scripts to test any new LNet feature.

Objectives 

This High Level Design Document describes the current LUTF design, code base, infrastructure requirements for its setup and the new features that can be added on top of the current design.

Reference Documents

Document Structure

This document is made up of the following sections:

• Design Overview
• Building the LUTF
• LUTF-Autotest Integration
• Infrastructure

LUTF Design Overview

The LUTF is designed with a Master-Agent approach to test LNet. The Master and Agent LUTF instance uses a telnet python module to communicate with each other and more than one Agent can communicate with single Master instance at the same time. The Master instance controls the execution of the python test scripts to test LNet on Agent instances. It collects the results of all the tests run on Agents and write them to a YAML file. It also controls the synchronization mechanism between test-scripts running on different Agents.

The below diagram shows how LUTF interacts with LNet

Figure 1: System Level Diagram

Building the LUTF

To build LUTF, it first requires to set up an environment with all the required packages installed and then building using GNU build system like Lustre tree is built.  
Following sub sections briefs on the steps for the building process.

Environment Set-Up

1. Python 2.7.5 is required along with some other python related packages like -
   
   1. netifaces
   2. PyYAML
   3. paramiko (some MR test scripts are written using paramiko, so need to have this installed too)
2. SWIG (Simplified Wrapper and Interface Generator) is required to generate a glue code to allow the python test scripts call DLC APIs.
3. Password less SSH - Nodes running LUTF are required to setup password less SSH to each other.

Build along Lustre tree using GNU tools

1. All the other test suites/scripts for lustre are placed under lustre/tests/ directory. Place LUTF as well under lustre/tests.
2. Mention LUTF as a subdirectory to be build in lustre/tests/Makefile.am
3. *  Create an autoMakefile.am under lustre/tests/ and also under lustre/tests/lutf/ .
4. Create a Makefile.am under lustre/tests/lutf/ to generate the required binary files and swig files. 
   1. It would also require to modify configure.ac under lustre tree parent directory to add python path and other dependencies.
   2. Add the LTLIBRARIES and SOURCES to generate the swig wrapper files.
5. Run "make distclean" to clean up any residual build artifacts.
6. cd to lustre tree parent directory and run "sh autogen.sh"
7. Run "./configure"
8. Run "make"

LUTF/AT Integration

For LUTF-Autotest integration, the first step in this process is to build LUTF along with lustre just like other test-suites are build. The previous step "Build along Lustre tree using GNU tools" discussed fulfills this purpose.  Once, the LUTF is build along with lustre, we have all the needed binary files and swig generated wrapper files used to run python test-scripts. After this -

1. The config file (similar to how Auster has) provided for LUTF under lustre/tests/cfg/  is used to identify nodes involved in test-suite and set up environment variables.
2. AT runs the Master script which reads the config file and set up LUTF on the identifies nodes and triggers the execution of test-suite on the Agent nodes.
3. AT collects the results of the test-suite in the form of a YAML file (similar to Auster) and then passes the results to Maloo.

Infrastructure

Automatic Deployment

With LUTF-Autotest integration, an infrastructure is created that makes AT to deploy LUTF on the test nodes, collect results of the tests run and then pass the test results to Maloo to be displayed there.

Deploy LUTF

1. A config file (similar to what Auster has) is provided by AT which can define and set the environment variables . This file would also have information about the nodes involved in test-suite and their IP addresses.
2. A Master script is created which can read the IP addresses of the nodes involved in the test-suite from the config file and run LUTF on the identified Agent and Master nodes.
3. This Master script also triggers to run a child script that can fetch the information about the Network Interfaces (NIDs) on all the nodes involved in test-suite. 
   
   1. This information of NIDs can then further be provided to each batch test (scripts to run all the similar tests related to one feature bundled together) to execute.
4. The Master script then triggers the batch test script to run on the Agent nodes through the Master node identified to be used for the test-suite.

Collect Results

1. A YAML format is decided for the results of the entire test-run and a result YAML file is generated per that format.
2. The YAML file also points to the path where the test result file for each test is stored.
3. This YAML file is then passed to AT which further passes it to Maloo.
   
   

 Maloo

1. A separate section is to be created in Maloo to display LUTF test results.
2. The results from output YAML file passed from AT are displayed in the LUTF results section.
3. A Test-parameter specifically for LUTF tests to be defined that will allow to run only LUTF tests. This will help in avoiding unnecessary tests to run for only LNet related changes.

C Backend

This allows for the setup of TCP connection (TCP sockets) to connect the Master and Agent nodes (lutf.c).  LUTF can be run on a node in either Master mode or an Agent mode. 

1. Master mode:

   1. Spawns a listener thread (lutf_listener_main) to listen to Agent connections (lutf.c).

   2. Maintains a list of the Agents, check on the health of Agents, associate and disassociate with Agents (liblutf_agent.c).
   3. Start up a python interpreter (lutf_python.c).

2. Agent mode:

   1. Spawns a heart beat thread (lutf_heartbeat_main) to send a heart beat  to master every 2 seconds. The master uses this Heart beat signal to determine the aliveness of the agents (lutf.c).
   2. Start up a python interpreter (lutf_python.c).

Python

Script execution and result collection

1. A telnet connection is established from Master to Agent when we create a Script instance by running lutf_script.Script('local_intf',  'script_path ',  'output_dir')  (lutf_script.py).
2. The scripts from 'script_path' in lutf_script.Script('local_intf',  'script_path ',  'output_dir') are copied over to Agent using scp (lutf_agent_ctrl.py).
3. The copied scripts are then executed by calling run_script() on the Script instance created. (lutf_agent_ctrl.py).
4. If an 'output_dir' path is specified then the results of the script execution are copied to the path given by calling push_results(). If no path is provided for the 'output_dir' then the results are ignored.

Improvements

1. Currently the LUTF is designed to have the Python infrastructure establish a Telnet connection to facilitate Master to scp the test scripts to Agent and then execute those test scripts. The Telnet approach can be improved upon by using SSH instead. 
2. A synchronization mechanism can be added to synchronize the different parts of one test script running on different Agents by providing an API that uses notification mechanism. The Master node will control this synchronization between different Agent nodes that are used for running a test script. An example scenario of how it would be implemented is -If a test script is such that it requires to do some operation on more than one Agent node, then as one part of a test script runs to it completion on one Agent, it would notify the Master about its status by calling this API and then Master can redirect this event to the main script waiting on it which will trigger the other part (operation) to start execution on another Agent node.

Batch test 

1. All the similar test scripts (pertaining to one feature like multi-rail or Dynamic Discovery) are bundled in one auto-test script which when executed, runs all the test-scripts listed in it and then post the cumulative results after execution. 

2. There is an auto-test script for each of the bundle of test-scripts related to one feature.

3. The result file for each individual test script is also placed in lutfTMP directory on Agent node.

Improvements

1. The above design can be changed to have all the test scripts related to a feature placed in separate directory under LUTF/python/tests/ and then have single auto-test script which will trigger the execution of all the test scripts under one folder. The name of the folder can be passed as a parameter to this auto-test script.