Introduction

Currently there is no dedicated test tool in Lustre test suites for LNet testing. Lustre Unit Test Framework (LUTF) fills that gap to provide a means for testing existing LNet features as well as new features that would be added in future. It facilitates an easy way of adding new test cases/scripts to test any new LNet feature.

Objectives 

This High Level Design Document describes the current LUTF design, code base, infrastructure requirements for its setup and the new features that can be added on top of the current design.

Reference Documents

Document Structure

This document is made up of the following sections:

• Design Overview
• Building the LUTF
• LUTF-Autotest Integration
• Infrastructure

LUTF Design Overview

The LUTF is designed with a Master-Agent approach to test LNet. The Master and Agent LUTF instance uses a telnet python module to communicate with each other and more than one Agent can communicate with single Master instance at the same time. The Master instance controls the execution of the python test scripts to test LNet on Agent instances. It collects the results of all the tests run on Agents and write them to a YAML file. It also controls the synchronization mechanism between test-scripts running on different Agents.

The below diagram shows how LUTF interacts with LNet

Figure 1: System Level Diagram

Building the LUTF

To build LUTF, it first requires to set up an environment with all the required packages installed and then building using GNU build system like Lustre tree is built.  
Following sub sections briefs on the steps for the building process.

Environment Set-Up

1. Python 2.7.5 is required along with some other python related packages like -
   
   1. netifaces
   2. PyYAML
   3. paramiko (some MR test scripts are written using paramiko, so need to have this installed too)
2. SWIG (Simplified Wrapper and Interface Generator) is required to generate a glue code to allow the python test scripts call DLC APIs.
3. Password less SSH - Nodes running LUTF are required to setup password less SSH to each other.

Build along Lustre tree using GNU tools

1. All the other test suites/scripts for lustre are placed under lustre/tests/ directory. Place LUTF as well under lustre/tests.
2. Mention LUTF as a subdirectory to be build in lustre/tests/Makefile.am
3. Create a Makefile.am under lustre/tests/lutf/ to generate the required object files and swig files. (This step might further require to modify configure.ac under lustre tree parent directory to add python path and other things).
4. Run "make distclean" to clean up any residual build artifacts
5. cd to lustre tree parent directory and run "sh autogen.sh"
6. Run "./configure"
7. Run "make"

LUTF/AT Integration

For LUTF-Autotest integration

Infrastructure

Automatic Deployment

TBD: How does the AT deploy the LUTF, collect results, show results in Maloo

C Backend

This allows for the setup of TCP connection (TCP sockets) to connect the Master and Agent nodes (lutf.c).  LUTF can be run on a node in either Master mode or an Agent mode. 

1. Master mode:

   1. Spawns a listener thread (lutf_listener_main) to listen to Agent connections (lutf.c).

   2. Maintains a list of the Agents, check on the health of Agents, associate and disassociate with Agents (liblutf_agent.c).
   3. Start up a python interpreter (lutf_python.c).

2. Agent mode:

   1. Spawns a heart beat thread (lutf_heartbeat_main) to send a heart beat  to master every 2 seconds. The master uses this Heart beat signal to determine the aliveness of the agents (lutf.c).
   2. Start up a python interpreter (lutf_python.c).

Python

Script execution and result collection

1. A telnet connection is established from Master to Agent when we create a Script instance by running lutf_script.Script('local_intf',  'script_path ',  'output_dir')  (lutf_script.py).
2. The scripts from 'script_path' in lutf_script.Script('local_intf',  'script_path ',  'output_dir') are copied over to Agent using scp (lutf_agent_ctrl.py).
3. The copied scripts are then executed by calling run_script() on the Script instance created. (lutf_agent_ctrl.py). 
4. If an 'output_dir' path is specified then the results of the script execution are copied to the path given by calling push_results(). If no path is provided for the 'output_dir' then the results are ignored.

Batch test 

how should we execute a collection of tests. You can discuss how it's currently done, and if it can be imporved.

1. Python Test infrastructure
   
   
   1. Infrastructure Level 1:
      A python master script for this infrastructure would facilitate the following:
      1. Deploy LUTF on all the Agent nodes and Master node.
      2. Provides a telnet server and client for Master<->Agent communication.
      3. Provides a mechanism to query IP addresses and the network interfaces (NIs) on the Agents. This information can further be fetched by the test scripts on demand using an API.
      4. Facilitates running individual python tests scripts on the Agents and collecting results.
      5. Facilitates running the auto-test script which is a test-suite of all the test scripts related to one particular feature.
      6. Facilitate synchronization between the tests running on different Agents by providing an API that uses notification mechanism. An example scenario of how it would be implemented is - as a test script runs to it completion on an Agent, it would notify the Master about its status by calling this API and then Master can redirect this event to any script waiting on it.
   2. Infrastructure Level 2:
      With its implementation, the functions which are used by multiple test scripts are defined in a base test infrastructure file (lnet_test_infra_utils.py) which is then imported in each test script. This ease out the process of writing new test scripts and avoids code redundancy.