...

Figure 1: System Level Diagram

...

1. Take as input the following parameters. NOTE: These parameters can be provided as a set of environment variables, or can be placed shall be specified in a YAML file and then the path of the YAML file can be passed to the lutf_perform_test.py script. The second option will be assumed in this HLD.
   
   1. IP address of node to be used for master
   2. IP addresses of nodes to be used as agents
   3. Two YAML configuration files for the Master and Agent nodes.
   4. YAML configuration file describing the tests to run.
2. Call the lutf_deploy.py script for each of the nodes provided.
3. Call the lutf_launch.py script for each of the nodes provided. Pass the Master YAML LUTF Configuration file to the master node that and the agent configuration file to the agent nodes.
   1. Query the LUTF master to ensure the expected number of agents are connected.
   2. If everything is correct, then continue with the tests, otherwise build a YAML block describing the error.
4. Send the test YAML configuration file to the LUTF master and wait.
5. Once the tests are completed the LUTF master will return a YAML block describing the test results, described below
   1. the LUTF Master will provide an API based around paramiko. The API is described below.

...

1. A separate section is to be created in Maloo to display LUTF test results.
2. The results from output YAML file passed from AT are displayed in the LUTF results section.
3. A Test-parameter specifically for LUTF tests to be defined that will allow to run only LUTF tests. This will help in avoiding unnecessary tests to run for only LNet related changesLUTF tests will only run if that Test-parameter is set. There is no need to run LUTF for every patch committed.

Improvements

1. Currently the LUTF is designed to have the Python infrastructure establish a Telnet connection to facilitate Master to scp the test scripts to Agent and then execute those test scripts. The Telnet approach can be improved upon by using SSH instead. 
2. A synchronization mechanism can be added to synchronize the different parts of one test script running on different Agents by providing an API that uses notification mechanism. The Master node will control this synchronization between different Agent nodes that are used for running a test script. An example scenario of how it would be implemented is -If a test script is such that it requires to do some operation on more than one Agent node, then as one part of a test script runs to it completion on one Agent, it would notify the Master about its status by calling this API and then Master can redirect this event to the main script waiting on it which will trigger the other part (operation) to start execution on another Agent node.

...

#!/bin/bash
#Key Exports
export master_HOST=onyx-15vm1
export agent1_HOST=onyx-16vm1
export agent2_HOST=onyx-17vm1
export agent3_HOST=onyx-18vm1
export AGENTCOUNT=3

VERBOSE=true
  
# ports for LUTF Telnet connection
export MASTER_PORT=8494
export AGENT_PORT=8094

# script and result paths
script_DIR=$LUSTRE/tests/lutf/python/test/dlc/
output_DIR=$LUSTRE/tests/lutf/python/tests/

...