Versions Compared

Old Version 102

changes.mady.by.user Amir Shehata

Saved on

compared with

New Version 103

changes.mady.by.user Amir Shehata

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. ncurses library
    1. yum install ncurses-devel
  2. readline library
    1. yum install readline-devel
  3. python 2.7.5
    1. https://www.python.org/download/releases/2.7.5/
    2. ./configure --prefix=<> --enable-shared # it is recommended to install in standard system path
    3. make; make install
  4. setuptools
    1. https://pypi.python.org/pypi/setuptools
    2. The way it worked for me:
      1. Download package and untar
      2. python2.7 setup.py install
  5. psutils
    1. https://pypi.python.org/pypi?:action=display&name=psutil
      1. untar
      2. cd to untared directory
      3. python2.7 setup.py install
  6. netifaces
    1. https://pypi.python.org/pypi/netifaces
  7. Install PyYAML
    1. pip isntall pyyaml

The LUTF will also require that passwordless ssh is setup for all the nodes which run the LUTF. This task is already done when the AT sets up the test cluster.

Building the LUTF

The LUTF shall be integrated with the Lustre tests under lustre/tests/lutf. The LUTF will be built and packaged with the standard

Code Block
sh ./autogen.sh
./configure --with-linux=<kernel path>
make
# optionally
make rpms
# optionally
make install

The make system will build the following items:

  1. 3.6+
    1. yum install python3
  2. paramiko
    1. pip3 install paramiko 
  3. netifaces
    1. pip3 install netifaces 
  4. Install PyYAML
    1. pip3 install pyyaml 

The LUTF will also require that passwordless ssh is setup for all the nodes which run the LUTF. This task is already done when the AT sets up the test cluster.

Building the LUTF

The LUTF shall be integrated with the Lustre tests under lustre/tests/lutf. The LUTF will be built and packaged with the standard

Code Block
sh ./autogen.sh
./configure --with-linux=<kernel path>
make
# optionally
make rpms
# optionally
make install

The make system will build the following items:

  1. lutf binary
  2. liblutf_agent.so - shared library to communicate with the LUTF backend.
  3. clutf_agent.py and _clutf_agent.so: glue code that allows python to call functions in liblutf_agent.so
  4. clutf_global.py  and _clutf_global
  5. lutf binary
  6. liblutf_agent.so - shared library to communicate with the LUTF backend.
  7. clutf_agen.py and _clutf_agent.so : glue code that allows python to call functions in liblutf_agentglobal.so

  8. lnetconfig.py and _lnetconfig.so  - glue code to allow python test scripts to utilize the DLC interface.

The build process will check if python 2.7.5 and SWIG 23.6 and SWIG 3.0 or higher is installed before building. If these requirements are not met the LUTF will not be built

...

TaskDescription
C infrastructure
  • lutf binary
  • listener thread
  • Heart beat
  • python integration
    • Look into having a choice between python 3.x and python 2.7.x
  • IPC
    • Manage connections between the master and the agents
    • Track the agents
    • Provide APIs for Request/Response Pair
      • These APIs will block in the calling thread until a response is received
      • TODO: What happens if we're calling these APIs from separate Python threads?
        • What I'm trying to get at is to see how a script can spawn python threads. These threads can do RPC. While the main test thread can continue doing other test logic.
  • API for managing and querying the state kept by the C infrastructure
    • agent information
SWIG
  • SWIG infrastructure to call C APIs
    • liblnetconfig
    • LUTF Agent Management
    • LUTF RPC
lutf.sh
  • Spawn the master and agents appropriately
  • Pass to the master the suite or specific test to run. If nothing is provided all suites are run.
  • Waits on the master until it exits after running the tests
lutf Python Library
  • Association between Agents and node roles (MGS/MDD/etc)
    • IE build a view of the clustre as identified by the provided environment variables.
  • API for querying the Agents
  • Automatically loaded and initialized
  • API for suites and scripts management and execution
  • Use the lutf Provisioning Library to clean the clustre before running each test.
lutf Provisioning Library
  • API to provision LNet and lnet_selftest
  • API to provision the Lustre File System
    • API should take a dictionary of the different nodes and based on the node types it spawns a simple File system
  • Both APIs can be used together.
    • use the LNet provisioning API to provision and configure LNet
    • use the Lustre FS provisioning API to provision the File system on top of the configured LNet 
  • API to un-provision a clustre described in a python dictionary 
lutf logging infrastructure
  • Set lustre logging levels
  • Collect lustre logs
  • collect syslogs
  • Provide debugging level infrastructure for the test scripts (probably just use the provided Python logging)
  • API for storing YAML results.

OLD INFORMATION

  • python dictionary 
lutf logging infrastructure
  • Set lustre logging levels
  • Collect lustre logs
  • collect syslogs
  • Provide debugging level infrastructure for the test scripts (probably just use the provided Python logging)
  • API for storing YAML results.


OLD INFORMATION

TODO: Below is old information still being cleaned up

Test Environment Set-Up

Each node which will run the LUTF will need to have the following installed

  1. ncurses library
    1. yum install ncurses-devel
  2. readline library
    1. yum install readline-devel
  3. python 2.7.5
    1. https://www.python.org/download/releases/2.7.5/
    2. ./configure --prefix=<> --enable-shared # it is recommended to install in standard system path
    3. make; make install
  4. setuptools
    1. https://pypi.python.org/pypi/setuptools
    2. The way it worked for me:
      1. Download package and untar
      2. python2.7 setup.py install
  5. psutils
    1. https://pypi.python.org/pypi?:action=display&name=psutil
      1. untar
      2. cd to untared directory
      3. python2.7 setup.py install
  6. netifaces
    1. https://pypi.python.org/pypi/netifaces
  7. Install PyYAML
    1. pip isntall pyyaml

The LUTF will also require that passwordless ssh is setup for all the nodes which run the LUTF. This task is already done when the AT sets up the test cluster.TODO: Below is old information still being cleaned up

LUTF Configuration Files

Setup YAML Configuration File

...