Third-Party Tools Overview

A number of high quality, production ready tools have been developed by members of the Lustre community. These include:

lltop

URL: https://github.com/jhammond/lltop

Description: Lustre load monitor with batch scheduler integration. Covered in LUG2011 presentation:

An enhanced version of the tool is also available, called xltop. It is included at the bottom of this page.

Author: John Hammond (TACC)

 

LMT

https://github.com/chaos/lmt

Description: LMT is a distributed system that provides a "top"-like display of the activity levels of the server-side (MDS, OSSes, and LNET routers) nodes of one or more Lustre-based filesystems.

Author: Christopher J. Morrone (LLNL)

PCP

URL: https://github.com/wtsi-ssg/pcp

Parallel copy tool for Lustre (PCP) is similar to cp -r ; simply give it a source directory and destination and pcp will recursively copy the source directory to the destination in parallel.

Robinhood Policy Engine

URL: http://sourceforge.net/apps/trac/robinhood

Robinhood Policy Engine is a multi-purpose tool for managing content of large filesystems: audit FS content, accounting, remove old unused files according to admin-defined policies, customizable alerts based on file properties, backup data to external storage.
It has advanced capabilities for Lustre filesystems: leverage OST usage, list or purge files per OST, policy criteria based on pools and OST index, processing of MDT changelogs (Lustre v2).

Shine

URL: http://lustre-shine.sourceforge.net/

Description: Shine is a command line tool designed to setup and manage the Lustre file system on a cluster. It aims at centralize and simplify Lustre filesystem management task with:

  • one human-readable configuration file to describe your filesystem
  • start, stop, format (and more) your filesystem from a unique machine, thanks to a simple command line interface.

Author: CEA/Bull

tacc-stat

URL: https://github.com/jhammond/tacc_stats

https://github.com/jhammond/tacc_stats

Description: Probe Lustre interfaces and collect statistics.

Author: John Hammond (TACC)

xltop

URL: https://github.com/jhammond/xltop

Description: xltop is a continuous Lustre monitor with batch scheduler intergation.

Client screenshot:

    FILESYSTEM      MDS/T  LOAD1  LOAD5 LOAD15  TASKS   OSS/T  LOAD1  LOAD5 LOAD15  TASKS  NIDS
    ranger-work       1/1   1.52   3.48   4.41    609   14/84   2.74   2.08   2.09   1347  4212
    ranger-scratch    1/1   0.13   0.20   0.54    584  50/300   2.52   1.94   1.52   1348  4213
    ranger-share      1/1   0.93   1.20   1.72    544    6/36   3.55   1.37   0.90   1203  3960
    JOB      FS                  WR_MB/S     RD_MB/S      REQS/S  OWNER     NAME        HOSTS
    2526717  ranger-scratch      321.557       5.994    3556.133  tg803155  NST3.28-r0     20
    login4   ranger-scratch       38.489      55.054     469.943  NONE      NONE            1
    2530927  ranger-scratch       16.526       0.000      39.942  dkcira    Parametric      1
    2529449  ranger-work          11.754       0.000      24.088  bealing   PE-OH           4
    2530975  ranger-work          11.108       0.007      23.620  vishnam2  batch          16

Author: John Hammond (TACC)

  • No labels