Page History

Motivation

The current implementation of the Lustre POSIX copytool (lhsmtool_posix) is intended for demonstration purposes and not production use. Several items appear to be included that are interesting features but not fully realized for a production use:

• Creating a shadow namespace. The copytool saves the name of each archived files in a “shadow tree,” but there is no mechanism to keep them in sync with changes made in the filesystem. In the case of hard links, only one name is saved.

• File striping data is copied and stored with the object in the archive, and is used when file is restored. However, there is no mechanism to allow the user to change the striping data before the file is restored.

• The copytool uses FIDs to identify objects in the archive. FIDs are not global unique identifiers and, like inode numbers, are not intended too be used to identify files outside of the filesystem.

• The HSM interface and data movement functionality is tightly coupled in the code, and provides no abstraction for different data movers.

• The copytool produces overly verbose logging, and does not capture performance metrics.

New Design for HSM 

This proposal aims address all of the above issues of the current copytool with a new design and implementation. 

...

This design divides a "copytool" into two separate components, a low level Agent and backend-specific data movers. In this approach, the data movers do not interact directly with Lustre. Each mover registers to the locally running agent, and the agent delivers actions for the mover to perform. The agent manages the incoming requests from the HSM coordinator, and forwards the state updates back to the coordinator. The agent also stores the keys provided by the mover for each archived file, and the provides this key in further actions on this file.

...