HSM Agent Design

HSM Agent

New Design for HSM Copytools

This design divides a "copytool" into two separate components, a low level Agent and backend-specific data movers. In this approach, the data movers do not interact directly with Lustre. Each mover registers to the locally running agent, and the agent delivers actions for the mover to perform. The agent manages the incoming requests from the HSM coordinator, and forwards the state updates back to the coordinator. The agent also stores the keys provided by the mover for each archived file, and the provides this key in further actions on this file.

The movers are standalone processes, and communicate with the agent using gRPC, an RPC protocol built using Google's Protocol Buffers. When the agent starts, it starts the configured movers automatically.

RPC Interface

service DataMover {
    rpc Register(Endpoint) returns (Handle);
    rpc GetActions(Handle) returns (stream ActionItem);
    rpc StatusStream(stream ActionStatus) returns (Empty);
}

Register(Endpoint) returns (Handle)

message Endpoint {
    string fs_url = 2;
    uint32 archive = 1;
}

The mover register a specific archive id with the agent. The fs_url is the identifier of the filesystem (such as mntent.fsname). This is used to confirm the agent and mover are referring to the same filesystem because all paths passed between the agent and movers are relative paths from the root of the filesystem.

Register returns a handle for a virtual connection with the agent. This handle is used with all the communication with the agent related to with this archive id.

GetActions(Handle) returns (stream ActionItem)

message ActionItem {
    uint64 id = 1; // Unique indentifier for this action, must be used in status messages
    Command op = 2; 
    string primary_path = 3; // Path to primary file (for metadata or reading)
    string write_path = 4; // Path for writing data (for restore)
    uint64 offset = 5; // Start IO at offset
    uint64 length = 6; // Number of bytes to copy
    bytes file_id = 7; // Archive ID of file (provided with Restore command)
    bytes data = 8; // Arbitrary data passed to action. Data Mover specific.
}

GetActions() returns a stream that will provide ActionItems. Each action include a command and parameters to perform on the file referenced in primary_path. If the action requires writing to the Lustre filesystem (i.e. Restore), then the data must be written to write_path. If a file_id is associated with this file, then it will be included in the ActionItem.

The op can be one of these Commands

RESTORE

A Restore command indicates the mover must copy length data starting at offset from the backend to the write_path. The file_id provided by the mover after the Archive is included.

REMOVE

A Remove command indicates the file in primary_path is no longer managed HSM and the copied data should be removed from the backend.

StatusStream(stream ActionStatus) returns (Empty)

message ActionStatus {
    uint64 id = 1; // Unique identifier for action
    bool  completed = 2; // True if this is last update
    int32  error = 3; // Non-zero indicates an error
    uint64 offset = 4; 
    uint64  length = 5;
    Handle handle = 6;
    bytes file_id = 7; // Included with completion of Archive
    int32 flags = 8; // Additial flags (used for errors only?)
}

StatusStream() creates a stream that is used to send ActionStatus messages. The messages are used to send completion notificationand (for long running actions) progress updates.

When completed is 0, the message is a progress update for the message id, and the length value will include the amount of data that has been copied since the previous update. This is useful for long running transactions to provide update for monitoring systems, and also prevents the action from being timed out and restarted. It is important for movers to send these messages periodically even when no data is being copied (such waiting for a tape to be loaded) to ensure the action does not time out.

When completed is non-0, the message indicates that message referenced by ID has completed. If error is non-zero, then the action has failed with given error code. If error is 0, then the action has succeed, and the offset and length range of data has been copied by the mover. If file_id is included, then the agent will store this and include this with future actions referring to this file.

POSIX Mover

The POSIX mover allows another filesystem to be used a second storage tier for Lustre. The mover copies data between Lustre and another local filesystem. The POSIX mover configuration includes the archive ID and path for each archive backend managed by the mover.

Each time a file is submitted to the mover for archiving, the mover generates a new UUID which is used to identify the data object in the archive directory. The path to the object is <archiveDir>/objects/xx/yy/UUID, where:

xx = UUID[0:2] 
yy = UUID [2:4]

Once the archive is complete, the mover returns the UUID to the agent. The UUID is used during restore operation to locate the data object in the archive directory.

S3 Mover

The S3 mover allows an S3-style object storage system to be used a second tier for Lustre. The mover copies data between Lustre and an S3 bucket. The configuration includes the archive ID and bucket to be used to store data.  A UUID is generated for each file, and the object is stored in the bucket with similar path as the POSIX mover. Once the archive is complete, however, a URL for the object is returned to the Agent:

s3//<bucketName>/<prefix>/objects/xx/yy/UUID

The S3 mover can also be used to import existing data set from S3 into Lustre. To do this, the “lhsm import” command is used to replicate the path names from S3 into the Lustre filesystem, and the import also adds the s3 URL to each file so the mover can retrieve the data.

Page tree