Overview

In order to access a Lustre filesystem, it must be mounted by a machine running the Lustre client software. Lustre client software comprises a set of kernel modules which must be compatible with the running kernel, as well as userspace tools for interacting with the filesystem.

Starting with version 1.2 of Cloud Edition, a Lustre client setup tool is available to assist with software installation, optional IPSec configuration, and mounting the filesystem. This version of the product also includes DKMS-enabled Lustre client packages.

Please note that DKMS requires a working build environment. This is because the Lustre client kernel modules must be built against the running kernel. If this requirement is not suitable for a particular site's environment, then non-DKMS client packages for specific kernels may be obtained from Whamcloud.

IMPORTANT NOTE: If encryption support was enabled when the filesystem was created, then this client setup tool MUST be used to configure the clients. Manual configuration of IPSec is not supported.

Supported Clients

At this time, we recommend el6 or el7 for the client instance distribution. The client setup tool can configure Amazon Linux clients, but this distribution has not received any significant testing to verify Lustre functionality.

Client Deployment Scenarios

Fully Automated Client Deployment

In this scenario, each client instance runs the setup tool to install packages, build appropriate kernel modules, and automatically mount the filesystem. It is intended to be used for quick setup and testing, but may not be the best choice for long-term deployments.

Requirements

• A Cloud Edition Lustre filesystem must be running and accessible by the client instances
• Each client instance's distribution must be in the table of supported distributions
• Each client instance must have access to online software updates via yum, in order to install required packages (gcc, kernel-devel, etc.)
• Each client instance must have been launched with an IAM Role which includes the 

  AmazonDynamoDBReadOnlyAccess policy, or some equivalent level of read-only access to the FilesystemTable DynamoDB table created by the filesystem stack

Procedure

1. On each Lustre client instance, run the following command after sudo-ing to the root user: curl http://<mgs-ip>/install-client | bash

After some time, the output should indicate that the Lustre client software has been installed and configured, and the filesystem has been mounted. The filesystem entry has been added to /etc/fstab, and therefore it should automatically remount in the event of a reboot. If the kernel is updated, the Lustre client modules should be automatically rebuilt for the new kernel, if necessary.

Fully Automated Client Deployment With Multiple Filesystems

This scenario is similar to the first, but adds an extra step in order to deal with more than one CE Filesystem in the AWS account. Without additional input, the setup tool has no way of knowing which filesystem should be used to configure the client.

Requirements

• A Cloud Edition Lustre filesystem must be running and accessible by the client instances
• Each client instance's distribution must be in the table of supported distributions
• Each client instance must have access to online software updates via yum, in order to install required packages (gcc, kernel-devel, etc.)
• Each client instance must have been launched with an IAM Role which includes the 

  AmazonDynamoDBReadOnlyAccess policy, or some equivalent level of read-only access to the FilesystemTable DynamoDB table created by the filesystem stack

Procedure

The following commands will need to be run as root on each Lustre client instance:

1. curl http://<mgs-ip>/client-setup-tool -o /usr/sbin/ce-client && chmod +x /usr/sbin/ce-client
2. ce-client auto setup --stack <cloudformation stack name>

The process should now proceed as detailed in the first scenario.

Automated Client Deployment With Configuration File

This scenario is similar to the first, but does not require read-only access to DynamoDB. It is useful for provisioning existing instances in a quick setup and testing environment, but may not be the best choice for long-term deployments.

Requirements

• A Cloud Edition Lustre filesystem must be running and accessible by the client instances
• Each client instance's distribution must be in the table of supported distributions
• Each client instance must have access to online software updates via yum, in order to install required packages (gcc, kernel-devel, etc.)

Procedure

First, generate a ce-client configuration file on a Lustre server instance, and then make it available to the client instances:

1. On a Lustre server instance (e.g. the MGS), run the following command as root: ce-client config generate -o <filename>
2. Copy the generated configuration file to each client node (or push it to an S3 bucket)

IMPORTANT NOTE: When filesystem encryption is enabled, the client configuration file contains security-sensitive information (the IPsec pre-shared-key). This file should be protected from access by unauthorized users. Secure copy (e.g. with scp) and restrictive file permissions (e.g. 0600, owned by root) are recommended.

The following commands will need to be run as root on each Lustre client instance:

1. curl http://<mgs-ip>/client-setup-tool -o /usr/sbin/ce-client && chmod +x /usr/sbin/ce-client
2. ce-client auto setup --config <filename> (this file may be hosted locally to the client instance, in s3, or via http)

After some time, the output should indicate that the Lustre client software has been installed and configured, and the filesystem has been mounted. The filesystem entry has been added to /etc/fstab, and therefore it should automatically remount in the event of a reboot. If the kernel is updated, the Lustre client modules should be automatically rebuilt for the new kernel, if necessary.

Custom AMI Deployment

In this scenario, a single client instance (called "prime") is used to run through the initial setup process to build and install the Lustre client software. Afterward, a new AMI is created from the instance's filesystem which can then be used for all clients in order to avoid the build and install process on each client.

Requirements, Initial Setup

• A Cloud Edition Lustre filesystem must be running and accessible by the prime instance
• The prime instance's distribution must be in the table of supported distributions
• The prime instance must have access to online software updates via yum, in order to install required packages (gcc, kernel-devel, etc.)

Procedure, Initial Setup

The following commands should be run as the root user on the prime instance:

1. curl http://<mgs-ip>/client-setup-tool -o /usr/sbin/ce-client && chmod +x /usr/sbin/ce-client
2. ce-client package add-client-source --url http://<mgs-ip>/client-packages

3. ce-client package install ipsec (optional, only needed if the client will be used to access IPSec-protected filesystems)
4. ce-client package install lustre

Upon successful completion of the final step, use the AWS EC2 console or API to create a new AMI from the prime instance.

Requirements, Custom Clients

• A Cloud Edition Lustre filesystem must be running and accessible by the client instances
• Each client instance must have been launched with an IAM Role which includes the 

  AmazonDynamoDBReadOnlyAccess policy, or some equivalent level of read-only access to the FilesystemTable DynamoDB table created by the filesystem stack, OR

• Alternatively, each client instance must have access to a configuration file created by running the following command on a Lustre server instance (e.g. the MGS): ce-client config generate -o <filename>

IMPORTANT NOTE: When filesystem encryption is enabled, the client configuration file contains security-sensitive information (the IPsec pre-shared-key). This file should be protected from access by unauthorized users. Secure copy (e.g. with scp) and restrictive file permissions (e.g. 0600, owned by root) are recommended.


Procedure, Custom Clients

If using a generated configuration file, then the procedure is as follows:

1. ce-client auto setup --skip-packages --config <filename> (this file may be hosted locally to the client instance, in s3, or via http)

If the clients have access to the filesystem's DynamoDB table as described in the requirements, then the procedure is as follows:

1. ce-client auto setup --skip-packages

Client Setup Tool Modes

This section documents the various subcommands available via the client setup tool interface. In most cases, automatic mode is preferred, but in some scenarios it may be necessary to run through various parts of the setup with more control. As is the case with most command-line utilities, the most up-to-date information will always be available via the ce-client help command.

IMPORTANT NOTE: If using the client setup tool in a non-automated capacity, it is extremely important to understand that when filesystem encryption is enabled, IPSec MUST be configured prior to any attempt at mounting the Lustre client. The Lustre client will not be able to communicate with any of the Lustre servers if the correct procedure is not followed. For this reason, the automated setup mode is preferred and recommended.

auto

Preferred mode of operation. In this mode, the client setup tool will automatically configure package sources, install packages, configure IPSec (if encryption is enabled on the filesystem), and mount a Lustre client. In order for the client setup tool to determine its configuration, the client instance must have role-based read access to DynamoDB, or a configuration file must be generated on a Lustre server instance and made available to the client instances (e.g. copied to a local file, hosted on S3, etc.).

Examples

• # ce-client auto setup
• # ce-client auto setup --config /path/to/config.json
• # ce-client auto setup --config s3://<bucket>/config.json


• # ce-client auto teardown
• # ce-client auto teardown --config s3://<bucket>/config.json

config

Used to generate a client configuration file. Must be run on an instance which has read access to DynamoDB, e.g. the MGS instance. The resulting configuration file may be hosted in an S3 bucket, via a HTTP server, or copied to each client instance.

IMPORTANT NOTE: When filesystem encryption is enabled, the client configuration file contains security-sensitive information (the IPsec pre-shared-key). This file should be protected from access by unauthorized users. Secure copy (e.g. with scp) and restrictive file permissions (e.g. 0600, owned by root) are recommended.

Example

• # ce-client config generate -o /tmp/client-config.json

package

Controls installation and uninstallation of packages required by ce-client. The list of packages managed by this tool is preconfigured, and therefore this tool is not meant to act as a general-purpose package management tool.

Examples

• # ce-client package add-client-source --url http://<mgs-ip>/client-packages
• # ce-client package install lustre
• # ce-client package install ipsec


• # ce-client package remove lustre
• # ce-client package remove ipsec
• # ce-client package remove-client-source

ipsec

Controls configuration of IPSec tunnels between the client and Lustre servers. Requires a valid client configuration as generated on a Lustre server (e.g. the MGS). Requires that IPSec packages have already been installed (e.g. via the ce-client package install ipsec command).

IMPORTANT NOTE: Client IPSec configuration WILL NOT work unless filesystem encryption was enabled at creation. Post-creation configuration of IPSec is not supported.

Examples

• # ce-client ipsec setup --key <ipsec-pre-shared-key> --peer hostname:ipaddr [--peer hostname:ipaddr]
• # ce-client ipsec teardown

lustre

Controls configuration of a Lustre client. Requires a valid client configuration as generated on a Lustre server (e.g. the MGS). Requires that lustre packages have already been installed (e.g. via the ce-client package install lustre command).

The Lustre filesystem will be added as an entry to /etc/fstab so that it is remounted on reboot.

IMPORTANT NOTE: If filesystem encryption has been enabled, then IPSec MUST be configured prior to Lustre client configuration.

Examples

• # ce-client lustre mount --create --mountpoint /mnt/<fsname> --device <mgsip>@tcp:/<fsname>

• # ce-client lustre unmount --all
• # ce-client lustre unmount --mountpoint /mnt/<fsname>