Launching Your First AWS Cluster

The following is an example to follow to get your first Lustre AWS cluster built.

You will first need access to the hpdd account. Follow the instructions to setup access to the account

You will need an SSH key for ssh access to your cluster. So create a public and private key and place it on your system.

For windows user, you can use puttygen to generate the key.

Once that is done, login and you will see this screen. To make it easy, hit edit and move some things to the access bar

Once you login you will see this screen. To make it easy, hit edit and drag and drop EC2, VPC and Cloud Formation to the access bar

Now the access bar will look like the following. The first thing to do will be to create a key pair for accessing our cluster once it is created.

So click on "EC2"

On the EC2 dashboard, look for "Key Pairs" in the left panel under "Network and Security"

Then Chose "Import Keys" and upload an OpenSSH compatible public key. The template we will run later will put this public key in all instances it creates.

This next screenshot also shows a puttygen window for Windows users. Puttygen does not save an OpenSSH compatible public key so you cannot upload it.

However if you load your private key into puttygen you can cut an OpenSSH compatible key from the box shown here and paste it into the import key box.

Now we see our key which we named "aws-demo" listed as a key pair. We will use this later when we create our cluster.

The next thing we need to setup is a VPC (Virtual Private Cloud). Start my choosing "VPC" from the access bar.

Click on "start VPC Wizard" to create our VPC.

Choose the first option on the right "VPC with a single Public Subnet" and click "Select"

Fill in the address range you would like. You do not need to use these values.

For this example I used "10.81.0.0/16" and named the VPC "aws-demo"

I also made my public subnet 10.81.12.0/24 and named this subnet "awsdemo-subnet".

When all the fields are filled in, click on "Create VPC".

Now browse to "Intel Cloud Edition for Lustre* - Self Support" and on this page are all the stacks that are available. For this example we are using the California AWS domain.

Just take the defaults on the next screen and click "Next"

On the next page give your stack a unique name. aws-demo-lustre-ha is used for this example.

Also gie your filesystem a name. "demofs" is used for this example

Choose the key pair we made from the drop down list.

Choose the VPC we created.

Fill out the CIDR and choose the subnet we created when we created our VPC.

go to the next screen and click on "Next"

check the box.. yes we know AWS is not free. And then click on "Create"

When you go to this page, you may need to refresh using the refresh symbol the upper right of the page.

But after a while you will see that your Lustre cluster is up and running. Look in the "Outputs" tab for the IP address to use to SSH to the NAT instance of your cluster.

You need to ssh using username "ec2-user".

You will also see a link that you can click that will open your browser to the ganglia output for your cluster.

You can return to this page by clicking on "Cloud Formation" that we put on the access bar.

To access the servers, you will need to ssh to the NAT node with a pagent and forwarding enabled and ssh from here to the servers.

For windows, start "pagent" and add you private key. Then when you use putty go to the ssh auth page and check the box to "allow agent forwarding"

Once you are SSHed to the NAT node, you can ssh to the servers, again as "ec2-user". sudo will work from this user.

To get the IP addresses of the servers, click on "EC2"  from the access bar and click on "instances" in the left hand pane and then select the server.

Here the MGS is shown. In the bottom pane the IP address of the MGS is shown at the top.

You can also access your clients, mount the filesystem and have some Lustre fun on AWS.