Page History
High Performance Parallel storage for Virtualized Cloud-Based Infrastructure
What is
the IntelCloud Edition for Lustre
*?
IntelCloud Edition for Lustre
* or ICEL(CE) is a
scalable, shared filesystem for HPC applications in the cloud. HPDD provides early access Lustre* software on Amazon Web Services (AWS). Thesoftware-defined storage cluster that runs on scalable cloud infrastructures. Cloud Edition provides a high performance parallel filesystem using virtualized resources. The full package includes CentOS, Lustre,
gangliaGanglia, and Lustre Monitoring Tool (LMT).
This is delivered in the form of an Amazon Machine Image (AMI).
Using Intel Cloud Edition Lustre* on AWS
The Intel Lustre AMI is designed to be used with a CloudFormation template that defines all the resources needed by the Lustre filesystem. The templates provided here can be further customized by setting parameters in the CloudFormation console when creating a new stack. There are numerous parameters to this template, but to get started you only need to set the KeyName to your ssh-key, and set the LustreAZ to be specific Availability Zone in the region you are using. The remaining default values to get a small Lustre filesystem and two test clients. The filesystem is not persistent, and is deleted when the stack is deleted.
EBS Template
The CloudFormation template will create a Lustre cluster and also a small cluster of clients that will automatically mount the newly created Lustre filesystem. This cluster uses EBS storage for Lustre, though the default volume sizes are initially small. The filesystem created with this template is not persistent, and is deleted when the stack is deleted.
This template is using spot pricing for all the instances except for the MGS, which will use a single, on-demand m1.medium instance. There is a mapping table for spot bid prices in the template, and are set to the on-demand price for the corresponding instance type.
The CloudFormation stack will completely successfully once the MGS and autoscaler groups for the remaining nodes have been created, and it does not wait for the actual Lustre instances to start.
How to Create an AWS CloudFormation Stack using the Template Below
[Template Link] (Right-Click > Save Link As)
Template Parameters
Parameter | Default | Description |
KeyName |
| Name of and existing EC2 KeyPair to enable SSH access to the instance |
FsName | scratch | Name of the lustre filesystem. |
LustreAZ | Comma delimted list of availability zones to launch cluster in. Should all be in a single region. |
What is the Intel Cloud Edition for Lustre*?
Intel Cloud Edition for Lustre* or ICEL is a scalable, shared filesystem for HPC applications in the cloud. HPDD provides early access Lustre* software on Amazon Web Services (AWS). The full package includes CentOS, Lustre, ganglia, and Lustre Monitoring Tool (LMT). This is delivered in the form of an Amazon Machine Image (AMI).
Using Intel Cloud Edition Lustre* on AWS
The Intel Lustre AMI is designed to be used with a CloudFormation template that defines all the resources needed by the Lustre filesystem. The templates provided here can be further customized by setting parameters in the CloudFormation console when creating a new stack. There are numerous parameters to this template, but to get started you only need to set the KeyName to your ssh-key, and set the LustreAZ to be specific Availability Zone in the region you are using. The remaining default values to get a small Lustre filesystem and two test clients. The filesystem is not persistent, and is deleted when the stack is deleted.
EBS Template
The CloudFormation template will create a Lustre cluster and also a small cluster of clients that will automatically mount the newly created Lustre filesystem. This cluster uses EBS storage for Lustre, though the default volume sizes are initially small. The filesystem created with this template is not persistent, and is deleted when the stack is deleted.
This template is using spot pricing for all the instances except for the MGS, which will use a single, on-demand m1.medium instance. There is a mapping table for spot bid prices in the template, and are set to the on-demand price for the corresponding instance type.
The CloudFormation stack will completely successfully once the MGS and autoscaler groups for the remaining nodes have been created, and it does not wait for the actual Lustre instances to start.
How to Create an AWS CloudFormation Stack using the Template Below
[Template Link] (Right-Click > Save Link As)
Template Parameters
Parameter
Default
Description
KeyName
Name of and existing EC2 KeyPair to enable SSH access to the instance
FsName
scratch
Name of the lustre filesystem.
LustreAZ
Comma delimted list of availability zones to launch cluster in. Should all be in a single region.
Cloud Edition is intended to be used as the working filesystem for a HPC or other IO intensive workloads. It is not intended to be used as long term storage or as an alternative to cloud storage options such as S3. We recommend that S3 or "cold-storage" be used for long term data storage, and Cloud Edition be used whenever a high-performance shared filesystem is required.
Cloud Edition on Amazon Web Services
Amazon Web Services is a collection of remote computing services that make up Amazon's cloud computing infrastructure. Cloud Edition uses Amazon Machine Images (AMI) and Elastic Compute Storage (EC2) to provide a parallel and highly scalable storage cluster on AWS.
Cloud Edition on Microsoft Azure
Azure is Microsoft’s cloud computing platform, a growing collection of integrated services—analytics, computing, database, mobile, networking, storage, and web—for moving faster, achieving more, and saving money.
Cloud Edition on Google Cloud Platform
Google Cloud Platform is a collection of computing resources consist of virtual machines (VMs), storage, databases, networking, Infrastructure as Service (SaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).
When combined with HPC and Technical computing applications running on AWS, Cloud Edition can improve storage performance and increase scalability by eliminating storage bottlenecks.
[Click Here for More Information]
Cloud Edition combines with HPC and Technical computing applications running on Azure to improve storage performance and increase scalability by eliminating storage bottlenecks.
[Click Here for More Information]
Cloud Edition on Google Cloud Platform combines with HPC and Technical computing applications to improve user experience, storage performance and increase scalability by eliminating storage bottlenecks.
Support Details
Cloud Edition for Lustre software is supported by the Lustre experts at Whamcloud. Product support includes the latest software updates, patches, and fixes to ensure a stable, flexible, and robust storage environment that leverages the benefits of cloud-based infrastructure.
To create a new support ticket, please setup an account in our JIRA ticketing system and file your ticket in the AWSP project: https://jira.whamcloud.com/secure/Dashboard.jspa and our support team will resolve your issue promptly. Please provide your Amazon ID and a description of the issue when filing a ticket.
For other questions please contact us at info@whamcloud.com
Overview
Content Tools