Table of Contents

maxLevel	2

How to deploy an EXAScaler Cloud Application

Anchor

	Install
	Install

EXAScaler Cloud application can be launched on Microsofts Azure Portal by navigating to the link below and and clicking the Create button.

Step by step deployment video

Widget Connector

url	https://www.youtube.com/watch?v=hGBujIKe3H8

Step by step deployment guide

The steps below will show how to create a EXAScaler Cloud Cluster on Microsoft Azure platform.

...

And you can customize the dashboard according to your requirements.

How to automate deployment with Terraform

Automated deployment is possible with EXAScaler Cloud Terraform Scripts.

The steps below will show how to create a EXAScaler Cloud environment on Microsoft Azure using Terraform.

Supported products

Product	Version	Base OS	Stock Keeping Unit (`SKU`)
EXAScaler Cloud	5.2.6	Red Hat Enterprise Linux 7.9	`exascaler_cloud_5_2_redhat`
EXAScaler Cloud	5.2.6	CentOS Linux 7.9	`exascaler_cloud_5_2_centos`
EXAScaler Cloud	6.0.1	Red Hat Enterprise Linux 7.9	`exascaler_cloud_6_0_redhat`
EXAScaler Cloud	6.0.1	CentOS Linux 7.9	`exascaler_cloud_6_0_centos`
EXAScaler Cloud	6.1.0	Red Hat Enterprise Linux 7.9	`exascaler_cloud_6_1_redhat`
EXAScaler Cloud	6.1.0	CentOS Linux 7.9	`exascaler_cloud_6_1_centos`
EXAScaler Cloud	6.2.0	Red Hat Enterprise Linux 8.7	`exascaler_cloud_6_2_redhat`
EXAScaler Cloud	6.2.0	Rocky Linux 8.7	`exascaler_cloud_6_2_rocky`
EXAScaler Cloud	6.3.2	Red Hat Enterprise Linux 8.10	`exascaler_cloud_6_3_redhat`
EXAScaler Cloud	6.3.2	Rocky Linux 8.10	`exascaler_cloud_6_3_rocky`

Client packages

EXAScaler Cloud deployment provides support for installing and configuring third-party clients. EXAScaler Cloud client software comprises a set of kernel modules which must be compatible with the running kernel, as well as userspace tools for interacting with the filesystem.

Vendor	Product	Version	Arch	Kernel Version for binary package	Kernel Version for DKMS package
Red Hat	RHEL	`7.6`	`x86_64`	`3.10.0-957.99.1.el7.x86_64`	`3.10.0`
Red Hat	RHEL	`7.7`	`x86_64`	`3.10.0-1062.77.1.el7.x86_64`	`3.10.0`
Red Hat	RHEL	`7.8`	`x86_64`	`3.10.0-1127.19.1.el7.x86_64`	`3.10.0`
Red Hat	RHEL	`7.9`	`x86_64`	`3.10.0-1160.119.1.el7.x86_64`	`3.10.0`
Red Hat	RHEL	`8.0`	`x86_64`	`4.18.0-80.31.1.el8_0.x86_64`	`4.18.0`
Red Hat	RHEL	`8.1`	`x86_64`	`4.18.0-147.94.1.el8_1.x86_64`	`4.18.0`
Red Hat	RHEL	`8.2`	`x86_64`	`4.18.0-193.141.1.el8_2.x86_64`	`4.18.0`
Red Hat	RHEL	`8.3`	`x86_64`	`4.18.0-240.22.1.el8_3.x86_64`	`4.18.0`
Red Hat	RHEL	`8.4`	`x86_64`	`4.18.0-305.148.1.el8_4.x86_64`	`4.18.0`
Red Hat	RHEL	`8.5`	`x86_64`	`4.18.0-348.23.1.el8_5.x86_64`	`4.18.0`
Red Hat	RHEL	`8.6`	`aarch64`	`4.18.0-372.105.1.el8_6.aarch64`	`4.18.0`
Red Hat	RHEL	`8.6`	`x86_64`	`4.18.0-372.134.1.el8_6.x86_64`	`4.18.0`
Red Hat	RHEL	`8.7`	`aarch64`	`4.18.0-425.19.2.el8_7.aarch64`	`4.18.0`
Red Hat	RHEL	`8.7`	`x86_64`	`4.18.0-425.19.2.el8_7.x86_64`	`4.18.0`
Red Hat	RHEL	`8.8`	`aarch64`	`4.18.0-477.86.1.el8_8.aarch64`	`4.18.0`
Red Hat	RHEL	`8.8`	`x86_64`	`4.18.0-477.86.1.el8_8.x86_64`	`4.18.0`
Red Hat	RHEL	`8.9`	`aarch64`	`4.18.0-513.24.1.el8_9.aarch64`	`4.18.0`
Red Hat	RHEL	`8.9`	`x86_64`	`4.18.0-513.24.1.el8_9.x86_64`	`4.18.0`
Red Hat	RHEL	`8.10`	`aarch64`	`4.18.0-553.40.1.el8_10.aarch64`	`4.18.0`
Red Hat	RHEL	`8.10`	`x86_64`	`4.18.0-553.40.1.el8_10.x86_64`	`4.18.0`
Red Hat	RHEL	`9.0`	`aarch64`	`5.14.0-70.101.1.el9_0.aarch64`	`5.14.0`
Red Hat	RHEL	`9.0`	`x86_64`	`5.14.0-70.122.1.el9_0.x86_64`	`5.14.0`
Red Hat	RHEL	`9.1`	`aarch64`	`5.14.0-162.23.1.el9_1.aarch64`	`5.14.0`
Red Hat	RHEL	`9.1`	`x86_64`	`5.14.0-162.23.1.el9_1.x86_64`	`5.14.0`
Red Hat	RHEL	`9.2`	`aarch64`	`5.14.0-284.99.1.el9_2.aarch64`	`5.14.0`
Red Hat	RHEL	`9.2`	`x86_64`	`5.14.0-284.99.1.el9_2.x86_64`	`5.14.0`
Red Hat	RHEL	`9.3`	`aarch64`	`5.14.0-362.24.1.el9_3.aarch64`	`5.14.0`
Red Hat	RHEL	`9.3`	`x86_64`	`5.14.0-362.24.1.el9_3.x86_64`	`5.14.0`
Red Hat	RHEL	`9.4`	`aarch64`	`5.14.0-427.50.1.el9_4.aarch64`	`5.14.0`
Red Hat	RHEL	`9.4`	`x86_64`	`5.14.0-427.50.1.el9_4.x86_64`	`5.14.0`
Red Hat	RHEL	`9.5`	`aarch64`	`5.14.0-503.26.1.el9_5.aarch64`	`5.14.0`
Red Hat	RHEL	`9.5`	`x86_64`	`5.14.0-503.26.1.el9_5.x86_64`	`5.14.0`
Canonical	Ubuntu	`16.04 LTS`	`amd64`	—	`4.4 - 4.15`
Canonical	Ubuntu	`18.04 LTS`	`amd64`	—	`4.15 - 5.4`
Canonical	Ubuntu	`20.04 LTS`	`amd64`	—	`5.4 - 5.15`
Canonical	Ubuntu	`20.04 LTS`	`arm64`	—	`5.4 - 5.15`
Canonical	Ubuntu	`22.04 LTS`	`amd64`	—	`5.15 - 6.2`
Canonical	Ubuntu	`22.04 LTS`	`arm64`	—	`5.15 - 6.2`
Canonical	Ubuntu	`24.04 LTS`	`amd64`	—	`6.8 - TBD`
Canonical	Ubuntu	`24.04 LTS`	`arm64`	—	`6.8 - TBD`

Notes:

Client packages for aarch64 and arm64 architectures are available only for EXAScaler Cloud 6.3
Client packages for Canonical Ubuntu 16.04 LTS are not available for EXAScaler Cloud 6.3

Prerequisites

You need a Microsoft account
Your system needs the Microsoft Azure CLI as well as Terraform

...

DDN EXAScaler Cloud in the Azure Marketplace have additional license and purchase terms that you must accept before you can deploy them programmatically. To deploy an environment from this image, you'll need to accept the image's terms the first time you use it, once per subscription.

Steps to authenticate via Microsoft account

Obtains access credentials for your user account via a web-based authorization flow. When this command completes successfully, it sets the active account in the current configuration to the account specified. Learn more about Azure authentication.

...

Please use the value of id property as an Azure subscription ID for Terraform based deployments.

Steps to accept the terms of use for DDN EXAScaler Cloud images

To deploy DDN EXAScaler Cloud, you need to accept the Azure Marketplace image terms so that the image can be used to create a deployment.

...

Learn more about the image terms.

Steps to configure Terraform

Download Terraform scripts and extract the tarball:

Code Block

language	bash
theme	Midnight

$ curl -sL https://github.com/DDNStorage/exascaler-cloud-terraform/archive/refs/tags/scripts/2.12.52.tar.gz | tar xz

Change Terraform variables according you requirements:

Code Block

language	bash
theme	Midnight

$ cd exascaler-cloud-terraform-scripts-2.12.52/az
$ vi terraform.tfvars

List of available variables

Common options

Variable	Type	Default	Description
`prefix`	`string`	`null`	EXAScaler Cloud custom deployment prefix. Set this option to add a custom prefix to all created objects.
`tags`	`map`	`{}`	EXAScaler Cloud custom deployment tags. Set this option to add a custom tags to all created objects. Learn more about Azure tags.
`fsname`	`string`	`exacloud`	EXAScaler filesystem name. Learn more about Lustre filesystem.
`subscription`	`string`	`00000000-0000-0000-0000-000000000000`	Subscription ID - please use ID of you active Azure subscription. Learn more Azure subscriptions.
`location`	`string`	`West US`	Azure region to manage resources. Learn more about Azure geography.

Availability options

Variable	Type	Default	Description
`availability.type`	`string`	`none`	Availability type: `none` - no infrastructure redundancy required, `set` - to create an availability set and automatically distribute resources across multiple fault domains, `zone` - to physically separate resources within an Azure region. Learn more about Azure availability options.
`availability.zone`	`integer`	`1`	Availability zone - unique physical locations within a Azure region. Use `1`, `2` or `3` to explicitly specify the availability zone. Learn more about Azure availability zones.

Resource group options

Variable	Type	Default	Description
`resource_group.new`	`bool`	`true`	Create a new resource group, or use an existing one: `true` or `false`.
`resource_group.name`	`string`	`existing-resource-group`	Existing resource group name, will be using if `new` is `false`.

Learn more about Azure resource groups.

Proximity placement group options

Variable	Type	Default	Description
`proximity_placement_group.new`	`bool`	`true`	Create a new proximity placement group, or use an existing one: `true` or `false`.
`proximity_placement_group.name`	`string`	`existing-proximity-placement-group`	Existing proximity placement group name, will be using if new is `false`.

Learn more about Azure proximity placement groups.

Network options

Variable	Type	Default	Description
`network.new`	`bool`	`true`	Create a new network, or use an existing one: `true` or `false`.
`network.name`	`string`	`existing-network`	Existing network name, will be using only if `new` option is `false`.
`network.address`	`string`	`10.0.0.0/8`	IP address in CIDR notation for the new virtual network.

Learn more about Azure virtual networks.

Subnet options

Variable	Type	Default	Description
`subnet.new`	`bool`	`true`	Create a new subnet, or use an existing one: `true` or `false`.
`subnet.name`	`string`	`existing-subnet`	Existing subnet name, will be using only if `new` option is `false`.
`network.address`	`string`	`10.0.0.0/24`	IP address in CIDR notation for the new subnet.

Learn more about Azure virtual networks and subnets.

Security options

Variable	Type	Default	Description
`security.username`	`string`	`stack`	User name for remote SSH access. Learn more about Azure SSH options.
`security.ssh_public_key`	`string`	`~/.ssh/id_rsa.pub`	Path to the local SSH public key. This file will be added to admin home directory as `.ssh/authorized_keys`. Learn more about Azure SSH options.
`security.enable_ssh`	`bool`	`true`	`true` or `false`: enable or disable remote SSH access.
`security.enable_http`	`bool`	`true`	`true` or `false`, enable or disable remote HTTP access.
`security.ssh_source_ranges`	`list(string)`	`[0.0.0.0/0]`	Source IP ranges for remote SSH access in CIDR notation. Learn more about Azure network security groups.
`security.http_source_ranges`	`list(string)`	`[0.0.0.0/0]`	Source IP ranges for remote HTTP access in CIDR notation. Learn more Azure network security groups.

Boot disk options

Variable	Type	Default	Description
`boot.disk_type`	`string`	`StandardSSD_LRS`	Specifies the type of managed disk to create: `Standard_LRS` `Premium_LRS` `StandardSSD_LRS`
`boot.disk_cache`	`string`	`ReadWrite`	Specifies the caching requirements for the target disk: `None` `ReadOnly` `ReadWrite`
`boot.auto_delete`	`bool`	`true`	Delete the boot disk automatically when deleting the virtual machine: `true`or `false`.
`boot.disk_size`	`integer`	`64`	Boot disk size in GB.

Learn more about Azure disks types.

Source image options

Variable	Type	Default	Description
`image.publisher`	`string`	`ddn-whamcloud-5345716`	Specifies the publisher of the image used to create the virtual machine.
`image.offer`	`string`	`exascaler_cloud`	Specifies the offer of the image used to create the virtual machine.
`image.sku`	`string`	`exascaler_cloud_6_13_centosrocky`	Specifies the `SKU` of the image used to create the virtual machine. EXAScaler Cloud 5.2 images: `exascaler_cloud_5_2_centos` `exascaler_cloud_5_2_redhat` EXAScaler Cloud 6. 0 1 images: `exascaler_cloud_6_` 0 `1_centos` `exascaler_cloud_6_1_redhat` EXAScaler Cloud 6.2 images: `exascaler_cloud_6` _0 `_2_rocky` `exascaler_cloud_6_2_redhat` EXAScaler Cloud 6. 1 3 images: `exascaler_cloud_6_` 1 `3_` centos `rocky` `exascaler_cloud_6_` 1 `3_redhat`
`image.version`	`string`	`latest`	Specifies the version of the image used to create the virtual machine.
`image.accept`	`bool`	`false`	Allows automatically accepting the legal terms for a Marketplace image.

Learn more about Azure disk images.

Storage account options

Variable	Type	Default	Description
`storage_account.kind`	`string`	`StorageV2`	Defines the kind of account. Valid options are: `BlobStorage` `BlockBlobStorage` `FileStorage` `Storage` `StorageV2`
`storage_account.tier`	`string`	`Standard`	Defines the tier to use for this storage account. Valid options are: `Standard` `Premium`
`storage_account.replication`	`string`	`LRS`	Defines the type of replication to use for this storage account. Valid options are: `LRS` `GRS` `RAGRS` `ZRS` `GZRS` `RAGZRS`

Learn more about Azure storage accounts.

Management server options

Variable	Type	Default	Description
`mgs.node_type`	`string`	`Standard_F4s`	Type of management server. Learn more about Azure performance considerations.
`mgs.node_count`	`integer`	`1`	Number of management servers: `1`.
`mgs.public_ip`	`bool`	`true`	Assign a public IP address: `true` or `false`. Learn more about Azure public IP addresses.
`mgs.accelerated_network`	`bool`	`false`	Enable accelerated networking. Learn more about Azure accelerated networking.

Management target options

Variable	Type	Default	Description
`mgt.disk_type`	`string`	`StandardSSD_LRS`	Specifies the type of managed disk to create the management target: `Standard_LRS` `StandardSSD_LRS` `Premium_LRS`
`mgt.disk_cache`	`string`	`None`	Specifies the caching requirements for the management target: `None` `ReadOnly` `ReadWrite`
`mgt.disk_size`	`integer`	`256`	Specifies the size of the management target in GB.
`mgt.disk_count`	`integer`	`1`	Specifies the number of management targets: `1-128`.
`mgt.disk_raid`	`bool`	`false`	Create striped management target: `true` or `false`.

Learn more about Azure disks types

Monitoring target options

Variable	Type	Default	Description
`mnt.disk_type`	`string`	`StandardSSD_LRS`	Specifies the type of managed disk to create the monitoring target: `Standard_LRS` `StandardSSD_LRS` `Premium_LRS`
`mnt.disk_cache`	`string`	`None`	Specifies the caching requirements for the monitoring target: `None` `ReadOnly` `ReadWrite`
`mnt.disk_size`	`integer`	`128`	Specifies the size of the monitoring target in GB.
`mnt.disk_count`	`integer`	`1`	Specifies the number of monitoring targets: `1-128`.
`mnt.disk_raid`	`bool`	`false`	Create striped monitoring target: `true` or `false`.

Learn more about Azure disks types

Metadata server options

Variable	Type	Default	Description
`mds.node_type`	`string`	`Standard_E8s_v3`	Type of metadata server. Learn more about Azure performance considerations.
`mds.node_count`	`integer`	`1`	Number of metadata servers: `1-32`.
`mds.public_ip`	`bool`	`false`	Assign a public IP address: `true` or `false`. Learn more about Azure public IP addresses.
`mds.accelerated_network`	`bool`	`false`	Enable accelerated networking. Learn more about Azure accelerated networking.

Metadata target options

Variable	Type	Default	Description
`mdt.disk_type`	`string`	`Premium_LRS`	Specifies the type of managed disk to create the metadata target: `Standard_LRS` `StandardSSD_LRS` `Premium_LRS`
`mdt.disk_cache`	`string`	`None`	Specifies the caching requirements for the metadata target: `None` `ReadOnly` `ReadWrite`
`mdt.disk_size`	`integer`	`512`	Specifies the size of the metadata target in GB.
`mdt.disk_count`	`integer`	`1`	Specifies the number of metadata targets: `1-128`.
`mdt.disk_raid`	`bool`	`false`	Create striped metadata target: `true` or `false`.

Learn more about Azure disks types

Object Storage server options

Variable	Type	Default	Description
`oss.node_type`	`string`	`Standard_Ds16_v3`	Type of object storage server. Learn more about Azure performance considerations.
`oss.node_count`	`integer`	`4`	Number of object storage servers: `1-2000`.
`oss.public_ip`	`bool`	`false`	Assign a public IP address: `true` or `false`. Learn more about Azure public IP addresses.
`oss.accelerated_network`	`bool`	`false`	Enable accelerated networking. Learn more about Azure accelerated networking.

Object Storage target options

Variable	Type	Default	Description
`ost.disk_type`	`string`	`Standard_LRS`	Specifies the type of managed disk to create the object storage target: `Standard_LRS` `StandardSSD_LRS` `Premium_LRS`
`ost.disk_cache`	`string`	`None`	Specifies the caching requirements for the object storage target: `None` `ReadOnly` `ReadWrite`
`ost.disk_size`	`integer`	`512`	Specifies the size of the object storage target in GB.
`ost.disk_count`	`integer`	`6`	Specifies the number of object storage targets: `1-128`.
`ost.disk_raid`	`bool`	`false`	Create striped object storage target: `true` or `false`.

Learn more about Azure disks types

Compute client options

Variable	Type	Default	Description
`cls.node_type`	`string`	`Standard_Ds16_v3`	Type of compute client. Learn more about Azure performance considerations.
`cls.node_count`	`integer`	`4`	Number of compute clients.
`cls.public_ip`	`bool`	`false`	Assign a public IP address: `true` or `false`. Learn more about Azure public IP addresses.
`cls.accelerated_network`	`bool`	`false`	Enable accelerated networking. Learn more about Azure accelerated networking.

Compute client target options

Variable	Type	Default	Description
`clt.disk_type`	`string`	`Standard_LRS`	Specifies the type of managed disk to create the compute target: `Standard_LRS` `StandardSSD_LRS` `Premium_LRS`
`clt.disk_cache`	`string`	`None`	Specifies the caching requirements for the compute target: `None` `ReadOnly` `ReadWrite`
`clt.disk_size`	`integer`	`32`	Specifies the size of the compute target in GB.
`clt.disk_count`	`integer`	`0`	Specifies the number of compute targets.

Learn more about Azure disks types

Deploy an EXAScaler Cloud environment

Initialize a working directory containing Terraform configuration files. This is the first command that should be run after writing a new Terraform configuration or cloning an existing one from version control. It is safe to run this command multiple times:

...

Code Block

language	bash
theme	Midnight

$ terraform apply
...
  Enter a value: yes
...
Apply complete! Resources: 103 added, 0 changed, 0 destroyed.

Outputs:

azure_dashboard = "https://portal.azure.com/#@00000000-0000-0000-0000-000000000000/dashboard/arm/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/exascaler-cloud-a108-resource-group/providers/Microsoft.Portal/dashboards/exascaler-cloud-a108-dashboard"

client_config = <<EOT
#!/bin/sh
# install new EXAScaler Cloud clients:
# all instances must be in the same location westus
# and connected to the network exascaler-cloud-a108-virtual-network
# and subnet exascaler-cloud-a108-subnet
# to set up EXAScaler Cloud filesystem on a new client instance,
# run the folowing commands on the client with root privileges:

cat >/etc/esc-client.conf<<EOF
{
  "Version": "2.0.0",
  "MountConfig": {
    "ClientDevice": "10.0.0.10@tcp:/exacloud",
    "Mountpoint": "/mnt/exacloud",
    "PackageSource": "http://10.0.0.10/client-packages"
  }
}
EOF

curl -fsSL http://10.0.0.10/client-setup-tool -o /usr/sbin/esc-client
chmod +x /usr/sbin/esc-client
esc-client auto setup --config /etc/esc-client.conf

EOT

http_console = "http://exascaler-cloud-a108-mgs0.westus.cloudapp.azure.com"

mount_command = "mount -t lustre 10.0.0.10@tcp:/exacloud /mnt/exacloud"

private_addresses = {
  "exascaler-cloud-a108-cls0" = "10.0.0.8"
  "exascaler-cloud-a108-cls1" = "10.0.0.7"
  "exascaler-cloud-a108-cls2" = "10.0.0.11"
  "exascaler-cloud-a108-cls3" = "10.0.0.12"
  "exascaler-cloud-a108-mds0" = "10.0.0.13"
  "exascaler-cloud-a108-mgs0" = "10.0.0.10"
  "exascaler-cloud-a108-oss0" = "10.0.0.9"
  "exascaler-cloud-a108-oss1" = "10.0.0.4"
  "exascaler-cloud-a108-oss2" = "10.0.0.5"
  "exascaler-cloud-a108-oss3" = "10.0.0.6"
}

ssh_console = {
  "exascaler-cloud-a108-mgs0" = "ssh -A stack@exascaler-cloud-a108-mgs0.westus.cloud
app.azure.com"
}

Access the EXAScaler Cloud environment

Now you can access the EXAScaler Cloud environment:

Code Block

language	bash
theme	Midnight

$ eval $(ssh-agent)
Agent pid 12313

$ ssh-add
Identity added: /Users/name/.ssh/id_rsa (/Users/name/.ssh/id_rsa)

$ ssh -A stack@exascaler-cloud-a108-mgs0.westus.cloudapp.azure.com

[stack@exascaler-cloud-a108-mgs0 ~]$ df -h -t lustre
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        249G  2.4M  247G   1% /mnt/targets/MGS

[stack@exascaler-cloud-a108-mgs0 ~]$ loci hosts
10.0.0.8	exascaler-cloud-a108-cls0
10.0.0.7	exascaler-cloud-a108-cls1
10.0.0.11	exascaler-cloud-a108-cls2
10.0.0.12	exascaler-cloud-a108-cls3
10.0.0.13	exascaler-cloud-a108-mds0
10.0.0.10	exascaler-cloud-a108-mgs0
10.0.0.9	exascaler-cloud-a108-oss0
10.0.0.4	exascaler-cloud-a108-oss1
10.0.0.5	exascaler-cloud-a108-oss2
10.0.0.6	exascaler-cloud-a108-oss3

[stack@exascaler-cloud-a108-mgs0 ~]$ ssh exascaler-cloud-a108-cls0

[stack@exascaler-cloud-a108-cls0 ~]$ lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
exacloud-MDT0000_UUID   315302464        6212   309927544   1% /mnt/exacloud[MDT:0]
exacloud-OST0000_UUID   529449792        1256   524063444   1% /mnt/exacloud[OST:0]
exacloud-OST0001_UUID   529449792        1272   524063428   1% /mnt/exacloud[OST:1]
exacloud-OST0002_UUID   529449792        1268   524063432   1% /mnt/exacloud[OST:2]
exacloud-OST0003_UUID   529449792        1264   524063436   1% /mnt/exacloud[OST:3]
exacloud-OST0004_UUID   529449792        1256   524063444   1% /mnt/exacloud[OST:4]
exacloud-OST0005_UUID   529449792        1264   524063436   1% /mnt/exacloud[OST:5]
exacloud-OST0006_UUID   529449792        1276   524063424   1% /mnt/exacloud[OST:6]
exacloud-OST0007_UUID   529449792        1268   524063432   1% /mnt/exacloud[OST:7]
exacloud-OST0008_UUID   529449792        1260   524063440   1% /mnt/exacloud[OST:8]
exacloud-OST0009_UUID   529449792        1272   524063428   1% /mnt/exacloud[OST:9]
exacloud-OST000a_UUID   529449792        1268   524063432   1% /mnt/exacloud[OST:10]
exacloud-OST000b_UUID   529449792        1272   524063428   1% /mnt/exacloud[OST:11]
exacloud-OST000c_UUID   529449792        1256   524063444   1% /mnt/exacloud[OST:12]
exacloud-OST000d_UUID   529449792        1264   524063436   1% /mnt/exacloud[OST:13]
exacloud-OST000e_UUID   529449792        1268   524063432   1% /mnt/exacloud[OST:14]
exacloud-OST000f_UUID   529449792        1260   524063440   1% /mnt/exacloud[OST:15]
exacloud-OST0010_UUID   529449792        1272   524063428   1% /mnt/exacloud[OST:16]
exacloud-OST0011_UUID   529449792        1260   524063440   1% /mnt/exacloud[OST:17]
exacloud-OST0012_UUID   529449792        1256   524063444   1% /mnt/exacloud[OST:18]
exacloud-OST0013_UUID   529449792        1264   524063436   1% /mnt/exacloud[OST:19]
exacloud-OST0014_UUID   529449792        1256   524063444   1% /mnt/exacloud[OST:20]
exacloud-OST0015_UUID   529449792        1260   524063440   1% /mnt/exacloud[OST:21]
exacloud-OST0016_UUID   529449792        1272   524063428   1% /mnt/exacloud[OST:22]
exacloud-OST0017_UUID   529449792        1264   524063436   1% /mnt/exacloud[OST:23]

filesystem_summary:  12706795008       30348 12577522452   1% /mnt/exacloud

Add storage capacity in an existing EXAScaler Cloud environment

The storage capacity can be added by increasing the number of storage servers. To add storage capacity in an existing EXAScaler Cloud environment, just modify the terraform.tfvars file and increase the number of storage servers (the value of the oss.node_countvariable) as required:

...

Code Block

language	bash
theme	Midnight

capacity = oss.node_count * ost.disk_count * ost.disk_size

Upgrade an existing EXAScaler Cloud environment

A software upgrade for an existing EXAScaler Cloud environment is possible by recreating the running VM instances using a new version of the OS image. And it requires some manual steps.

...

Code Block

language	bash
theme	Midnight

$ cd /path/to
$ curl -sL https://github.com/DDNStorage/exascaler-cloud-terraform/archive/refs/tags/scripts/2.12.52.tar.gz | tar xz
$ cd exascaler-cloud-terraform-scripts-2.12.52/az

Copy the terraform.tfstate file from the existing Terraform directory:

...

Unmount the existing EXAScaler Cloud filesystem using the provided esc exascaler-cloud-ctl script. This step is required to ensure data consistency during the upgrade:

Code Block

language	bash
theme	Midnight

$ ./scripts/escexascaler-cloud-ctl

Usage:

List resource groups : ./scripts/escexascaler-cloud-ctl list
List deployments     : ./scripts/escexascaler-cloud-ctl <resource_group> list
List instances       : ./scripts/escexascaler-cloud-ctl <resource_group> <deployment> list
Stop instances       : ./scripts/escexascaler-cloud-ctl <resource_group> <deployment> stop
Start instances      : ./scripts/escexascaler-cloud-ctl <resource_group> <deployment> start
Unmount filesystem   : ./scripts/escexascaler-cloud-ctl <resource_group> <deployment> umount

$ ./scripts/escexascaler-cloud-ctl list
Name                                 Location    Status
-----------------------------------  ----------  ---------
exascaler-cloud-f7cd-resource-group  eastus      Succeeded
NetworkWatcherRG                     westus      Succeeded

$ ./scripts/escexascaler-cloud-ctl exascaler-cloud-f7cd-resource-group list
Name                            Created                    Status
------------------------------  -------------------------  ---------
exascaler-cloud-f7cd            2021-08-21T01:19:36+00:00  Succeeded

$ ./scripts/escexascaler-cloud-ctl exascaler-cloud-f7cd-resource-group exascaler-cloud-f7cd umount
Umount compute client exascaler-cloud-f7cd-cls0
Umount compute client exascaler-cloud-f7cd-cls1
Umount storage server exascaler-cloud-f7cd-oss0
Umount storage server exascaler-cloud-f7cd-oss1
Umount storage server exascaler-cloud-f7cd-oss2
Umount storage server exascaler-cloud-f7cd-oss3
Umount metadata server exascaler-cloud-f7cd-mds0
Umount management server exascaler-cloud-f7cd-mgs0

...

Code Block

language	bash
theme	Midnight

$ terraform apply
...
  Enter a value: yes
...

Outputs:

Apply complete! Resources: 18 added, 8 changed, 16 destroyed.

Outputs:

azure_dashboard = "https://portal.azure.com/#@753b6e26-6fd3-43e6-8248-3f1735d59bb4/dashboard/arm/subscriptions/9978cd1b-936a-4296-8061-67c9d963dd40/resourceGroups/exascaler-cloud-f7cd-resource-group/providers/Microsoft.Portal/dashboards/exascaler-cloud-f7cd-dashboard"

http_console = "http://exascaler-cloud-f7cd-mgs0.eastus.cloudapp.azure.com"

mount_command = "mount -t lustre 10.0.0.11@tcp:/exacloud /mnt/exacloud"

private_addresses = {
  "exascaler-cloud-f7cd-cls0" = "10.0.0.6"
  "exascaler-cloud-f7cd-cls1" = "10.0.0.7"
  "exascaler-cloud-f7cd-mds0" = "10.0.0.8"
  "exascaler-cloud-f7cd-mgs0" = "10.0.0.11"
  "exascaler-cloud-f7cd-oss0" = "10.0.0.9"
  "exascaler-cloud-f7cd-oss1" = "10.0.0.4"
  "exascaler-cloud-f7cd-oss2" = "10.0.0.10"
  "exascaler-cloud-f7cd-oss3" = "10.0.0.5"
}

ssh_console = {
  "exascaler-cloud-f7cd-mgs0" = "ssh -A stack@exascaler-cloud-f7cd-mgs0.eastus.cloudapp.azure.com"
}

Steps to destroy the EXAScaler Cloud environment

Destroy the EXAScaler Cloud environment:

Code Block

language	bash
theme	Midnight

$ terraform destroy
...
  Enter a value: yes
...
Destroy complete! Resources: 103 destroyed.

How to access a deployment

To remote access the EXAScaler Cloud deployment (management, metadata, storage servers and compute clients), we must go through the management server public IP address, using the EXAScaler Cloud SSH Console command.

...

Code Block

language	bash
theme	Midnight

localhost$ eval $(ssh-agent)
Agent pid 5095

localhost$ ssh-add 
Identity added: /Users/deiter/.ssh/id_rsa (/Users/deiter/.ssh/id_rsa)

localhost$ ssh -A stack@20.62.171.73
Last login: Tue Aug 24 19:11:18 2021 from 181.241.16.227

[stack@exascaler-cloud-65f1-mgs0 ~]$ loci hosts
10.0.0.7	exascaler-cloud-65f1-cls0
10.0.0.5	exascaler-cloud-65f1-cls1
10.0.0.8	exascaler-cloud-65f1-cls2
10.0.0.13	exascaler-cloud-65f1-cls3
10.0.0.12	exascaler-cloud-65f1-mds0
10.0.0.11	exascaler-cloud-65f1-mgs0
10.0.0.10	exascaler-cloud-65f1-oss0
10.0.0.4	exascaler-cloud-65f1-oss1
10.0.0.6	exascaler-cloud-65f1-oss2
10.0.0.9	exascaler-cloud-65f1-oss3

[stack@exascaler-cloud-65f1-mgs0 ~]$ ssh exascaler-cloud-65f1-cls0

[stack@exascaler-cloud-65f1-cls0 ~]$ lfs df
UUID                   1K-blocks        Used   Available Use% Mounted on
exacloud-MDT0000_UUID   315302464        5744   309928012   1% /mnt/exacloud[MDT:0]
exacloud-OST0000_UUID  3181466888   570662088  2578541648  19% /mnt/exacloud[OST:0]
exacloud-OST0001_UUID  3181466888   590910696  2558305972  19% /mnt/exacloud[OST:1]
exacloud-OST0002_UUID  3181466888   580374740  2568825852  19% /mnt/exacloud[OST:2]
exacloud-OST0003_UUID  3181466888   570645704  2578552816  19% /mnt/exacloud[OST:3]

filesystem_summary:  12725867552  2312593228 10284226288  19% /mnt/exacloud

How to install new EXAScaler Cloud clients

New EXAScaler Cloud client instances must be in the same location and connected to the virtual network and subnet. The process of installing and configuring new clients can be performed automatically. All required information is contained in the Terraform output. To configure EXAScaler Cloud filesystem on a new client instance create a configuration file /etc/

...

exascaler-cloud-client.

...

conf using the actual IP address of the management server:

Code Block

language	bash
theme	Midnight

{
    "Version": "2.0.0",
    "MountConfig": {
        "ClientDevice": "10.0.0.10@tcp:/exacloud",
        "Mountpoint": "/mnt/exacloud",
        "PackageSource": "http://10.0.0.10/client-packages"
    }
}

...

Code Block

language	bash
theme	Midnight

# curl -fsSL http://10.0.0.10/exasccaler-cloud-client-setup-tool$(arch) -o /usr/sbin/escexascaler-cloud-client
# chmod +x /usr/sbin/escexascaler-cloud-client
# escexascaler-cloud-client auto setup --config /etc/escexascaler-cloud-client.cfgconf

Ubuntu Linux

Code Block

language	bash
theme	Midnight

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy

# escexascaler-cloud-client auto setup --config /etc/escexascaler-cloud-client.cfgconf
Discovering platform ... Done.
Configuring firewall rules for Lustre ... Done.
Configuring Lustre client package source ... Done.
Installing Lustre client packages and building DKMS modules ... Done.
Mounting 10.0.0.10@tcp0:/exacloud at /mnt/exacloud ... Done.

# mount -t lustre
10.0.0.10@tcp:/exacloud on /mnt/exacloud type lustre (rw,flock,user_xattr,lazystatfs,encrypt)

RHEL, CentOS, Alma and Rocky Linux

Code Block

language	bash
theme	Midnight

# cat /etc/redhat-release
AlmaLinux release 8.6 (Sky Tiger)

# escexascaler-cloud-client auto setup --config /etc/escexascaler-cloud-client.cfgconf
Discovering platform ... Done.
Configuring firewall rules for Lustre ... Done.
Configuring Lustre client package source ... Done.
Installing Lustre client packages ... Done.
Mounting 10.0.0.10@tcp0:/exacloud at /mnt/exacloud ... Done.

# mount -t lustre
10.0.0.10@tcp:/exacloud on /mnt/exacloud type lustre (rw,seclabel,flock,user_xattr,lazystatfs,encrypt)

How to use client-side encryption

The purpose that client-side encryption wants to serve is to be able to provide a special directory for each user, to safely store sensitive files. The goals are to protect data in transit between clients and servers, and protect data at rest.

...

Code Block

language	bash
theme	Midnight

# cat > /etc/escexascaler-cloud-client.cfgconf <<EOF
{
    "Version": "2.0.0",
    "MountConfig": {
        "ClientDevice": "10.0.0.10@tcp:/exacloud",
        "Mountpoint": "/mnt/exacloud",
        "PackageSource": "http://10.0.0.10/client-packages"
    }
}
EOF

# curl -fsSL http://10.0.0.10/exascaler-cloud-client-setup-tool$(arch) -o /usr/sbin/escexascaler-cloud-client
# chmod +x /usr/sbin/escexascaler-cloud-client
# escexascaler-cloud-client auto setup --config /etc/esc-client.cfgconf

Output:

Code Block

language	bash
theme	Midnight

# escexascaler-cloud-client auto setup --config /etc/escexascaler-cloud-client.cfgconf
Discovering platform ... Done.
Configuring firewall rules for Lustre ... Done.
Configuring Lustre client package source ... Done.
Installing Lustre client packages ... Done.
Mounting 10.0.0.10@tcp0:/exacloud at /mnt/exacloud ... Done.

# rpm -q fscrypt lustre-client kmod-lustre-client
fscrypt-0.3.3-1.wc2.x86_64
lustre-client-2.14.0_ddn52-1.el8.x86_64
kmod-lustre-client-2.14.0_ddn52-1.el8.x86_64

...

Learn more about client-side encryption.

How to run benchmarks

Steps to run IOR benchmark on the EXAScaler Cloud deployment:

...

Code Block

language	bash
theme	Midnight

localhost$ eval $(ssh-agent)
Agent pid 5095

localhost$ ssh-add 
Identity added: /Users/deiter/.ssh/id_rsa (/Users/deiter/.ssh/id_rsa)

localhost$ ssh -A stack@20.62.171.73

[stack@exascaler-cloud-65f1-mgs0 ~]$ escexascaler-cloud-ior

IOR-3.3.0: MPI Coordinated Test of Parallel I/O

Began               : Wed Aug 25 14:43:01 2021
Command line        : /usr/bin/ior -C -F -e -r -w -a POSIX -b 16777216 -t 1048576 -s 251 -o /mnt/exacloud/0b21199cf9682b9d/0b21199cf9682b9d -s 512
Machine             : Linux exascaler-cloud-65f1-cls0
TestID              : 0
StartTime           : Wed Aug 25 14:43:01 2021
Path                : /mnt/exacloud/0b21199cf9682b9d
FS                  : 11.9 TiB   Used FS: 0.0%   Inodes: 96.0 Mi   Used Inodes: 0.0%

Options: 
api                 : POSIX
apiVersion          : 
test filename       : /mnt/exacloud/0b21199cf9682b9d/0b21199cf9682b9d
access              : file-per-process
type                : independent
segments            : 512
ordering in a file  : sequential
ordering inter file : constant task offset
task offset         : 1
nodes               : 4
tasks               : 64
clients per node    : 16
repetitions         : 1
xfersize            : 1 MiB
blocksize           : 16 MiB
aggregate filesize  : 512 GiB

Results: 

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)    wr/rd(s)   close(s)   total(s)   iter
------    ---------  ----       ----------  ---------- ---------  --------   --------   --------   --------   ----
write     1482.70    1482.70    21.52       16384      1024.00    0.014559   353.60     19.92      353.60     0   
read      1480.89    1480.91    21.99       16384      1024.00    0.299514   354.03     43.40      354.04     0   
remove    -          -          -           -          -          -          -          -          3.36       0   

Max Write: 1482.70 MiB/sec (1554.72 MB/sec)
Max Read:  1480.89 MiB/sec (1552.83 MB/sec)

Finished            : Wed Aug 25 14:54:52 2021

...

Code Block

language	bash
theme	Midnight

localhost$ eval $(ssh-agent)
Agent pid 5095

localhost$ ssh-add 
Identity added: /Users/deiter/.ssh/id_rsa (/Users/deiter/.ssh/id_rsa)

localhost$ ssh -A stack@20.62.171.71

[stack@exascaler-cloud-2ed3-mgs0 ~]$ $ loci hosts -c
10.0.0.17   exascaler-cloud-2ed3-cls0
10.0.0.8    exascaler-cloud-2ed3-cls1
10.0.0.18   exascaler-cloud-2ed3-cls2
10.0.0.12   exascaler-cloud-2ed3-cls3
10.0.0.14   exascaler-cloud-2ed3-cls4
10.0.0.15   exascaler-cloud-2ed3-cls5
10.0.0.11   exascaler-cloud-2ed3-cls6
10.0.0.9    exascaler-cloud-2ed3-cls7
10.0.0.7    exascaler-cloud-2ed3-cls8
10.0.0.16   exascaler-cloud-2ed3-cls9
 
[stack@exascaler-cloud-2ed3-mgs0 ~]$ ssh -A exascaler-cloud-2ed3-cls0
 
[stack@exascaler-cloud-2ed3-cls0 ~]$ escexascaler-cloud-io500
 
Start IO500 benchmark with options:
 
data directory:     /mnt/exacloud/071dfa36e6b20ca7/workload
hosts list:         10.0.0.17,10.0.0.9,10.0.0.14,10.0.0.8,10.0.0.12,10.0.0.5,10.0.0.11,10.0.0.19,10.0.0.18,10.0.0.7
processes per host: 16
files per process:  39637
number of tasks:    160
number of segments: 31500
block size:         4227858432
transfer size:      1048576
 
IO500 version io500-sc20_v3
[RESULT]       ior-easy-write        1.445976 GiB/s : time 364.894 seconds
[RESULT]    mdtest-easy-write       15.411987 kIOPS : time 304.382 seconds
[RESULT]       ior-hard-write        0.461174 GiB/s : time 410.219 seconds
[RESULT]    mdtest-hard-write        2.538281 kIOPS : time 449.131 seconds
[RESULT]                 find      583.795841 kIOPS : time 9.842 seconds
[RESULT]        ior-easy-read        1.450889 GiB/s : time 363.624 seconds
[RESULT]     mdtest-easy-stat       61.106840 kIOPS : time 75.517 seconds
[RESULT]        ior-hard-read        0.543306 GiB/s : time 348.233 seconds
[RESULT]     mdtest-hard-stat       20.753560 kIOPS : time 54.080 seconds
[RESULT]   mdtest-easy-delete        5.836530 kIOPS : time 789.832 seconds
[RESULT]     mdtest-hard-read       10.320768 kIOPS : time 108.658 seconds
[RESULT]   mdtest-hard-delete        4.647816 kIOPS : time 241.181 seconds
[SCORE] Bandwidth 0.851483 GiB/s : IOPS 17.322863 kiops : TOTAL 3.840589
 
/mnt/exacloud/b44731b2e4ac4fc2/sources/results
2021.06.17-20.26.19  io500-exascaler-cloud-2ed3-cls0-2021.06.17-20.26.19.tgz

How to collect inventory and support bundle

Steps to collect a support bundle on the EXAScaler Cloud deployment:

...

Code Block

language	bash
theme	Midnight

localhost$ eval $(ssh-agent)
Agent pid 5095

localhost$ ssh-add 
Identity added: /Users/deiter/.ssh/id_rsa (/Users/deiter/.ssh/id_rsa)

localhost$ ssh -A stack@20.62.171.33

[stack@exascaler-cloud-55d2-mgs0 ~]$ escexascaler-cloud-collector
 
The following is a list of nodes to collect from:
                              
    exascaler-cloud-55d2-cls0
    exascaler-cloud-55d2-cls1
    exascaler-cloud-55d2-cls2
    exascaler-cloud-55d2-cls3
    exascaler-cloud-55d2-cls4
    exascaler-cloud-55d2-cls5
    exascaler-cloud-55d2-cls6
    exascaler-cloud-55d2-cls7
    exascaler-cloud-55d2-cls8
    exascaler-cloud-55d2-cls9
    exascaler-cloud-55d2-mds0
    exascaler-cloud-55d2-mgs0
    exascaler-cloud-55d2-oss0
    exascaler-cloud-55d2-oss1
    exascaler-cloud-55d2-oss2
    exascaler-cloud-55d2-oss3
 
Connecting to nodes...
 
Beginning collection of sosreports from 16 nodes, collecting a maximum of 4 concurrently
 
Successfully captured 16 of 16 sosreports
Creating archive of sosreports...
 
The following archive has been created. Please provide it to your support team.
    /var/tmp/sos-collector-2021-06-18-nzsnm.tar.gz

...

Code Block

language	bash
theme	Midnight

localhost$ $ ssh -A stack@13.87.189.176

[stack@exascaler-cloud-7819-mgs0 ~]$ about_this_deployment 
cloudName: AzurePublicCloud
subscriptionId: 9978cd1b-936a-4296-8061-67c9d963dd40
location: westus
resourceGroupName: CustomDeployment
deployment: exascaler-cloud-7819
filesystem: exacloud
capacityGB: 1024
configurationProfile: custom
instances:
- hostName: exascaler-cloud-7819-cls0
  userName: stack
  proximityPlacementGroup: exascaler-cloud-7819-proximity-placement-group
  passwordAuthentication: false
  instanceName: exascaler-cloud-7819-cls0
  instanceType: Standard_D16s_v3
  role: clt
  memoryGB: 64
  VCPUs: 16
  IOPS: 25600
  bandwidthMBps: 384
  network:
    interfaces:
    - name: exascaler-cloud-7819-cls0-network-interface
      acceleratedNetworking: true
      macAddress: 00-22-48-04-1F-A9
      ipAddresses:
      - privateIpAddress: 10.0.0.6
        subnet: exascaler-cloud-7819-subnetwork
  storage:
    image:
      offer: exascaler_cloud
      publisher: ddn-whamcloud-5345716
      sku: exascaler_cloud_523_redhat
      version: 5.2.3
    bootDisk:
      caching: ReadWrite
      sizeGB: 64
      name: exascaler-cloud-7819-cls0-boot-disk
      tier: StandardSSD_LRS
      type: E6
      IOPS: 500
      bandwidthMBps: 60
  publicKeys:
  - data: |
      ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4haGlji/jE0B35HrBKXeIxos2YN8jMP2m1+cx1G7UYnqr7c1I4wv073ubEfOQM2Cww4+CKEAQT3ZC+EW79NQuwJPAzPr2y961INN1zTYEX84ejbmydXzdWJT5u8LKgwKFkkKPzOU7tWJ2Co6rhz796siOpH1bvRg8gFt1oEk9B2fd3O3OA62zq6La0IUYkMQzTlU2b0hNvrVuX+Ea38fi18R7op7tSulystSrvjW/8fzRU8r7c8oTD8kELRh9H3zm0S8mk+Nq1fO4liXtNldnnpl3qAV9NipGXb+L9De15ogtqJE1/2wyaEdbXjqpA2BkHeUY8B2K7K6afpndR+AT stack@host
    path: /home/stack/.ssh/authorized_keys
- hostName: exascaler-cloud-7819-mds0
  userName: stack
  proximityPlacementGroup: exascaler-cloud-7819-proximity-placement-group
  passwordAuthentication: false
  instanceName: exascaler-cloud-7819-mds0
  instanceType: Standard_E8s_v3
  role: mdt
  memoryGB: 64
  VCPUs: 8
  IOPS: 12800
  bandwidthMBps: 192
  network:
    interfaces:
    - name: exascaler-cloud-7819-mds0-network-interface
      acceleratedNetworking: true
      macAddress: 00-22-48-0A-05-EE
      ipAddresses:
      - privateIpAddress: 10.0.0.7
        subnet: exascaler-cloud-7819-subnetwork
  storage:
    image:
      offer: exascaler_cloud
      publisher: ddn-whamcloud-5345716
      sku: exascaler_cloud_523_redhat
      version: 5.2.3
    bootDisk:
      caching: ReadWrite
      sizeGB: 64
      name: exascaler-cloud-7819-mds0-boot-disk
      tier: StandardSSD_LRS
      type: E6
      IOPS: 500
      bandwidthMBps: 60
    dataDisks:
    - lun: 0
      caching: None
      sizeGB: 512
      name: exascaler-cloud-7819-mds0-mdt0-disk
      tier: Premium_LRS
      type: P20
      IOPS: 2300
      bandwidthMBps: 150
  publicKeys:
  - data: |
      ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4haGlji/jE0B35HrBKXeIxos2YN8jMP2m1+cx1G7UYnqr7c1I4wv073ubEfOQM2Cww4+CKEAQT3ZC+EW79NQuwJPAzPr2y961INN1zTYEX84ejbmydXzdWJT5u8LKgwKFkkKPzOU7tWJ2Co6rhz796siOpH1bvRg8gFt1oEk9B2fd3O3OA62zq6La0IUYkMQzTlU2b0hNvrVuX+Ea38fi18R7op7tSulystSrvjW/8fzRU8r7c8oTD8kELRh9H3zm0S8mk+Nq1fO4liXtNldnnpl3qAV9NipGXb+L9De15ogtqJE1/2wyaEdbXjqpA2BkHeUY8B2K7K6afpndR+AT stack@host
    path: /home/stack/.ssh/authorized_keys
- hostName: exascaler-cloud-7819-mgs0
  userName: stack
  proximityPlacementGroup: exascaler-cloud-7819-proximity-placement-group
  passwordAuthentication: false
  instanceName: exascaler-cloud-7819-mgs0
  instanceType: Standard_F4s
  role: mgt
  memoryGB: 8
  VCPUs: 4
  IOPS: 12800
  bandwidthMBps: 192
  network:
    interfaces:
    - name: exascaler-cloud-7819-mgs0-network-interface
      acceleratedNetworking: true
      macAddress: 00-22-48-04-1B-91
      ipAddresses:
      - privateIpAddress: 10.0.0.5
        publicIpAddress: 13.87.189.176
        subnet: exascaler-cloud-7819-subnetwork
  storage:
    image:
      offer: exascaler_cloud
      publisher: ddn-whamcloud-5345716
      sku: exascaler_cloud_523_redhat
      version: 5.2.3
    bootDisk:
      caching: ReadWrite
      sizeGB: 64
      name: exascaler-cloud-7819-mgs0-boot-disk
      tier: StandardSSD_LRS
      type: E6
      IOPS: 500
      bandwidthMBps: 60
    dataDisks:
    - lun: 0
      caching: None
      sizeGB: 256
      name: exascaler-cloud-7819-mgs0-mgt0-disk
      tier: StandardSSD_LRS
      type: E15
      IOPS: 500
      bandwidthMBps: 60
    - lun: 1
      caching: None
      sizeGB: 128
      name: exascaler-cloud-7819-mgs0-mnt0-disk
      tier: StandardSSD_LRS
      type: E10
      IOPS: 500
      bandwidthMBps: 60
  publicKeys:
  - data: |
      ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4haGlji/jE0B35HrBKXeIxos2YN8jMP2m1+cx1G7UYnqr7c1I4wv073ubEfOQM2Cww4+CKEAQT3ZC+EW79NQuwJPAzPr2y961INN1zTYEX84ejbmydXzdWJT5u8LKgwKFkkKPzOU7tWJ2Co6rhz796siOpH1bvRg8gFt1oEk9B2fd3O3OA62zq6La0IUYkMQzTlU2b0hNvrVuX+Ea38fi18R7op7tSulystSrvjW/8fzRU8r7c8oTD8kELRh9H3zm0S8mk+Nq1fO4liXtNldnnpl3qAV9NipGXb+L9De15ogtqJE1/2wyaEdbXjqpA2BkHeUY8B2K7K6afpndR+AT stack@host
    path: /home/stack/.ssh/authorized_keys
- hostName: exascaler-cloud-7819-oss0
  userName: stack
  proximityPlacementGroup: exascaler-cloud-7819-proximity-placement-group
  passwordAuthentication: false
  instanceName: exascaler-cloud-7819-oss0
  instanceType: Standard_D16s_v3
  role: ost
  memoryGB: 64
  VCPUs: 16
  IOPS: 25600
  bandwidthMBps: 384
  network:
    interfaces:
    - name: exascaler-cloud-7819-oss0-network-interface
      acceleratedNetworking: true
      macAddress: 00-22-48-0A-0D-54
      ipAddresses:
      - privateIpAddress: 10.0.0.4
        subnet: exascaler-cloud-7819-subnetwork
  storage:
    image:
      offer: exascaler_cloud
      publisher: ddn-whamcloud-5345716
      sku: exascaler_cloud_523_redhat
      version: 5.2.3
    bootDisk:
      caching: ReadWrite
      sizeGB: 64
      name: exascaler-cloud-7819-oss0-boot-disk
      tier: StandardSSD_LRS
      type: E6
      IOPS: 500
      bandwidthMBps: 60
    dataDisks:
    - lun: 0
      caching: None
      sizeGB: 512
      name: exascaler-cloud-7819-oss0-ost0-disk
      tier: Standard_LRS
      type: S20
      IOPS: 500
      bandwidthMBps: 60
    - lun: 1
      caching: None
      sizeGB: 512
      name: exascaler-cloud-7819-oss0-ost1-disk
      tier: Standard_LRS
      type: S20
      IOPS: 500
      bandwidthMBps: 60
  publicKeys:
  - data: |
      ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4haGlji/jE0B35HrBKXeIxos2YN8jMP2m1+cx1G7UYnqr7c1I4wv073ubEfOQM2Cww4+CKEAQT3ZC+EW79NQuwJPAzPr2y961INN1zTYEX84ejbmydXzdWJT5u8LKgwKFkkKPzOU7tWJ2Co6rhz796siOpH1bvRg8gFt1oEk9B2fd3O3OA62zq6La0IUYkMQzTlU2b0hNvrVuX+Ea38fi18R7op7tSulystSrvjW/8fzRU8r7c8oTD8kELRh9H3zm0S8mk+Nq1fO4liXtNldnnpl3qAV9NipGXb+L9De15ogtqJE1/2wyaEdbXjqpA2BkHeUY8B2K7K6afpndR+AT stack@host
    path: /home/stack/.ssh/authorized_keys

How to stop and start a deployment

If you want to stop and start the servers in your EXAScaler Cloud deployment (for example, to save cost on weekends) - please use the esc-ctl script.

Prerequisites

You need a Microsoft account
Your system needs the Microsoft Azure CLI

Before use the Microsoft Azure CLI for Microsoft Azure, you will need to authenticate under the Microsoft account you used to log into the Microsoft Azure Portal. You will use a Microsoft account and its credentials to allow the shell script to start/stop the EXAScaler Cloud servers.

Steps to authenticate via Microsoft account

Obtains access credentials for your user account via a web-based authorization flow. When this command completes successfully, it sets the active account in the current configuration to the account specified. Learn more.

...

Code Block

language	bash
theme	Midnight

$ az account set --subscription XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

List of existing resource groups

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl list
Name              Location    Status
----------------  ----------  ---------
EXAScaler-Cloud   eastus      Succeeded

List of existing deployments for the given resource group

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud list
Name                            Created                    Status
------------------------------  -------------------------  ---------
exascaler-cloud-65f1            2021-08-24T18:56:27+00:00  Succeeded

List of EXAScaler Cloud servers for the given deployment

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud exascaler-cloud-65f1 list
Name                       Size              Role    Version    PrivateIP    PublicIP      Status
-------------------------  ----------------  ------  ---------  -----------  ------------  ----------
exascaler-cloud-65f1-cls0  Standard_D16s_v3  clt     5.2.3      10.0.0.7                   VM running
exascaler-cloud-65f1-cls1  Standard_D16s_v3  clt     5.2.3      10.0.0.5                   VM running
exascaler-cloud-65f1-cls2  Standard_D16s_v3  clt     5.2.3      10.0.0.8                   VM running
exascaler-cloud-65f1-cls3  Standard_D16s_v3  clt     5.2.3      10.0.0.13                  VM running
exascaler-cloud-65f1-mds0  Standard_E8s_v3   mdt     5.2.3      10.0.0.12                  VM running
exascaler-cloud-65f1-mgs0  Standard_F4s      mgt     5.2.3      10.0.0.11    20.62.171.73  VM running
exascaler-cloud-65f1-oss0  Standard_D16s_v3  ost     5.2.3      10.0.0.10                  VM running
exascaler-cloud-65f1-oss1  Standard_D16s_v3  ost     5.2.3      10.0.0.4                   VM running
exascaler-cloud-65f1-oss2  Standard_D16s_v3  ost     5.2.3      10.0.0.6                   VM running
exascaler-cloud-65f1-oss3  Standard_D16s_v3  ost     5.2.3      10.0.0.9                   VM running

Stop the EXAScaler Cloud servers

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud exascaler-cloud-65f1 stop
Stop compute client exascaler-cloud-65f1-cls0
Stop compute client exascaler-cloud-65f1-cls1
Stop compute client exascaler-cloud-65f1-cls2
Stop compute client exascaler-cloud-65f1-cls3
Stop storage server exascaler-cloud-65f1-oss0
Stop storage server exascaler-cloud-65f1-oss1
Stop storage server exascaler-cloud-65f1-oss2
Stop storage server exascaler-cloud-65f1-oss3
Stop metadata server exascaler-cloud-65f1-mds0
Stop management server exascaler-cloud-65f1-mgs0

Start the EXAScaler Cloud servers

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud exascaler-cloud-65f1 start
Start management server exascaler-cloud-65f1-mgs0
Start metadata server exascaler-cloud-65f1-mds0
Start storage server exascaler-cloud-65f1-oss0
Start storage server exascaler-cloud-65f1-oss1
Start storage server exascaler-cloud-65f1-oss2
Start storage server exascaler-cloud-65f1-oss3
Start compute client exascaler-cloud-65f1-cls0
Start compute client exascaler-cloud-65f1-cls1
Start compute client exascaler-cloud-65f1-cls2
Start compute client exascaler-cloud-65f1-cls3

All required services will automatically start and the filesystem will be mounted and available on all compute clients.

How to upgrade an existing deployment

Anchor

	Upgrade
	Upgrade

A software upgrade for an existing EXAScaler Cloud deployment is possible by creating a new deployment using a copy of the existing file system.

...

Code Block

language	bash
theme	Midnight

$ .scripts/escexascaler-cloud-ctl list                                                 
Name              Location    Status                                                                                              
----------------  ----------  ---------                                                                                           
EXAScaler-Cloud   eastus      Succeeded

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud list
Name                            Created                    Status
------------------------------  -------------------------  ---------
exascaler-cloud-65f1            2021-08-24T18:56:27+00:00  Succeeded

$ .scripts/escexascaler-cloud-ctl EXAScaler-Cloud exascaler-cloud-65f1 stop
Stop compute client exascaler-cloud-65f1-cls0
Stop compute client exascaler-cloud-65f1-cls1
Stop compute client exascaler-cloud-65f1-cls2
Stop compute client exascaler-cloud-65f1-cls3
Stop storage server exascaler-cloud-65f1-oss0
Stop storage server exascaler-cloud-65f1-oss1
Stop storage server exascaler-cloud-65f1-oss2
Stop storage server exascaler-cloud-65f1-oss3
Stop metadata server exascaler-cloud-65f1-mds0
Stop management server exascaler-cloud-65f1-mgs0

...

Page tree

Page History

Versions Compared

Old Version 68

New Version Current

Key

How to deploy an EXAScaler Cloud Application

Step by step deployment video

Step by step deployment guide

How to automate deployment with Terraform

Supported products

Client packages

Prerequisites

Steps to authenticate via Microsoft account

Steps to accept the terms of use for DDN EXAScaler Cloud images

Steps to configure Terraform

List of available variables

Common options

Availability options

Resource group options

Proximity placement group options

Network options

Subnet options

Security options

Boot disk options

Source image options

Storage account options

Management server options

Management target options

Monitoring target options

Metadata server options

Metadata target options

Object Storage server options

Object Storage target options

Compute client options

Compute client target options

Deploy an EXAScaler Cloud environment

Access the EXAScaler Cloud environment

Add storage capacity in an existing EXAScaler Cloud environment

Upgrade an existing EXAScaler Cloud environment

Steps to destroy the EXAScaler Cloud environment

How to access a deployment

How to install new EXAScaler Cloud clients

Ubuntu Linux

RHEL, CentOS, Alma and Rocky Linux

How to use client-side encryption

How to run benchmarks

How to collect inventory and support bundle

How to stop and start a deployment

Prerequisites

Steps to authenticate via Microsoft account

List of existing resource groups

List of existing deployments for the given resource group

List of EXAScaler Cloud servers for the given deployment

Stop the EXAScaler Cloud servers

Start the EXAScaler Cloud servers

How to upgrade an existing deployment