Page History

Lustre server machines are simple to setup with CentOS 56.52. Pre-built kernel images are available from Whamclouds build server. A number of different kernels and tools are continuasly built and tested. For this Walk-thru the centoslustre-master - build will be used.

Prerequisite.

• A x86_64 machine with CentOS 56.5 2 installed. This The machine has the hostname client-10 rhel6_vanilla.

Procedure

...

Install Lustre patched

...

kernel.

1. From lustre-master build, download the correct kernel and kernel-firmware RPMs for your platform.
2. Install these RPMs with yum localinstall ./kernel*

   Code Block
   [root@rhel6_vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./

Installing a patched kernel.

1. Identify the recent, Lustre patched, kernel.
   This can be found by either navigaing from the lustre-master page or by visiting a link the the last successful build for x86_64.
2. Identify the most recent kernel RPM.
   Recent kernel builds are of the form

   kernel-2.6.

   18

   32-

   194.17.1.el5

   220.el6_lustre.

   <buildid>.

   x86_64.rpm

3. Download this RPM to the machine client-10.
4. Install the kernel.

   Code Block
   [root@client-10 ~]# rpm -ivh : kernel-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-2.6.1832-194.17.1.el5220.el6_lustre.<buildid>.x86_64.rpm Preparing...as an update to kernel-2.6.32-220.el6.x86_64 base ########################################### [100%] 1:kernel ########################################### [100%] [root@client-10 ~]#

5. Reboot
6. Verify that the new kernel has been booted:

   Code Block
   [root@client-10 ~]# uname -a Linux client-10.lab.whamcloud.com 2.6.18-194.17.1.el5_lustre.gdfe90dc #1 SMP Sun Mar 13 03:18:21 EET 2011 x86_64 x86_64 x86_64 GNU/Linux

Installing Ldiskfs.

1. From the page that linked to the kernel, identify the Lustre modules RPM
   The package will be named in the form lustre-ldiskfs-3.3.0-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm
2. Download the Lustre modules RPM.
3. Install the RPM.

   Code Block
   [root@client-10 ~]# rpm -ivh ./lustre-ldiskfs-3.3.0-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm Preparing... ########################################### [100%] 1:lustre-ldiskfs ########################################### [100%] [root@client-10 ~]#

Install Lustre modules.

1. From the page that linked to the kernel, identify the Lustre modules RPM
   The package will be named in the form lustre-modules-2.0.59-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm
2. Download the Lustre modules RPM.
3. Install the RPM.

   Code Block
   [root@client-10 ~]# rpm -ivh ./lustre-modules-2.0.59-2.6.18_194.17.1.el5_lustre.gdfe90dc_g98bf9c9.x86_64.rpm Preparing... ########################################### [100%] 1:lustre-modules ########################################### [100%] [root@client-10 ~]#

Install e2fsprogs.

e2fsprogs is built in a project separate from the Lustre kernel builds. e2fsprogs is found at http://build.whamcloud.com/job/e2fsprogs-master/ on the Whamcloud build server.

1. Identify the most recent build of e2fsprogs.
   The RPM will have a name in the form: e2fsprogs-1.41.90.wc3-0redhat.x86_64.rpm
2. Download the most recent RPM.
3. Install the RPM

...

1. | 3.7 kB 00:00 base/primary_db | 4.5 MB 00:02 extras | 3.5 kB 00:00 extras/primary_db | 6.3 kB 00:00 updates | 3.5 kB 00:00 updates/primary_db | 1.2 MB 00:00 Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package Arch Version Repository Size ============================================================================================================= Installing: kernel x86_64 2.6.32-220.el6_lustre /kernel-2.6.32-220.el6_lustre.x86_64 112 M Updating for dependencies: kernel-firmware noarch 2.6.32-220.4.1.el6 updates 6.2 M Transaction Summary ============================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 118 M Total download size: 6.2 M Is this ok [y/N]: y Downloading Packages: kernel-firmware-2.6.32-220.4.1.el6.noarch.rpm | 6.2 MB 00:01 warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-220.4.1.el6.noarch 1/3 Installing : kernel-2.6.32-220.el6_lustre.x86_64 2/3 Cleanup : kernel-firmware-2.6.32-220.el6.noarch 3/3 Installed: kernel.x86_64 0:2.6.32-220.el6_lustre Dependency Updated: kernel-firmware.noarch 0:2.6.32-220.4.1.el6 Complete! [root@rhel6_vanilla ~]#

Download e2fsprogs and Lustre server RPMSs.

1. Download the following e2fsprogs RPMs from http://build.whamcloud.com/job/e2fsprogs-master
   • e2fsprogs
   • e2fsprogs-libs
2. Download the following Lustre server RPMs
   • lustre
   • lustre-ldiskfs
   • lustre-modules
   • lustre-tests

Install Lustre.

1. yum localinstall ./lustre-* ./e2fsprogs-* should do it:

   Code Block

   [root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Loading mirror speeds from cached hostfile * base: ftp.usf.edu * extras: mirror.raystedman.net * updates: mirror.raystedman.net Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-1.41.90.wc4-7.el6.x86_64 Marking ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-11.el6.x86_64 Examining ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 Marking ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-11.el6.x86_64 Resolving Dependencies --> Running transaction check ---> Package e2fsprogs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs.x86_64 0:1.41.90.wc4-7.el6 will be an update ---> Package e2fsprogs-libs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6 will be an update ---> Package lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed --> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed ---> Package lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed ---> Package lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed --> Processing Dependency: libmpi.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libopen-pal.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libopen-rte.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Running transaction check ---> Package compat-openmpi.x86_64 0:1.4.3-1.el6 will be installed --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: environment-modules for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibcm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libplpa.so.0()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libgfortran.so.3()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libnuma.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 ---> Package net-snmp-libs.x86_64 1:5.5-37.el6_2.1 will be installed --> Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-37.el6_2.1.x86_64 --> Running transaction check ---> Package environment-modules.x86_64 0:3.2.7b-6.el6 will be installed --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.7b-6.el6.x86_64 ---> Package libgfortran.x86_64 0:4.4.6-3.el6 will be installed ---> Package libibcm.x86_64 0:1.0.5-2.el6 will be installed ---> Package libibverbs.x86_64 0:1.1.5-3.el6 will be installed ---> Package librdmacm.x86_64 0:1.0.14.1-3.el6 will be installed ---> Package lm_sensors-libs.x86_64 0:3.1.1-10.el6 will be installed ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed ---> Package plpa-libs.x86_64 0:1.3.2-2.1.el6 will be installed --> Running transaction check ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package Arch Version Repository Size ============================================================================================================= Installing: lustre x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 24 M lustre-ldiskfs x86_64 3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 8.8 M lustre-modules x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 139 M lustre-tests x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 12 M Updating: e2fsprogs x86_64 1.41.90.wc4-7.el6 /e2fsprogs-1.41.90.wc4-7.el6.x86_64 2.3 M e2fsprogs-libs x86_64 1.41.90.wc4-7.el6 /e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 308 k Installing for dependencies: compat-openmpi x86_64 1.4.3-1.el6 base 1.3 M environment-modules x86_64 3.2.7b-6.el6 base 96 k libgfortran x86_64 4.4.6-3.el6 base 263 k libibcm x86_64 1.0.5-2.el6 base 19 k libibverbs x86_64 1.1.5-3.el6 base 44 k librdmacm x86_64 1.0.14.1-3.el6 base 27 k lm_sensors-libs x86_64 3.1.1-10.el6 base 37 k net-snmp-libs x86_64 1:5.5-37.el6_2.1 updates 1.5 M numactl x86_64 2.0.3-9.el6 base 55 k plpa-libs x86_64 1.3.2-2.1.el6 base 25 k tcl x86_64 1:8.5.7-6.el6 base 1.9 M Transaction Summary ============================================================================================================= Install 15 Package(s) Upgrade 2 Package(s) Total size: 192 M Total download size: 5.3 M Is this ok [y/N]: y Downloading Packages: (1/11): compat-openmpi-1.4.3-1.el6.x86_64.rpm | 1.3 MB 00:00 (2/11): environment-modules-3.2.7b-6.el6.x86_64.rpm | 96 kB 00:00 (3/11): libgfortran-4.4.6-3.el6.x86_64.rpm | 263 kB 00:00 (4/11): libibcm-1.0.5-2.el6.x86_64.rpm | 19 kB 00:00 (5/11): libibverbs-1.1.5-3.el6.x86_64.rpm | 44 kB 00:00 (6/11): librdmacm-1.0.14.1-3.el6.x86_64.rpm | 27 kB 00:00 (7/11): lm_sensors-libs-3.1.1-10.el6.x86_64.rpm | 37 kB 00:00 (8/11): net-snmp-libs-5.5-37.el6_2.1.x86_64.rpm | 1.5 MB 00:03 (9/11): numactl-2.0.3-9.el6.x86_64.rpm | 55 kB 00:00 (10/11): plpa-libs-1.3.2-2.1.el6.x86_64.rpm | 25 kB 00:00 (11/11): tcl-8.5.7-6.el6.x86_64.rpm | 1.9 MB 00:00 ------------------------------------------------------------------------------------------------------------- Total 1.0 MB/s | 5.3 MB 00:05 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : libibverbs-1.1.5-3.el6.x86_64 1/19 Updating : e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 2/19 Updating : e2fsprogs-1.41.90.wc4-7.el6.x86_64 3/19 Installing : lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 4/19 Installing : lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 5/19 Installing : libibcm-1.0.5-2.el6.x86_64 6/19 Installing : librdmacm-1.0.14.1-3.el6.x86_64 7/19 Installing : lm_sensors-libs-3.1.1-10.el6.x86_64 8/19 Installing : 1:net-snmp-libs-5.5-37.el6_2.1.x86_64 9/19 Installing : lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 10/19 Installing : libgfortran-4.4.6-3.el6.x86_64 11/19 Installing : 1:tcl-8.5.7-6.el6.x86_64 12/19 Installing : environment-modules-3.2.7b-6.el6.x86_64 13/19 Installing : numactl-2.0.3-9.el6.x86_64 14/19 Installing : plpa-libs-1.3.2-2.1.el6.x86_64 15/19 Installing : compat-openmpi-1.4.3-1.el6.x86_64 16/19 Installing : lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 17/19 Cleanup : e2fsprogs-1.41.12-11.el6.x86_64 18/19 Cleanup : e2fsprogs-libs-1.41.12-11.el6.x86_64 19/19 Installed: lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b Dependency Installed: compat-openmpi.x86_64 0:1.4.3-1.el6 environment-modules.x86_64 0:3.2.7b-6.el6 libgfortran.x86_64 0:4.4.6-3.el6 libibcm.x86_64 0:1.0.5-2.el6 libibverbs.x86_64 0:1.1.5-3.el6 librdmacm.x86_64 0:1.0.14.1-3.el6 lm_sensors-libs.x86_64 0:3.1.1-10.el6 net-snmp-libs.x86_64 1:5.5-37.el6_2.1 numactl.x86_64 0:2.0.3-9.el6 plpa-libs.x86_64 0:1.3.2-2.1.el6 tcl.x86_64 1:8.5.7-6.el6 Updated: e2fsprogs.x86_64 0:1.41.90.wc4-7.el6 e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6 Complete!

Reboot

1. reboot

Verify that installation is compelete:

1. uname -a

   Code Block
   [root@rhel6_vanilla ~]# uname -a Linux rhel6_vanilla 2.6.32-220.el6_lustre.x86_64 #1 SMP Wed Jan 25 12:21:36 PST 2012 x86_64 x86_64 x86_64 GNU/Linux

2. Create a local Lustre filesystem with: llmount.sh

   Code Block

   [root@rhel6_vanilla ~]# /usr/lib64/lustre/tests/llmount.sh Stopping clients: rhel6_vanilla /mnt/lustre (opts:) Stopping clients: rhel6_vanilla /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6_vanilla environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel6_vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required [root@rhel6_vanilla ~]# mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl) /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1) /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2) rhel6_vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)

Testing

Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem