Lustre server machines are simple to setup with CentOS 6.2. Pre-built kernel images are available from Whamclouds build server. A number of different kernels and tools are continually built and tested. For this Walk-thru the lustre-master build will be used.
Whamcloud Lustre GA
Whamcloud produce Maintenance Releases and Feature Releases. This walk through covers running Lustre from the Master release. The Master release is subject to continuous integration testing but may have known or unknown issues. If you are evaluating Lustre for your production environment, you are advised to contact Whamcloud and substitute http://downloads.whamcloud.com/public/ for the RPMs you need for the walk-thru below.
Prerequisite.
- A x86_64 machine with CentOS 6.2 installed. The machine has the hostname
rhel6_vanilla.
Procedure
Install Lustre patched kernel.
- From lustre-master build, download the correct kernel RPMs for your platform
kernelkernel-firmware
- Install these RPMs with
yum localinstall ./kernel*[root@rhel6_vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./kernel-2.6.32-220.el6_lustre.x86_64.rpm: kernel-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-220.el6.x86_64 base | 3.7 kB 00:00 base/primary_db | 4.5 MB 00:02 extras | 3.5 kB 00:00 extras/primary_db | 6.3 kB 00:00 updates | 3.5 kB 00:00 updates/primary_db | 1.2 MB 00:00 Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package Arch Version Repository Size ============================================================================================================= Installing: kernel x86_64 2.6.32-220.el6_lustre /kernel-2.6.32-220.el6_lustre.x86_64 112 M Updating for dependencies: kernel-firmware noarch 2.6.32-220.4.1.el6 updates 6.2 M Transaction Summary ============================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 118 M Total download size: 6.2 M Is this ok [y/N]: y Downloading Packages: kernel-firmware-2.6.32-220.4.1.el6.noarch.rpm | 6.2 MB 00:01 warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-220.4.1.el6.noarch 1/3 Installing : kernel-2.6.32-220.el6_lustre.x86_64 2/3 Cleanup : kernel-firmware-2.6.32-220.el6.noarch 3/3 Installed: kernel.x86_64 0:2.6.32-220.el6_lustre Dependency Updated: kernel-firmware.noarch 0:2.6.32-220.4.1.el6 Complete! [root@rhel6_vanilla ~]#
Download e2fsprogs and Lustre server RPMSs.
- Download the following e2fsprogs RPMs from http://build.whamcloud.com/job/e2fsprogs-master
e2fsprogse2fsprogs-libs
- Download the following Lustre server RPMs from lustre-master build.
lustrelustre-ldiskfslustre-moduleslustre-tests
Install Lustre.
yum localinstall ./lustre-* ./e2fsprogs-*should do it:[root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Loading mirror speeds from cached hostfile * base: ftp.usf.edu * extras: mirror.raystedman.net * updates: mirror.raystedman.net Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 Marking ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed Examining ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-1.41.90.wc4-7.el6.x86_64 Marking ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-11.el6.x86_64 Examining ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 Marking ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-11.el6.x86_64 Resolving Dependencies --> Running transaction check ---> Package e2fsprogs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs.x86_64 0:1.41.90.wc4-7.el6 will be an update ---> Package e2fsprogs-libs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6 will be an update ---> Package lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed --> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed ---> Package lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed ---> Package lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed --> Processing Dependency: libmpi.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libopen-pal.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Processing Dependency: libopen-rte.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 --> Running transaction check ---> Package compat-openmpi.x86_64 0:1.4.3-1.el6 will be installed --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: environment-modules for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibcm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libplpa.so.0()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libgfortran.so.3()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 --> Processing Dependency: libnuma.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64 ---> Package net-snmp-libs.x86_64 1:5.5-37.el6_2.1 will be installed --> Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-37.el6_2.1.x86_64 --> Running transaction check ---> Package environment-modules.x86_64 0:3.2.7b-6.el6 will be installed --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.7b-6.el6.x86_64 ---> Package libgfortran.x86_64 0:4.4.6-3.el6 will be installed ---> Package libibcm.x86_64 0:1.0.5-2.el6 will be installed ---> Package libibverbs.x86_64 0:1.1.5-3.el6 will be installed ---> Package librdmacm.x86_64 0:1.0.14.1-3.el6 will be installed ---> Package lm_sensors-libs.x86_64 0:3.1.1-10.el6 will be installed ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed ---> Package plpa-libs.x86_64 0:1.3.2-2.1.el6 will be installed --> Running transaction check ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package Arch Version Repository Size ============================================================================================================= Installing: lustre x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 24 M lustre-ldiskfs x86_64 3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 8.8 M lustre-modules x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 139 M lustre-tests x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 12 M Updating: e2fsprogs x86_64 1.41.90.wc4-7.el6 /e2fsprogs-1.41.90.wc4-7.el6.x86_64 2.3 M e2fsprogs-libs x86_64 1.41.90.wc4-7.el6 /e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 308 k Installing for dependencies: compat-openmpi x86_64 1.4.3-1.el6 base 1.3 M environment-modules x86_64 3.2.7b-6.el6 base 96 k libgfortran x86_64 4.4.6-3.el6 base 263 k libibcm x86_64 1.0.5-2.el6 base 19 k libibverbs x86_64 1.1.5-3.el6 base 44 k librdmacm x86_64 1.0.14.1-3.el6 base 27 k lm_sensors-libs x86_64 3.1.1-10.el6 base 37 k net-snmp-libs x86_64 1:5.5-37.el6_2.1 updates 1.5 M numactl x86_64 2.0.3-9.el6 base 55 k plpa-libs x86_64 1.3.2-2.1.el6 base 25 k tcl x86_64 1:8.5.7-6.el6 base 1.9 M Transaction Summary ============================================================================================================= Install 15 Package(s) Upgrade 2 Package(s) Total size: 192 M Total download size: 5.3 M Is this ok [y/N]: y Downloading Packages: (1/11): compat-openmpi-1.4.3-1.el6.x86_64.rpm | 1.3 MB 00:00 (2/11): environment-modules-3.2.7b-6.el6.x86_64.rpm | 96 kB 00:00 (3/11): libgfortran-4.4.6-3.el6.x86_64.rpm | 263 kB 00:00 (4/11): libibcm-1.0.5-2.el6.x86_64.rpm | 19 kB 00:00 (5/11): libibverbs-1.1.5-3.el6.x86_64.rpm | 44 kB 00:00 (6/11): librdmacm-1.0.14.1-3.el6.x86_64.rpm | 27 kB 00:00 (7/11): lm_sensors-libs-3.1.1-10.el6.x86_64.rpm | 37 kB 00:00 (8/11): net-snmp-libs-5.5-37.el6_2.1.x86_64.rpm | 1.5 MB 00:03 (9/11): numactl-2.0.3-9.el6.x86_64.rpm | 55 kB 00:00 (10/11): plpa-libs-1.3.2-2.1.el6.x86_64.rpm | 25 kB 00:00 (11/11): tcl-8.5.7-6.el6.x86_64.rpm | 1.9 MB 00:00 ------------------------------------------------------------------------------------------------------------- Total 1.0 MB/s | 5.3 MB 00:05 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : libibverbs-1.1.5-3.el6.x86_64 1/19 Updating : e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64 2/19 Updating : e2fsprogs-1.41.90.wc4-7.el6.x86_64 3/19 Installing : lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 4/19 Installing : lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 5/19 Installing : libibcm-1.0.5-2.el6.x86_64 6/19 Installing : librdmacm-1.0.14.1-3.el6.x86_64 7/19 Installing : lm_sensors-libs-3.1.1-10.el6.x86_64 8/19 Installing : 1:net-snmp-libs-5.5-37.el6_2.1.x86_64 9/19 Installing : lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 10/19 Installing : libgfortran-4.4.6-3.el6.x86_64 11/19 Installing : 1:tcl-8.5.7-6.el6.x86_64 12/19 Installing : environment-modules-3.2.7b-6.el6.x86_64 13/19 Installing : numactl-2.0.3-9.el6.x86_64 14/19 Installing : plpa-libs-1.3.2-2.1.el6.x86_64 15/19 Installing : compat-openmpi-1.4.3-1.el6.x86_64 16/19 Installing : lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64 17/19 Cleanup : e2fsprogs-1.41.12-11.el6.x86_64 18/19 Cleanup : e2fsprogs-libs-1.41.12-11.el6.x86_64 19/19 Installed: lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b Dependency Installed: compat-openmpi.x86_64 0:1.4.3-1.el6 environment-modules.x86_64 0:3.2.7b-6.el6 libgfortran.x86_64 0:4.4.6-3.el6 libibcm.x86_64 0:1.0.5-2.el6 libibverbs.x86_64 0:1.1.5-3.el6 librdmacm.x86_64 0:1.0.14.1-3.el6 lm_sensors-libs.x86_64 0:3.1.1-10.el6 net-snmp-libs.x86_64 1:5.5-37.el6_2.1 numactl.x86_64 0:2.0.3-9.el6 plpa-libs.x86_64 0:1.3.2-2.1.el6 tcl.x86_64 1:8.5.7-6.el6 Updated: e2fsprogs.x86_64 0:1.41.90.wc4-7.el6 e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6 Complete!
Reboot
reboot
Verify that installation is complete:
uname -a[root@rhel6_vanilla ~]# uname -a Linux rhel6_vanilla 2.6.32-220.el6_lustre.x86_64 #1 SMP Wed Jan 25 12:21:36 PST 2012 x86_64 x86_64 x86_64 GNU/Linux
- Create a local Lustre filesystem with:
llmount.sh[root@rhel6_vanilla ~]# /usr/lib64/lustre/tests/llmount.sh Stopping clients: rhel6_vanilla /mnt/lustre (opts:) Stopping clients: rhel6_vanilla /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6_vanilla environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel6_vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required [root@rhel6_vanilla ~]# mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl) /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1) /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2) rhel6_vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)
Testing
Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem
Overview
Content Tools