Lustre server machines are simple to setup with CentOS 6.2. Pre-built kernel images are available from Whamclouds build server. A number of different kernels and tools are continually built and tested. For this Walk-thru the lustre-master build will be used.
|
Whamcloud produce Maintenance Releases and Feature Releases. This walk through covers running Lustre from the Master release. The Master release is subject to continuous integration testing but may have known or unknown issues. If you are evaluating Lustre for your production environment, you are advised to contact Whamcloud and substitute http://downloads.whamcloud.com/public/ for the RPMs you need for the walk-thru below. |
rhel6_vanilla.kernelkernel-firmwareyum localinstall ./kernel*
[root@rhel6_vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./kernel-2.6.32-220.el6_lustre.x86_64.rpm: kernel-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-220.el6.x86_64 base | 3.7 kB 00:00 base/primary_db | 4.5 MB 00:02 extras | 3.5 kB 00:00 extras/primary_db | 6.3 kB 00:00 updates | 3.5 kB 00:00 updates/primary_db | 1.2 MB 00:00 Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package Arch Version Repository Size ============================================================================================================= Installing: kernel x86_64 2.6.32-220.el6_lustre /kernel-2.6.32-220.el6_lustre.x86_64 112 M Updating for dependencies: kernel-firmware noarch 2.6.32-220.4.1.el6 updates 6.2 M Transaction Summary ============================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 118 M Total download size: 6.2 M Is this ok [y/N]: y Downloading Packages: kernel-firmware-2.6.32-220.4.1.el6.noarch.rpm | 6.2 MB 00:01 warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-220.4.1.el6.noarch 1/3 Installing : kernel-2.6.32-220.el6_lustre.x86_64 2/3 Cleanup : kernel-firmware-2.6.32-220.el6.noarch 3/3 Installed: kernel.x86_64 0:2.6.32-220.el6_lustre Dependency Updated: kernel-firmware.noarch 0:2.6.32-220.4.1.el6 Complete! [root@rhel6_vanilla ~]# |
e2fsprogse2fsprogs-libslibcom_errlibsslustrelustre-ldiskfslustre-moduleslustre-testsyum localinstall ./lustre-* ./e2fsprogs-* ./lib* should do it:
[root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* ./lib* Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Local Package Process Examining ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Loading mirror speeds from cached hostfile * base: centos-distro.cavecreek.net * extras: mirror.raystedman.net * updates: centosv.centos.org Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Examining ./lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Examining ./lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Examining ./e2fsprogs-1.42.3.wc1-7.el6.x86_64.rpm: e2fsprogs-1.42.3.wc1-7.el6.x86_64 Marking ./e2fsprogs-1.42.3.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-11.el6.x86_64 Examining ./e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64.rpm: e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64 Marking ./e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-11.el6.x86_64 Examining ./libcom_err-1.42.3.wc1-7.el6.x86_64.rpm: libcom_err-1.42.3.wc1-7.el6.x86_64 Marking ./libcom_err-1.42.3.wc1-7.el6.x86_64.rpm as an update to libcom_err-1.41.12-11.el6.x86_64 Examining ./libss-1.42.3.wc1-7.el6.x86_64.rpm: libss-1.42.3.wc1-7.el6.x86_64 Marking ./libss-1.42.3.wc1-7.el6.x86_64.rpm as an update to libss-1.41.12-11.el6.x86_64 Resolving Dependencies --> Running transaction check ---> Package e2fsprogs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs.x86_64 0:1.42.3.wc1-7.el6 will be an update ---> Package e2fsprogs-libs.x86_64 0:1.41.12-11.el6 will be updated ---> Package e2fsprogs-libs.x86_64 0:1.42.3.wc1-7.el6 will be an update ---> Package libcom_err.x86_64 0:1.41.12-11.el6 will be updated ---> Package libcom_err.x86_64 0:1.42.3.wc1-7.el6 will be an update ---> Package libss.x86_64 0:1.41.12-11.el6 will be updated ---> Package libss.x86_64 0:1.42.3.wc1-7.el6 will be an update ---> Package lustre.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed ---> Package lustre-modules.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed ---> Package lustre-tests.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed --> Processing Dependency: libmpi.so.1()(64bit) for package: lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 --> Running transaction check ---> Package openmpi.x86_64 0:1.5.3-3.el6 will be installed --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: environment-modules for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libesmtp.so.5()(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libnuma.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Running transaction check ---> Package environment-modules.x86_64 0:3.2.7b-6.el6 will be installed --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.7b-6.el6.x86_64 ---> Package libesmtp.x86_64 0:1.0.4-15.el6 will be installed ---> Package libibverbs.x86_64 0:1.1.5-3.el6 will be installed ---> Package librdmacm.x86_64 0:1.0.14.1-3.el6 will be installed ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed --> Running transaction check ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================= Package Arch Version Repository Size ============================================================================================================================= Installing: lustre x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 /lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 26 M lustre-ldiskfs x86_64 3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 /lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 8.8 M lustre-modules x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 /lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 154 M lustre-tests x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 /lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 13 M Updating: e2fsprogs x86_64 1.42.3.wc1-7.el6 /e2fsprogs-1.42.3.wc1-7.el6.x86_64 2.6 M e2fsprogs-libs x86_64 1.42.3.wc1-7.el6 /e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64 344 k libcom_err x86_64 1.42.3.wc1-7.el6 /libcom_err-1.42.3.wc1-7.el6.x86_64 56 k libss x86_64 1.42.3.wc1-7.el6 /libss-1.42.3.wc1-7.el6.x86_64 67 k Installing for dependencies: environment-modules x86_64 3.2.7b-6.el6 base 96 k libesmtp x86_64 1.0.4-15.el6 base 57 k libibverbs x86_64 1.1.5-3.el6 base 44 k librdmacm x86_64 1.0.14.1-3.el6 base 27 k numactl x86_64 2.0.3-9.el6 base 55 k openmpi x86_64 1.5.3-3.el6 base 2.1 M tcl x86_64 1:8.5.7-6.el6 base 1.9 M Transaction Summary ============================================================================================================================= Install 11 Package(s) Upgrade 4 Package(s) Total size: 210 M Total download size: 4.3 M Is this ok [y/N]: y Downloading Packages: (1/7): environment-modules-3.2.7b-6.el6.x86_64.rpm | 96 kB 00:00 (2/7): libesmtp-1.0.4-15.el6.x86_64.rpm | 57 kB 00:00 (3/7): libibverbs-1.1.5-3.el6.x86_64.rpm | 44 kB 00:00 (4/7): librdmacm-1.0.14.1-3.el6.x86_64.rpm | 27 kB 00:00 (5/7): numactl-2.0.3-9.el6.x86_64.rpm | 55 kB 00:00 (6/7): openmpi-1.5.3-3.el6.x86_64.rpm | 2.1 MB 00:01 (7/7): tcl-8.5.7-6.el6.x86_64.rpm | 1.9 MB 00:01 ----------------------------------------------------------------------------------------------------------------------------- Total 849 kB/s | 4.3 MB 00:05 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : libcom_err-1.42.3.wc1-7.el6.x86_64 1/19 Updating : e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64 2/19 Installing : libibverbs-1.1.5-3.el6.x86_64 3/19 Installing : librdmacm-1.0.14.1-3.el6.x86_64 4/19 Updating : libss-1.42.3.wc1-7.el6.x86_64 5/19 Updating : e2fsprogs-1.42.3.wc1-7.el6.x86_64 6/19 Installing : lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 7/19 Installing : lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 8/19 Installing : lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 9/19 Installing : 1:tcl-8.5.7-6.el6.x86_64 10/19 Installing : environment-modules-3.2.7b-6.el6.x86_64 11/19 Installing : libesmtp-1.0.4-15.el6.x86_64 12/19 Installing : numactl-2.0.3-9.el6.x86_64 13/19 Installing : openmpi-1.5.3-3.el6.x86_64 14/19 Installing : lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 15/19 Cleanup : e2fsprogs-1.41.12-11.el6.x86_64 16/19 Cleanup : e2fsprogs-libs-1.41.12-11.el6.x86_64 17/19 Cleanup : libss-1.41.12-11.el6.x86_64 18/19 Cleanup : libcom_err-1.41.12-11.el6.x86_64 19/19 Installed: lustre.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 lustre-modules.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 lustre-tests.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 Dependency Installed: environment-modules.x86_64 0:3.2.7b-6.el6 libesmtp.x86_64 0:1.0.4-15.el6 libibverbs.x86_64 0:1.1.5-3.el6 librdmacm.x86_64 0:1.0.14.1-3.el6 numactl.x86_64 0:2.0.3-9.el6 openmpi.x86_64 0:1.5.3-3.el6 tcl.x86_64 1:8.5.7-6.el6 Updated: e2fsprogs.x86_64 0:1.42.3.wc1-7.el6 e2fsprogs-libs.x86_64 0:1.42.3.wc1-7.el6 libcom_err.x86_64 0:1.42.3.wc1-7.el6 libss.x86_64 0:1.42.3.wc1-7.el6 Complete! [root@rhel6_vanilla ~]# |
Lustre doesn't seem to play well with SELinux (at least, the llmount.sh script fails when SELinux is enabled), so it must be disabled. With your editor of choice, open /boot/grub/grub.conf. Append "selinux=0" to the end of the "kernel" line for the Lustre patched kernel entry and save. This will cause SELinux to be disabled after you reboot into the patched kernel. Your grub.conf show now look something like this:
# grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/mapper/vg_rhel6vanilla-lv_root # initrd /initrd-[generic-]version.img #boot=/dev/sda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.32-220.17.1.el6_lustre.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-220.17.1.el6_lustre.x86_64 ro root=/dev/mapper/vg_rhel6vanilla-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_rhel6vanilla/lv_root rd_LVM_LV=vg_rhel6vanilla/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM selinux=0 initrd /initramfs-2.6.32-220.17.1.el6_lustre.x86_64.img title CentOS (2.6.32-220.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-220.el6.x86_64 ro root=/dev/mapper/vg_rhel6vanilla-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_rhel6vanilla/lv_root rd_LVM_LV=vg_rhel6vanilla/lv_swap rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM selinux=0 initrd /initramfs-2.6.32-220.el6.x86_64.img |
rebootuname -a
[root@rhel6_vanilla ~]# uname -a Linux rhel6_vanilla 2.6.32-220.el6_lustre.x86_64 #1 SMP Wed Jan 25 12:21:36 PST 2012 x86_64 x86_64 x86_64 GNU/Linux |
llmount.sh
[root@rhel6_vanilla ~]# /usr/lib64/lustre/tests/llmount.sh Stopping clients: rhel6_vanilla /mnt/lustre (opts:) Stopping clients: rhel6_vanilla /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6_vanilla environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel6_vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required [root@rhel6_vanilla ~]# mount /dev/mapper/VolGroup-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl) /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1) /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2) rhel6_vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock) |
Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem