Page History

Lustre server machines are simple to setup with CentOS 6. 2. Pre-built kernel images are available from Whamclouds Whamcloud's build server. A number of different kernels and tools are continually built and tested. For this Walk-thru the lustre-master build will be used.

Prerequisite.

• A x86_64 machine with CentOS 6 .2 installed. The machine has the hostname rhel6_vanilla rhel6vanilla.

Procedure

Install Lustre

...

directly with yum

1. Include the following entries into /etc/yum.repos.d/lustre.repo as follows:

   Code Block
   title /etc/repos.d/lustre.repo
   [hpddLustreserver] name=CentOS-$releasever - Lustre baseurl=http://build.whamcloud.com/job/lustre-master/arch=x86_64%2Cbuild_type=server%2Cdistro=el6%2Cib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/ gpgcheck=0 [e2fsprogs] name=CentOS-$releasever - Ldiskfs baseurl=http://build.whamcloud.com/job/e2fsprogs-master/arch=x86_64%2Cdistro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/ gpgcheck=0 [hpddLustreclient] name=CentOS-$releasever - Lustre baseurl=http://build.whamcloud.com/job/lustre-master/arch=x86_64%2Cbuild_type=client%2Cdistro=el6%2Cib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/ gpgcheck=0

2. run yum install lustre-tests

3. provided the step above completes successfully, you can skip down to the section:Disable SELinux for Lustre patched kernel.
4. If you experience dependency issues with yum, the best known method is to downgrade the current kernel to the version that is needed and try again.
5. If you are installing on RHEL and don't have Red Hat Network enabled, you will need to use the DVD ISO as a repo to satisfy unmet dependencies. This is covered on http://serverfault.com/questions/156458/

Install Lustre patched kernel.

If you choose not to use the yum install method above, you can install Lustre as follows:

1. From lustre-master build, download the correct kernel RPMs for your platform
   • kernel
   • kernel-firmware

2. Install these RPMs with yum localinstall ./kernel*

   Code Block

   [root@rhel6vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, presto Setting up Local Package Process Examining ./kernel-2.6.32-279.19.1.el6_lustre.x86_64.rpm: kernel-2.6.32-279.19.1.el6_lustre.x86_64 Marking ./kernel-2.6.32-279.19.1.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-279.el6.x86_64 Determining fastest mirrors * base: ftp.byfly.by * extras: ftp.byfly.by * updates: ftp.byfly.by Examining ./kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-279.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-279.19.1.el6_lustre will be installed .x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-279.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-358.6.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================= Package

3. From lustre-master build, download the correct kernel RPMs for your platform
   • kernel
   • kernel-firmware
4. Install these RPMs with yum localinstall ./kernel*

   Code Block

   [root@rhel6_vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, security Setting up Local Package Process Examining ./kernel-2.6.32-220.el6_lustre.x86_64.rpm: kernel-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-220.el6.x86_64 base | 3.7 kB 00:00 base/primary_db | 4.5 MB 00:02 extras Arch Version Repository | 3.5 kB 00:00 extras/primary_db | 6.3 kB 00:00 updates Size ======================================================================================================================= Installing: kernel x86_64 2.6.32-279.19.1.el6_lustre /kernel-2.6.32-279.19.1.el6_lustre.x86_64 113 M Updating | 3.5 kB 00:00 updates/primary_db for dependencies: kernel-firmware noarch 2.6.32-358.6.1.el6 updates | 1.2 MB 00:00 Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================= Package11 M Transaction Summary ======================================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 124 M Total download size: 11 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata updates/prestodelta | 339 kB 00:00 Processing delta metadata Download delta size: 3.4 M kernel-firmware-2.6.32-279.el6_2.6.32-358.6.1.el6.noarch.drpm | 3.4 MB 00:01 Finishing rebuild of rpms, from deltarpms <delta rebuild> Arch Version Repository | 11 MB 00:03 Size ============================================================================================================= Installing: kernel x86_64 Presto reduced the update size by 69% (from 11 M to 3.4 M). Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-220.el6_lustre358.6.1.el6.noarch /kernel-2.6.32-220.el6_lustre.x86_64 112 M Updating for dependencies: kernel-firmware noarch 1/3 Installing : kernel-2.6.32-220279.419.1.el6_lustre.x86_64 updates 6.2 M Transaction Summary ============================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 118 M Total download size: 6.2 M Is this ok [y/N]: y Downloading Packages: 2/3 Cleanup : kernel-firmware-2.6.32-220279.4.1.el6.noarch.rpm | 6.2 MB 00:01 warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Importing GPG key 0xC105B9DE: Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org> Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6 Is this ok [y/N]: y Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-220.4.1.el6.noarch 3/3 Verifying : kernel-2.6.32-279.19.1.el6_lustre.x86_64 1/3 Verifying Installing : kernel-firmware-2.6.32-220358.6.1.el6_lustre.x86_64.noarch 2/3 CleanupVerifying : kernel-firmware-2.6.32-220279.el6.noarch 3/3 Installed: kernel.x86_64 0:2.6.32-220279.19.1.el6_lustre Dependency Updated: kernel-firmware.noarch 0:2.6.32-220.4.1.el6 Complete! [root@rhel6_vanilla ~]#

Download e2fsprogs and Lustre server RPMSs.

1. Download the following e2fsprogs (and dependencies) RPMs from http://build.whamcloud.com/job/e2fsprogs-master
   • e2fsprogs
   • e2fsprogs-libs
   • libcom_err
   • libss
2. Download the following Lustre server RPMs from lustre-master build.
   • lustre
   • lustre-ldiskfs
   • lustre-modules
   • lustre-tests

Install Lustre.

1. Dependency Updated: kernel-firmware.noarch 0:2.6.32-358.6.1.el6 Complete! [root@rhel6vanilla ~]#

Download e2fsprogs and Lustre server RPMSs.

1. Download the following e2fsprogs (and dependencies) RPMs from http://build.whamcloud.com/job/e2fsprogs-master
   • e2fsprogs
   • e2fsprogs-libs
   • libss
   • libcom_err
2. Download the following Lustre server RPMs from lustre-master build.
   • lustre
   • lustre-ldiskfs
   • lustre-modules
   • lustre-tests

Install Lustre.

1. yum localinstall ./lustre-* ./e2fsprogs-*  ./libss-* ./libcom* should do it:

   Code Block
   [root@rhel6vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* ./libss-* ./libcom* Loaded plugins: fastestmirror, presto Setting up Local Package Process Examining ./lustre-2.1.5-2.6.32_279.19

   yum localinstall ./lustre-* ./e2fsprogs-* ./lib* should do it:

   Code Block

   [root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* ./lib* Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Local Package Process Examining ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Loading mirror speeds from cached hostfile  * base: centos-distro.cavecreek.net  * extras: mirror.raystedman.net  * updates: centosv.centos.org Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Examining ./lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-modules-2.2.54.rpm: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-2.1.5-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Loading mirror speeds from cached hostfile * base: ftp.byfly.by * extras: ftp.byfly.by * updates: ftp.byfly.by Examining ./lustre-testsldiskfs-23.23.540-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-testsldiskfs-23.23.540-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5.x86_64 Marking ./lustre-testsldiskfs-23.23.540-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed Examining ./e2fsprogslustre-modules-2.1.42.3.wc1-7.el6.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: e2fsprogslustre-modules-2.1.42.3.wc1-7.el65-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./e2fsprogslustre-modules-2.1.42.3.wc1-7.el65-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm asto anbe update to e2fsprogs-1.41.12-11.el6.x86_64 installed Examining ./lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm to be installed Examining ./e2fsprogs-libs-1.42.37.wc1-7.el6.x86_64.rpm: e2fsprogs-libs-1.42.37.wc1-7.el6.x86_64 Marking ./e2fsprogs-libs-1.42.37.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-1112.el6.x86_64 Examining ./libcom_erre2fsprogs-libs-1.42.37.wc1-7.el6.x86_64.rpm: libcom_erre2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 Examining ./libss-1.42.7.wc1-7.el6.x86_64.rpm: libss-1.42.37.wc1-7.el6.x86_64 Marking ./libcom_errlibss-1.42.37.wc1-7.el6.x86_64.rpm as an update to libcom_errlibss-1.41.12-1112.el6.x86_64 Examining ./libsslibcom_err-1.42.37.wc1-7.el6.x86_64.rpm: libsslibcom_err-1.42.37.wc1-7.el6.x86_64 Marking ./libsslibcom_err-1.42.37.wc1-7.el6.x86_64.rpm as an update to libsslibcom_err-1.41.12-1112.el6.x86_64 Resolving Dependencies --> Running transaction check ---> Package e2fsprogs.x86_64 0:1.41.12-1112.el6 will be updated ---> Package e2fsprogs.x86_64 0:1.42.37.wc1-7.el6 will be an update ---> Package e2fsprogs-libs.x86_64 0:1.41.12-1112.el6 will be updated ---> Package e2fsprogs-libs.x86_64 0:1.42.37.wc1-7.el6 will be an update ---> Package libcom_err.x86_64 0:1.41.12-1112.el6 will be updated ---> Package libcom_err.x86_64 0:1.42.37.wc1-7.el6 will be an update ---> Package libss.x86_64 0:1.41.12-1112.el6 will be updated ---> Package libss.x86_64 0:1.42.37.wc1-7.el6 will be an update ---> Package lustre.x86_64 0:2.21.545-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5 will be installed ---> Package lustre-ldiskfs.x86_64 0:3.3.0Processing Dependency: /usr/bin/perl for package: lustre-2.1.5-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5 will be installed -.x86_64 --> Package Processing Dependency: perl(File::Path) for package: lustre-modules.x86_64 0:2.21.545-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5 will be installed -.x86_64 --> Package Processing Dependency: perl(File::Path) for package: lustre-tests.x86_64 0:2.21.545-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5 will be installed --> Processing.x86_64 --> Processing Dependency: libmpi.so.1()(64bitperl(Getopt::Long) for package: lustre-tests-2.21.545-2.6.32_220279.1719.1.el6_lustre.x86_64_g4bd9bf5.x86_64 --> Processing Running transaction check ---> Package openmpi.x86_64 0:1.5.3-3.el6 will be installedDependency: perl(Getopt::Std) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bitperl(POSIX) for package: openmpilustre-2.1.5.3-3.el62.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: librdmacmlibnetsnmp.so.120(RDMACM_1.0)(64bit) for package: openmpilustre-2.1.5.3-3.el6-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnumalibnetsnmpagent.so.120(libnuma_1.2)(64bit) for package: openmpilustre-2.1.5.3-3.el6-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libibverbslibnetsnmphelpers.so.120(IBVERBS_1.1)(64bit) for package: openmpilustre-2.1.5.3-3.el6-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libibverbslibnetsnmpmibs.so.120(IBVERBS_1.0)(64bit) for package: openmpilustre-2.1.5.3-3.el6-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 ---> Processing Dependency: environment-modules for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libesmtp.so.5()(64bit) for package: openmpi-1.5.3-3.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 will be installed ---> Package lustre-modules.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 will be installed ---> Package lustre-tests.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 will be installed --> Processing Dependency: libnumalibmpi.so.1()(64bit) for package: openmpilustre-tests-2.1.5.3-32.el66.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: openmpi-1.5.3-3.el632_279.19.1.el6_lustre.x86_64.x86_64 --> Running transaction check ---> Package environmentnet-snmp-moduleslibs.x86_64 01:35.2.7b-6.el65-44.el6_4.1 will be installed --> Processing Dependency: libtcl8libsensors.5so.so4()(64bit) for package: environment-modules-3.2.7b-6.el61:net-snmp-libs-5.5-44.el6_4.1.x86_64 ---> Processing PackageDependency: libesmtp.x86_64 0:1.0.4-15.el6 will be installedlibperl.so()(64bit) for package: 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 ---> Package libibverbsopenmpi.x86_64 0:1.15.54-31.el6 will be installed ---> PackageProcessing Dependency: librdmacm.so.x86_64 0:1.0.14.1-3.el6 will be installed ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed1(RDMACM_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> RunningProcessing transaction check ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installedDependency: libnuma.so.1(libnuma_1.1)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> FinishedProcessing Dependency Resolution Dependencies Resolved =============================================================================================================================  Package             Arch   Version           Repository                                                                Size ============================================================================================================================= Installing:  lustre              x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                               /lustre-2.2.54-2.6.32_220.17: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1(OMP_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1(GOMP_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: environment-modules for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libpsm_infinipath.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libnuma.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libltdl.so.7()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgfortran.so.3()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libesmtp.so.5()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 ---> Package perl.x86_64 4:5.10.1-130.el6_4 will be installed --> Processing Dependency: perl(version) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Processing Dependency: perl(Pod::Simple) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Processing Dependency: perl(Module::Pluggable) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Running transaction check ---> Package environment-modules.x86_64 0:3.2.9c-4.el6 will be installed --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.9c-4.el6.x86_64 --> Processing Dependency: libX11.so.6()(64bit) for package: environment-modules-3.2.9c-4.el6.x86_64 ---> Package infinipath-psm.x86_64 0:3.0.1-115.1015_open.1.1.el6_4 will be installed ---> Package libesmtp.x86_64 0:1.0.4-15.el6 will be installed ---> Package libgfortran.x86_64 0:4.4.7-3.el6 will be installed ---> Package libgomp.x86_64 0:4.4.7-3.el6 will be installed ---> Package libibverbs.x86_64 0:1.1.6-5.el6 will be installed ---> Package librdmacm.x86_64 0:1.0.17-0.git4b5c1aa.el6 will be installed ---> Package libtool-ltdl.x86_64 0:2.2.6-15.5.el6 will be installed ---> Package lm_sensors-libs.x86_64 0:3.1.1-17.el6 will be installed ---> Package numactl.x86_64 0:2.0.7-6.el6 will be installed ---> Package perl-Module-Pluggable.x86_64 1:3.90-130.el6_4 will be installed ---> Package perl-Pod-Simple.x86_64 1:3.13-130.el6_4 will be installed --> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 ---> Package perl-libs.x86_64 4:5.10.1-130.el6_4 will be installed ---> Package perl-version.x86_64 3:0.77-130.el6_4 will be installed --> Running transaction check ---> Package libX11.x86_64 0:1.5.0-4.el6 will be installed --> Processing Dependency: libX11-common = 1.5.0-4.el6 for package: libX11-1.5.0-4.el6.x86_64 --> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.5.0-4.el6.x86_64 ---> Package perl-Pod-Escapes.x86_64 1:1.04-130.el6_4 will be installed ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed --> Running transaction check ---> Package libX11-common.noarch 0:1.5.0-4.el6 will be installed ---> Package libxcb.x86_64 0:1.8.1-1.el6 will be installed --> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.8.1-1.el6.x86_64 --> Running transaction check ---> Package libXau.x86_64 0:1.0.6-4.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================= Package Arch Version Repository Size ======================================================================================================================= Installing: lustre x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 26 M lustre-ldiskfs x86_64 3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 8.4 M lustre-modules x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 144 M lustre-tests x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Updating: e2fsprogs x86_64 1.42.7.wc1-7.el6 /e2fsprogs-1.42.7.wc1-7.el6.x86_64 2.6 M e2fsprogs-libs x86_64 1.42.7.wc1-7.el6 /e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 349 k libcom_err x86_64 1.42.7.wc1-7.el6 /libcom_err-1.42.7.wc1-7.el6.x86_64 56 k libss x86_64 1.42.7.wc1-7.el6 /libss-1.42.7.wc1-7.el6.x86_64 67 k Installing for dependencies: environment-modules x86_64 3.2.9c-4.el6 base 98 k infinipath-psm x86_64 3.0.1-115.1015_open.1.1.el6_4 updates 160 k libX11 x86_64 1.5.0-4.el6 base 584 k libX11-common noarch 1.5.0-4.el6 base 192 k libXau x86_64 1.0.6-4.el6 base 24 k libesmtp x86_64 1.0.4-15.el6 base 57 k libgfortran x86_64 4.4.7-3.el6 base 265 k libgomp x86_64 4.4.7-3.el6 base 118 k libibverbs x86_64 1.1.6-5.el6 base 45 k librdmacm x86_64 1.0.17-0.git4b5c1aa.el6 base 47 k libtool-ltdl x86_64 2.2.6-15.5.el6 base 44 k libxcb x86_64 1.8.1-1.el6 base 110 k lm_sensors-libs x86_64 3.1.1-17.el6 base 38 k net-snmp-libs x86_64 1:5.5-44.el6_4.1 updates 1.5 M numactl x86_64 2.0.7-6.el6 base 69 k openmpi x86_64 1.5.4-1.el6 base 2.1 M perl x86_64 4:5.10.1-130.el6_4 updates perl-Module-Pluggable x86_64 1:3.90-130.el6_4 updates 38 k perl-Pod-Escapes x86_64 1:1.04-130.el6_4 updates 31 k perl-Pod-Simple x86_64 1:3.13-130.el6_4 updates 210 k perl-libs x86_64 4:5.10.1-130.el6_4 updates 576 k perl-version x86_64 3:0.77-130.el6_4 updates 50 k tcl x86_64 1:8.5.7-6.el6 base 1.9 M Transaction Summary ======================================================================================================================= Install 27 Package(s) Upgrade 4 Package(s) Total size: 213 M Total download size: 18 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 18 M (1/23): environment-modules-3.2.9c-4.el6.x86_64.rpm | 98 kB 00:00 (2/23): infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64.rpm | 160 kB 00:00 (3/23): libX11-1.5.0-4.el6.x86_64.rpm | 584 kB 00:00 (4/23): libX11-common-1.5.0-4.el6.noarch.rpm | 192 kB (5/23): libXau-1.0.6-4.el6.x86_64.rpm | 24 kB 00:00 (6/23): libesmtp-1.0.4-15.el6.x86_64.rpm | 57 kB 00:00 (7/23): libgfortran-4.4.7-3.el6.x86_64.rpm | 265 kB 00:00 (8/23): libgomp-4.4.7-3.el6.x86_64.rpm | 118 kB 00:00 (9/23): libibverbs-1.1.6-5.el6.x86_64.rpm | 45 kB 00:00 (10/23): librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64.rpm | 47 kB 00:00 (11/23): libtool-ltdl-2.2.6-15.5.el6.x86_64.rpm | 44 kB 00:00 (12/23): libxcb-1.8.1-1.el6.x86_64.rpm | 110 kB 00:00 (13/23): lm_sensors-libs-3.1.1-17.el6.x86_64.rpm | 38 kB 00:00 (14/23): net-snmp-libs-5.5-44.el6_4.1.x86_64.rpm | 1.5 MB 00:01 (15/23): numactl-2.0.7-6.el6.x86_64.rpm | 69 kB 00:00 (16/23): openmpi-1.5.4-1.el6.x86_64.rpm | 2.1 MB 00:01 (17/23): perl-5.10.1-130.el6_4.x86_64.rpm | 10 MB 00:08 (18/23): perl-Module-Pluggable-3.90-130.el6_4.x86_64.rpm | 38 kB 00:00 (19/23): perl-Pod-Escapes-1.04-130.el6_4.x86_64.rpm | 31 kB 00:00 (20/23): perl-Pod-Simple-3.13-130.el6_4.x86_64.rpm | 210 kB 00:00 (21/23): perl-libs-5.10.1-130.el6_4.x86_64.rpm | 576 kB 00:00 (22/23): perl-version-0.77-130.el6_4.x86_64.rpm | 50 kB 00:00 (23/23): tcl-8.5.7-6.el6.x86_64.rpm | 1.9 MB 00:01 ----------------------------------------------------------------------------------------------------------------------- Total 1.2 MB/s | 18 MB Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : 1:perl-Pod-Escapes-1.04-130.el6_4.x86_64 1/35 Installing : 4:perl-libs-5.10.1-130.el6_4.x86_64 2/35 Installing : 1:perl-Module-Pluggable-3.90-130.el6_4.x86_64 3/35 Installing : 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 4/35 Installing : 3:perl-version-0.77-130.el6_4.x86_64 5/35 Installing : 4:perl-5.10.1-130.el6_4.x86_64 6/35 Updating : libcom_err-1.42.7.wc1-7.el6.x86_64 7/35 Updating : e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 8/35 Installing : libibverbs-1.1.6-5.el6.x86_64 9/35 Installing : librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64 10/35 Updating : libss-1.42.7.wc1-7.el6.x86_64 11/35 Updating : e2fsprogs-1.42.7.wc1-7.el6.x86_64 12/35 Installing : lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 13/35 Installing : lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 14/35 Installing : libgomp-4.4.7-3.el6.x86_64 15/35 Installing : lm_sensors-libs-3.1.1-17.el6.x86_64 16/35 Installing : 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 17/35 Installing : lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Installing : libXau-1.0.6-4.el6.x86_64 19/35 Installing : libxcb-1.8.1-1.el6.x86_64 20/35 Installing : libX11-common-1.5.0-4.el6.noarch 21/35 Installing : libX11-1.5.0-4.el6.x86_64 22/35 Installing : libgfortran-4.4.7-3.el6.x86_64 23/35 Installing : libesmtp-1.0.4-15.el6.x86_64 24/35 Installing : numactl-2.0.7-6.el6.x86_64 25/35 Installing : infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64 26/35 Installing : 1:tcl-8.5.7-6.el6.x86_64 27/35 Installing : environment-modules-3.2.9c-4.el6.x86_64 28/35 Installing : libtool-ltdl-2.2.6-15.5.el6.x86_64 29/35 Installing : openmpi-1.5.4-1.el6.x86_64 30/35 Installing : lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 31/35 Cleanup : e2fsprogs-1.41.12-12.el6.x86_64 32/35 Cleanup : e2fsprogs-libs-1.41.12-12.el6.x86_64 33/35 Cleanup : libss-1.41.12-12.el6.x86_64 34/35 Cleanup : libcom_err-1.41.12-12.el6.x86_64 35/35 Verifying : libtool-ltdl-2.2.6-15.5.el6.x86_64 1/35 Verifying : libss-1.42.7.wc1-7.el6.x86_64 2/35 Verifying : e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 3/35 Verifying : 1:tcl-8.5.7-6.el6.x86_64 4/35 Verifying : 4:perl-5.10.1-130.el6_4.x86_64 Verifying : lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 6/35 Verifying : libcom_err-1.42.7.wc1-7.el6.x86_64 7/35 Verifying : lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 8/35 Verifying : infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64 9/35 Verifying : environment-modules-3.2.9c-4.el6.x86_64 10/35 Verifying : libX11-1.5.0-4.el6.x86_64 11/35 Verifying : numactl-2.0.7-6.el6.x86_64 12/35 Verifying : lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 13/35 Verifying : 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 14/35 Verifying : libesmtp-1.0.4-15.el6.x86_64 15/35 Verifying : libgfortran-4.4.7-3.el6.x86_64 16/35 Verifying : openmpi-1.5.4-1.el6.x86_64 17/35 Verifying : libX11-common-1.5.0-4.el6.noarch 18/35 Verifying : 4:perl-libs-5.10.1-130.el6_4.x86_64 19/35 Verifying : libxcb-1.8.1-1.el6.x86_64 20/35 Verifying : libXau-1.0.6-4.el6.x86_64 21/35 Verifying : 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 22/35 Verifying : 1:perl-Module-Pluggable-3.90-130.el6_4.x86_64 23/35 Verifying : lm_sensors-libs-3.1.1-17.el6.x86_64 24/35 Verifying : 1:perl-Pod-Escapes-1.04-130.el6_4.x86_64 25/35 Verifying : libgomp-4.4.7-3.el6.x86_64 26/35 Verifying : lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64_g4bd9bf5.x86_64          26 M  lustre-ldiskfs      x86_64 3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                               /lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64  8.8 M  lustre-modules      x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                               /lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 154 M  lustre-tests        x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                               /lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64    13 M Updating:  e2fsprogs           x86_64 1.42.3.wc1-7.el6  /e2fsprogs-1.42.3.wc1-7.el6.x86_64                                       2.6 M  e2fsprogs-libs      x86_64 1.42.3.wc1-7.el6  /e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64                                  344 k  libcom_err          x86_64 1.42.3.wc1-7.el6  /libcom_err-1.42.3.wc1-7.el6.x86_64                                       56 k  libss               x86_64 1.42.3.wc1-7.el6  /libss-1.42.3.wc1-7.el6.x86_64                                            67 k Installing for dependencies:  environment-modules x86_64 3.2.7b-6.el6      base                                                                      96 k  libesmtp            x86_64 1.0.4-15.el6      base                                                                      57 k  libibverbs          x86_64 1.1.5-3.el6       base                                                                      44 k  librdmacm           x86_64 1.0.14.1-3.el6    base                                                                      27 k  numactl             x86_64 2.0.3-9.el6       base                                                                      55 k  openmpi             x86_64 1.5.3-3.el6       base                                                                     2.1 M  tcl                 x86_64 1:8.5.7-6.el6     base                                                                     1.9 M Transaction Summary ============================================================================================================================= Install      11 Package(s) Upgrade       4 Package(s) Total size: 210 M Total download size: 4.3 M Is this ok [y/N]: y Downloading Packages: (1/7): environment-modules-3.2.7b-6.el6.x86_64.rpm                                                    |  96 kB     00:00      (2/7): libesmtp-1.0.4-15.el6.x86_64.rpm                                                               |  57 kB     00:00      (3/7): libibverbs-1.1.5-3.el6.x86_64.rpm                                                              |  44 kB     00:00      (4/7): librdmacm-1.0.14.1-3.el6.x86_64.rpm                                                            |  27 kB     00:00      (5/7): numactl-2.0.3-9.el6.x86_64.rpm                                                                 |  55 kB     00:00      (6/7): openmpi-1.5.3-3.el6.x86_64.rpm                                                                 | 2.1 MB     00:01      (7/7): tcl-8.5.7-6.el6.x86_64.rpm                                                                     | 1.9 MB     00:01      ----------------------------------------------------------------------------------------------------------------------------- Total                                                                                        849 kB/s | 4.3 MB     00:05      Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction   Updating   : libcom_err-1.42.3.wc1-7.el6.x86_64                                                                       1/19   Updating   : e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64                                                                   2/19   Installing : libibverbs-1.1.5-3.el6.x86_64                                                                            3/19   Installing : librdmacm-1.0.14.1-3.el6.x86_64                                                                          4/19   Updating   : libss-1.42.3.wc1-7.el6.x86_64                                                                            5/19   Updating   : e2fsprogs-1.42.3.wc1-7.el6.x86_64                                                                        6/19   Installing : lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                   7/19   Installing : lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                  8/19   Installing : lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                          9/19   Installing : 1:tcl-8.5.7-6.el6.x86_64                                                                                10/19   Installing : environment-modules-3.2.7b-6.el6.x86_64                                                                 11/19   Installing : libesmtp-1.0.4-15.el6.x86_64                                                                            12/19   Installing : numactl-2.0.3-9.el6.x86_64                                                                              13/19   Installing : openmpi-1.5.3-3.el6.x86_64                                                                              14/19   Installing : lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                   15/19   Cleanup    : e2fsprogs-1.41.12-11.el6.x86_64                                                                         16/19   Cleanup    : e2fsprogs-libs-1.41.12-11.el6.x86_64                                                                    17/19   Cleanup    : libss-1.41.12-11.el6.x86_64                                                                             18/19   Cleanup    : libcom_err-1.41.12-11.el6.x86_64                                                                        19/19 Installed:   lustre.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                             lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                      lustre-modules.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                     lustre-tests.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                     Dependency Installed:   environment-modules.x86_64 0:3.2.7b-6.el6       libesmtp.x86_64 0:1.0.4-15.el6       libibverbs.x86_64 0:1.1.5-3.el6         librdmacm.x86_64 0:1.0.14.1-3.el6               numactl.x86_64 0:2.0.3-9.el6         openmpi.x86_64 0:1.5.3-3.el6           64 27/35 Verifying : 3:perl-version-0.77-130.el6_4.x86_64 Verifying : libibverbs-1.1.6-5.el6.x86_64 29/35 Verifying : e2fsprogs-1.42.7.wc1-7.el6.x86_64 30/35 Verifying : librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64 31/35 Verifying : libss-1.41.12-12.el6.x86_64 32/35 Verifying : e2fsprogs-1.41.12-12.el6.x86_64 33/35 Verifying : libcom_err-1.41.12-12.el6.x86_64 34/35 Verifying : e2fsprogs-libs-1.41.12-12.el6.x86_64 35/35 Installed: lustre.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 lustre-modules.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 lustre-tests.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 Dependency Installed: environment-modules.x86_64 0:3.2.9c-4.el6 infinipath-psm.x86_64 0:3.0.1-115.1015_open.1.1.el6_4 libX11.x86_64 0:1.5.0-4.el6 libX11-common.noarch 0:1.5.0-4.el6 libXau.x86_64 0:1.0.6-4.el6 libesmtp.x86_64 0:1.0.4-15.el6 libgfortran.x86_64 0:4.4.7-3.el6 libgomp.x86_64 0:4.4.7-3.el6 libibverbs.x86_64 0:1.1.6-5.el6 librdmacm.x86_64 0:1.0.17-0.git4b5c1aa.el6 libtool-ltdl.x86_64 0:2.2.6-15.5.el6 libxcb.x86_64 0:1.8.1-1.el6 lm_sensors-libs.x86_64 0:3.1.1-17.el6 net-snmp-libs.x86_64 1:5.5-44.el6_4.1 numactl.x86_64 0:2.0.7-6.el6 openmpi.x86_64 0:1.5.4-1.el6 perl.x86_64 4:5.10.1-130.el6_4 perl-Module-Pluggable.x86_64 1:3.90-130.el6_4 perl-Pod-Escapes.x86_64 1:1.04-130.el6_4 perl-Pod-Simple.x86_64 1:3.13-130.el6_4 perl-libs.x86_64 4:5.10.1-130.el6_4 perl-version.x86_64 3:0.77-130.el6_4 tcl.x86_64 1:8.5.7-6.el6                       el6 Updated:   e2fsprogs.x86_64 0:1.42.37.wc1-7.el6   el6 e2fsprogs-libs.x86_64 0:1.42.37.wc1-7.el6   el6 libcom_err.x86_64 0:1.42.37.wc1-7.el6 el6     libss.x86_64 0:1.42.37.wc1-7.el6     el6   Complete! [root@rhel6_vanillaroot@rhel6vanilla ~]#

Disable SELinux for Lustre patched kernel.

Lustre does not play well with SELinux (at least, the llmount.sh script fails when SELinux is enabled), so SELinux must be disabled. One option is to disable SELinux with GRUB. With your editor of choice, open /boot/grub/grub.conf. Append "the option selinux=0" to the end of the "kernel" line for the Lustre patched kernel entry and save. This will cause SELinux to be disabled after you reboot into the patched kernel.

Reboot.

1. reboot

Verify that installation is complete:

1. uname -a

   Code Block
   [root@rhel6_vanillaroot@rhel6vanilla ~]# uname -a Linux rhel6_vanillarhel6vanilla 2.6.32-220279.19.1.el6_lustre.x86_64 #1 SMP Wed JanMar 2520 1216:2137:3618 PSTPDT 20122013 x86_64 x86_64 x86_64 GNU/Linux

2. Create a local Lustre filesystem with: llmount.sh

   Code Block

   [root@rhel6_vanillaroot@rhel6vanilla ~]# /usr/lib64/lustre/tests/llmount.sh Stopping clients: rhel6_vanillarhel6vanilla /mnt/lustre (opts:) Stopping clients: rhel6_vanillarhel6vanilla /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6_vanillarhel6vanilla environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=102 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=102 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=102 Started lustre-OST0001 Starting client: rhel6_vanillarhel6vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcprhel6vanilla@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=102 Using TIMEOUT=20 disable quota as required [root@rhel6_vanillaroot@rhel6vanilla ~]# mount /dev/mapper/VolGroupvg_rhel6vanilla-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/sda1vda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl) /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1) /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2) rhel6_vanilla@tcprhel6vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)

Testing

Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem