Page History
Lustre server machines are simple to setup with CentOS 5.56. Pre-built kernel images are available from Whamclouds Whamcloud's build server. A number of different kernels and tools are continuasly continually built and tested. For this Walk-thru the centoslustre-master - build will be used.
Note | ||
---|---|---|
| ||
Whamcloud produce Maintenance Releases and Feature Releases. This walk through covers running Lustre from the Master release. The Master release is subject to continuous integration testing but may have known or unknown issues. If you are evaluating Lustre for your production environment, you are advised to contact Whamcloud and substitute http://downloads.whamcloud.com/public/ for the RPMs you need for the walk-thru below. |
Prerequisite.
- A x86_64 machine with CentOS 5.5 6 installed. This The machine has the hostname client-10
rhel6vanilla
.
Procedure
First step is to download and install a Lustre patched CentOS kernel. The subsequent step is to add in modules and tools necessary to build a Lustre filesystem.
Installing a patched kernel.
Install Lustre directly with yum
Include the following entries into
/etc/yum.repos.d/lustre.repo
as follows:
Identify the recent, Lustre patched, kernel.Code Block title /etc/repos.d/lustre.repo [hpddLustreserver] name=CentOS-$releasever - Lustre baseurl=http://build
This can be found by either navigaing from the centos-master-build page or by visiting a link the the [last successful build for x86_64| http://build
64,build.whamcloud.com/job/lustre-master/arch=x86_
server,distro=el5,ib64%2Cbuild_type=
/].server%2Cdistro=el6%2Cib_stack=inkernel
- Identify the most recent kernel RPM.
Recent kernel builds are of the formkernel-2.6.18-194.17.1.el5_lustre.<buildid>.x86_64.rpm
- Download this RPM to the machine client-10.
- Install the kernel.
Code Block [root@client-10 ~]# rpm -ivh ./kernel-2.6.18-194.17.1.el5_lustre.<buildid>.x86_64.rpm Preparing... ########################################### [100%] 1:kernel ########################################### [100%] [root@client-10 ~]#
/lastSuccessfulBuild/artifact/artifacts/ gpgcheck=0 [e2fsprogs] name=CentOS-$releasever - Ldiskfs baseurl=http://build.whamcloud.com/job/e2fsprogs-master/arch=x86_64%2Cdistro=el6/lastSuccessfulBuild/artifact/_topdir/RPMS/ gpgcheck=0 [hpddLustreclient] name=CentOS-$releasever - Lustre baseurl=http://build.whamcloud.com/job/lustre-master/arch=x86_64%2Cbuild_type=client%2Cdistro=el6%2Cib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/ gpgcheck=0
run
yum install lustre-tests
- provided the step above completes successfully, you can skip down to the section:Disable SELinux for Lustre patched kernel.
- If you experience dependency issues with yum, the best known method is to downgrade the current kernel to the version that is needed and try again.
- If you are installing on RHEL and don't have Red Hat Network enabled, you will need to use the DVD ISO as a repo to satisfy unmet dependencies. This is covered on http://serverfault.com/questions/156458/
Install Lustre patched kernel.
If you choose not to use the yum install method above, you can install Lustre as follows:
- From lustre-master build, download the correct kernel RPMs for your platform
kernel
kernel-firmware
Install these RPMs with
yum localinstall ./kernel*
Code Block [root@rhel6vanilla ~]# yum localinstall ./kernel* Loaded plugins: fastestmirror, presto Setting up Local Package Process Examining ./kernel-2.6.32-279.19.1.el6_lustre.x86_64.rpm: kernel-2.6.32-279.19.1.el6_lustre.x86_64 Marking ./kernel-2.6.32-279.19.1.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-279.el6.x86_64 Determining fastest mirrors * base: ftp.byfly.by * extras: ftp.byfly.by * updates: ftp.byfly.by Examining ./kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64 Marking ./kernel-firmware-2.6.32-279.19.1.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-279.el6.noarch Resolving Dependencies --> Running transaction check ---> Package kernel.x86_64 0:2.6.32-279.19.1.el6_lustre will be installed .x86_64 --> Running transaction check ---> Package kernel-firmware.noarch 0:2.6.32-279.el6 will be updated ---> Package kernel-firmware.noarch 0:2.6.32-358.6.1.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================= Package Arch Version Repository Size ======================================================================================================================= Installing: kernel x86_64 2.6.32-279.19.1.el6_lustre /kernel-2.6.32-279.19.1.el6_lustre.x86_64 113 M Updating for dependencies: kernel-firmware noarch 2.6.32-358.6.1.el6 updates 11 M Transaction Summary ======================================================================================================================= Install 1 Package(s) Upgrade 1 Package(s) Total size: 124 M Total download size: 11 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata updates/prestodelta | 339 kB 00:00 Processing delta metadata Download delta size: 3.4 M kernel-firmware-2.6.32-279.el6_2.6.32-358.6.1.el6.noarch.drpm | 3.4 MB 00:01 Finishing rebuild of rpms, from deltarpms <delta rebuild> | 11 MB 00:03 Presto reduced the update size by 69% (from 11 M to 3.4 M). Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : kernel-firmware-2.6.32-358.6.1.el6.noarch 1/3 Installing : kernel-2.6.32-279.19.1.el6_lustre.x86_64 2/3 Cleanup : kernel-firmware-2.6.32-279.el6.noarch 3/3 Verifying : kernel-2.6.32-279.19.1.el6_lustre.x86_64 1/3 Verifying : kernel-firmware-2.6.32-358.6.1.el6.noarch 2/3 Verifying : kernel-firmware-2.6.32-279.el6.noarch 3/3 Installed: kernel.x86_64 0:2.6.32-279.19.1.el6_lustre Dependency Updated: kernel-firmware.noarch 0:2.6.32-358.6.1.el6 Complete! [root@rhel6vanilla ~]#
Download e2fsprogs and Lustre server RPMSs.
- Download the following e2fsprogs (and dependencies) RPMs from http://build.whamcloud.com/job/e2fsprogs-master
e2fsprogs
e2fsprogs-libs
libss
libcom_err
- Download the following Lustre server RPMs from lustre-master build.
lustre
lustre-ldiskfs
lustre-modules
lustre-tests
Install Lustre.
yum localinstall ./lustre-* ./e2fsprogs-*
./libss-* ./libcom* should do it:Code Block [root@rhel6vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* ./libss-* ./libcom* Loaded plugins: fastestmirror, presto Setting up Local Package Process Examining ./lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm to be installed Loading mirror speeds from cached hostfile * base: ftp.byfly.by * extras: ftp.byfly.by * updates: ftp.byfly.by Examining ./lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm to be installed Examining ./lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm to be installed Examining ./lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm: lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Marking ./lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64.rpm to be installed Examining ./e2fsprogs-1.42.7.wc1-7.el6.x86_64.rpm: e2fsprogs-1.42.7.wc1-7.el6.x86_64 Marking ./e2fsprogs-1.42.7.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-12.el6.x86_64 Examining ./e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64.rpm: e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 Examining ./libss-1.42.7.wc1-7.el6.x86_64.rpm: libss-1.42.7.wc1-7.el6.x86_64 Marking ./libss-1.42.7.wc1-7.el6.x86_64.rpm as an update to libss-1.41.12-12.el6.x86_64 Examining ./libcom_err-1.42.7.wc1-7.el6.x86_64.rpm: libcom_err-1.42.7.wc1-7.el6.x86_64 Marking ./libcom_err-1.42.7.wc1-7.el6.x86_64.rpm as an update to libcom_err-1.41.12-12.el6.x86_64 Resolving Dependencies --> Running transaction check ---> Package e2fsprogs.x86_64 0:1.41.12-12.el6 will be updated ---> Package e2fsprogs.x86_64 0:1.42.7.wc1-7.el6 will be an update ---> Package e2fsprogs-libs.x86_64 0:1.41.12-12.el6 will be updated ---> Package e2fsprogs-libs.x86_64 0:1.42.7.wc1-7.el6 will be an update ---> Package libcom_err.x86_64 0:1.41.12-12.el6 will be updated ---> Package libcom_err.x86_64 0:1.42.7.wc1-7.el6 will be an update ---> Package libss.x86_64 0:1.41.12-12.el6 will be updated ---> Package libss.x86_64 0:1.42.7.wc1-7.el6 will be an update ---> Package lustre.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 will be installed --> Processing Dependency: /usr/bin/perl for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: perl(File::Path) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: perl(File::Path) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: perl(Getopt::Long) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: perl(Getopt::Std) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: perl(POSIX) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 will be installed ---> Package lustre-modules.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 will be installed ---> Package lustre-tests.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 will be installed --> Processing Dependency: libmpi.so.1()(64bit) for package: lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 --> Running transaction check ---> Package net-snmp-libs.x86_64 1:5.5-44.el6_4.1 will be installed --> Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 --> Processing Dependency: libperl.so()(64bit) for package: 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 ---> Package openmpi.x86_64 0:1.5.4-1.el6 will be installed --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1(OMP_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1(GOMP_1.0)(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: environment-modules for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: librdmacm.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libpsm_infinipath.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libnuma.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libltdl.so.7()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libibverbs.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgomp.so.1()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libgfortran.so.3()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 --> Processing Dependency: libesmtp.so.5()(64bit) for package: openmpi-1.5.4-1.el6.x86_64 ---> Package perl.x86_64 4:5.10.1-130.el6_4 will be installed --> Processing Dependency: perl(version) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Processing Dependency: perl(Pod::Simple) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Processing Dependency: perl(Module::Pluggable) for package: 4:perl-5.10.1-130.el6_4.x86_64 --> Running transaction check ---> Package environment-modules.x86_64 0:3.2.9c-4.el6 will be installed --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.9c-4.el6.x86_64 --> Processing Dependency: libX11.so.6()(64bit) for package: environment-modules-3.2.9c-4.el6.x86_64 ---> Package infinipath-psm.x86_64 0:3.0.1-115.1015_open.1.1.el6_4 will be installed ---> Package libesmtp.x86_64 0:1.0.4-15.el6 will be installed ---> Package libgfortran.x86_64 0:4.4.7-3.el6 will be installed ---> Package libgomp.x86_64 0:4.4.7-3.el6 will be installed ---> Package libibverbs.x86_64 0:1.1.6-5.el6 will be installed ---> Package librdmacm.x86_64 0:1.0.17-0.git4b5c1aa.el6 will be installed ---> Package libtool-ltdl.x86_64 0:2.2.6-15.5.el6 will be installed ---> Package lm_sensors-libs.x86_64 0:3.1.1-17.el6 will be installed ---> Package numactl.x86_64 0:2.0.7-6.el6 will be installed ---> Package perl-Module-Pluggable.x86_64 1:3.90-130.el6_4 will be installed ---> Package perl-Pod-Simple.x86_64 1:3.13-130.el6_4 will be installed --> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 ---> Package perl-libs.x86_64 4:5.10.1-130.el6_4 will be installed ---> Package perl-version.x86_64 3:0.77-130.el6_4 will be installed --> Running transaction check ---> Package libX11.x86_64 0:1.5.0-4.el6 will be installed --> Processing Dependency: libX11-common = 1.5.0-4.el6 for package: libX11-1.5.0-4.el6.x86_64 --> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.5.0-4.el6.x86_64 ---> Package perl-Pod-Escapes.x86_64 1:1.04-130.el6_4 will be installed ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed --> Running transaction check ---> Package libX11-common.noarch 0:1.5.0-4.el6 will be installed ---> Package libxcb.x86_64 0:1.8.1-1.el6 will be installed --> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.8.1-1.el6.x86_64 --> Running transaction check ---> Package libXau.x86_64 0:1.0.6-4.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================= Package Arch Version Repository Size ======================================================================================================================= Installing: lustre x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 26 M lustre-ldiskfs x86_64 3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 8.4 M lustre-modules x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 144 M lustre-tests x86_64 2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 /lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Updating: e2fsprogs x86_64 1.42.7.wc1-7.el6 /e2fsprogs-1.42.7.wc1-7.el6.x86_64 2.6 M e2fsprogs-libs x86_64 1.42.7.wc1-7.el6 /e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 349 k libcom_err x86_64 1.42.7.wc1-7.el6 /libcom_err-1.42.7.wc1-7.el6.x86_64 56 k libss x86_64 1.42.7.wc1-7.el6 /libss-1.42.7.wc1-7.el6.x86_64 67 k Installing for dependencies: environment-modules x86_64 3.2.9c-4.el6 base 98 k infinipath-psm x86_64 3.0.1-115.1015_open.1.1.el6_4 updates 160 k libX11 x86_64 1.5.0-4.el6 base 584 k libX11-common noarch 1.5.0-4.el6 base 192 k libXau x86_64 1.0.6-4.el6 base 24 k libesmtp x86_64 1.0.4-15.el6 base 57 k libgfortran x86_64 4.4.7-3.el6 base 265 k libgomp x86_64 4.4.7-3.el6 base 118 k libibverbs x86_64 1.1.6-5.el6 base 45 k librdmacm x86_64 1.0.17-0.git4b5c1aa.el6 base 47 k libtool-ltdl x86_64 2.2.6-15.5.el6 base 44 k libxcb x86_64 1.8.1-1.el6 base 110 k lm_sensors-libs x86_64 3.1.1-17.el6 base 38 k net-snmp-libs x86_64 1:5.5-44.el6_4.1 updates 1.5 M numactl x86_64 2.0.7-6.el6 base 69 k openmpi x86_64 1.5.4-1.el6 base 2.1 M perl x86_64 4:5.10.1-130.el6_4 updates perl-Module-Pluggable x86_64 1:3.90-130.el6_4 updates 38 k perl-Pod-Escapes x86_64 1:1.04-130.el6_4 updates 31 k perl-Pod-Simple x86_64 1:3.13-130.el6_4 updates 210 k perl-libs x86_64 4:5.10.1-130.el6_4 updates 576 k perl-version x86_64 3:0.77-130.el6_4 updates 50 k tcl x86_64 1:8.5.7-6.el6 base 1.9 M Transaction Summary ======================================================================================================================= Install 27 Package(s) Upgrade 4 Package(s) Total size: 213 M Total download size: 18 M Is this ok [y/N]: y Downloading Packages: Setting up and reading Presto delta metadata Processing delta metadata Package(s) data still to download: 18 M (1/23): environment-modules-3.2.9c-4.el6.x86_64.rpm | 98 kB 00:00 (2/23): infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64.rpm | 160 kB 00:00 (3/23): libX11-1.5.0-4.el6.x86_64.rpm | 584 kB 00:00 (4/23): libX11-common-1.5.0-4.el6.noarch.rpm | 192 kB (5/23): libXau-1.0.6-4.el6.x86_64.rpm | 24 kB 00:00 (6/23): libesmtp-1.0.4-15.el6.x86_64.rpm | 57 kB 00:00 (7/23): libgfortran-4.4.7-3.el6.x86_64.rpm | 265 kB 00:00 (8/23): libgomp-4.4.7-3.el6.x86_64.rpm | 118 kB 00:00 (9/23): libibverbs-1.1.6-5.el6.x86_64.rpm | 45 kB 00:00 (10/23): librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64.rpm | 47 kB 00:00 (11/23): libtool-ltdl-2.2.6-15.5.el6.x86_64.rpm | 44 kB 00:00 (12/23): libxcb-1.8.1-1.el6.x86_64.rpm | 110 kB 00:00 (13/23): lm_sensors-libs-3.1.1-17.el6.x86_64.rpm | 38 kB 00:00 (14/23): net-snmp-libs-5.5-44.el6_4.1.x86_64.rpm | 1.5 MB 00:01 (15/23): numactl-2.0.7-6.el6.x86_64.rpm | 69 kB 00:00 (16/23): openmpi-1.5.4-1.el6.x86_64.rpm | 2.1 MB 00:01 (17/23): perl-5.10.1-130.el6_4.x86_64.rpm | 10 MB 00:08 (18/23): perl-Module-Pluggable-3.90-130.el6_4.x86_64.rpm | 38 kB 00:00 (19/23): perl-Pod-Escapes-1.04-130.el6_4.x86_64.rpm | 31 kB 00:00 (20/23): perl-Pod-Simple-3.13-130.el6_4.x86_64.rpm | 210 kB 00:00 (21/23): perl-libs-5.10.1-130.el6_4.x86_64.rpm | 576 kB 00:00 (22/23): perl-version-0.77-130.el6_4.x86_64.rpm | 50 kB 00:00 (23/23): tcl-8.5.7-6.el6.x86_64.rpm | 1.9 MB 00:01 ----------------------------------------------------------------------------------------------------------------------- Total 1.2 MB/s | 18 MB Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : 1:perl-Pod-Escapes-1.04-130.el6_4.x86_64 1/35 Installing : 4:perl-libs-5.10.1-130.el6_4.x86_64 2/35 Installing : 1:perl-Module-Pluggable-3.90-130.el6_4.x86_64 3/35 Installing : 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 4/35 Installing : 3:perl-version-0.77-130.el6_4.x86_64 5/35 Installing : 4:perl-5.10.1-130.el6_4.x86_64 6/35 Updating : libcom_err-1.42.7.wc1-7.el6.x86_64 7/35 Updating : e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 8/35 Installing : libibverbs-1.1.6-5.el6.x86_64 9/35 Installing : librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64 10/35 Updating : libss-1.42.7.wc1-7.el6.x86_64 11/35 Updating : e2fsprogs-1.42.7.wc1-7.el6.x86_64 12/35 Installing : lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 13/35 Installing : lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 14/35 Installing : libgomp-4.4.7-3.el6.x86_64 15/35 Installing : lm_sensors-libs-3.1.1-17.el6.x86_64 16/35 Installing : 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 17/35 Installing : lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 Installing : libXau-1.0.6-4.el6.x86_64 19/35 Installing : libxcb-1.8.1-1.el6.x86_64 20/35 Installing : libX11-common-1.5.0-4.el6.noarch 21/35 Installing : libX11-1.5.0-4.el6.x86_64 22/35 Installing : libgfortran-4.4.7-3.el6.x86_64 23/35 Installing : libesmtp-1.0.4-15.el6.x86_64 24/35 Installing : numactl-2.0.7-6.el6.x86_64 25/35 Installing : infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64 26/35 Installing : 1:tcl-8.5.7-6.el6.x86_64 27/35 Installing : environment-modules-3.2.9c-4.el6.x86_64 28/35 Installing : libtool-ltdl-2.2.6-15.5.el6.x86_64 29/35 Installing : openmpi-1.5.4-1.el6.x86_64 30/35 Installing : lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 31/35 Cleanup : e2fsprogs-1.41.12-12.el6.x86_64 32/35 Cleanup : e2fsprogs-libs-1.41.12-12.el6.x86_64 33/35 Cleanup : libss-1.41.12-12.el6.x86_64 34/35 Cleanup : libcom_err-1.41.12-12.el6.x86_64 35/35 Verifying : libtool-ltdl-2.2.6-15.5.el6.x86_64 1/35 Verifying : libss-1.42.7.wc1-7.el6.x86_64 2/35 Verifying : e2fsprogs-libs-1.42.7.wc1-7.el6.x86_64 3/35 Verifying : 1:tcl-8.5.7-6.el6.x86_64 4/35 Verifying : 4:perl-5.10.1-130.el6_4.x86_64 Verifying : lustre-modules-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 6/35 Verifying : libcom_err-1.42.7.wc1-7.el6.x86_64 7/35 Verifying : lustre-tests-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 8/35 Verifying : infinipath-psm-3.0.1-115.1015_open.1.1.el6_4.x86_64 9/35 Verifying : environment-modules-3.2.9c-4.el6.x86_64 10/35 Verifying : libX11-1.5.0-4.el6.x86_64 11/35 Verifying : numactl-2.0.7-6.el6.x86_64 12/35 Verifying : lustre-2.1.5-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 13/35 Verifying : 1:perl-Pod-Simple-3.13-130.el6_4.x86_64 14/35 Verifying : libesmtp-1.0.4-15.el6.x86_64 15/35 Verifying : libgfortran-4.4.7-3.el6.x86_64 16/35 Verifying : openmpi-1.5.4-1.el6.x86_64 17/35 Verifying : libX11-common-1.5.0-4.el6.noarch 18/35 Verifying : 4:perl-libs-5.10.1-130.el6_4.x86_64 19/35 Verifying : libxcb-1.8.1-1.el6.x86_64 20/35 Verifying : libXau-1.0.6-4.el6.x86_64 21/35 Verifying : 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 22/35 Verifying : 1:perl-Module-Pluggable-3.90-130.el6_4.x86_64 23/35 Verifying : lm_sensors-libs-3.1.1-17.el6.x86_64 24/35 Verifying : 1:perl-Pod-Escapes-1.04-130.el6_4.x86_64 25/35 Verifying : libgomp-4.4.7-3.el6.x86_64 26/35 Verifying : lustre-ldiskfs-3.3.0-2.6.32_279.19.1.el6_lustre.x86_64.x86_64 27/35 Verifying : 3:perl-version-0.77-130.el6_4.x86_64 Verifying : libibverbs-1.1.6-5.el6.x86_64 29/35 Verifying : e2fsprogs-1.42.7.wc1-7.el6.x86_64 30/35 Verifying : librdmacm-1.0.17-0.git4b5c1aa.el6.x86_64 31/35 Verifying : libss-1.41.12-12.el6.x86_64 32/35 Verifying : e2fsprogs-1.41.12-12.el6.x86_64 33/35 Verifying : libcom_err-1.41.12-12.el6.x86_64 34/35 Verifying : e2fsprogs-libs-1.41.12-12.el6.x86_64 35/35 Installed: lustre.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_279.19.1.el6_lustre.x86_64 lustre-modules.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 lustre-tests.x86_64 0:2.1.5-2.6.32_279.19.1.el6_lustre.x86_64 Dependency Installed: environment-modules.x86_64 0:3.2.9c-4.el6 infinipath-psm.x86_64 0:3.0.1-115.1015_open.1.1.el6_4 libX11.x86_64 0:1.5.0-4.el6 libX11-common.noarch 0:1.5.0-4.el6 libXau.x86_64 0:1.0.6-4.el6 libesmtp.x86_64 0:1.0.4-15.el6 libgfortran.x86_64 0:4.4.7-3.el6 libgomp.x86_64 0:4.4.7-3.el6 libibverbs.x86_64 0:1.1.6-5.el6 librdmacm.x86_64 0:1.0.17-0.git4b5c1aa.el6 libtool-ltdl.x86_64 0:2.2.6-15.5.el6 libxcb.x86_64 0:1.8.1-1.el6 lm_sensors-libs.x86_64 0:3.1.1-17.el6 net-snmp-libs.x86_64 1:5.5-44.el6_4.1 numactl.x86_64 0:2.0.7-6.el6 openmpi.x86_64 0:1.5.4-1.el6 perl.x86_64 4:5.10.1-130.el6_4 perl-Module-Pluggable.x86_64 1:3.90-130.el6_4 perl-Pod-Escapes.x86_64 1:1.04-130.el6_4 perl-Pod-Simple.x86_64 1:3.13-130.el6_4 perl-libs.x86_64 4:5.10.1-130.el6_4 perl-version.x86_64 3:0.77-130.el6_4 tcl.x86_64 1:8.5.7-6.el6 Updated: e2fsprogs.x86_64 0:1.42.7.wc1-7.el6 e2fsprogs-libs.x86_64 0:1.42.7.wc1-7.el6 libcom_err.x86_64 0:1.42.7.wc1-7.el6 libss.x86_64 0:1.42.7.wc1-7.el6 Complete! [root@rhel6vanilla ~]#
Disable SELinux for Lustre patched kernel.
Lustre does not play well with SELinux (at least, the llmount.sh
script fails when SELinux is enabled), so SELinux must be disabled. One option is to disable SELinux with GRUB. With your editor of choice, open /boot/grub/grub.conf
. Append the option selinux=0
to the end of the kernel
line for the Lustre patched kernel entry and save. This will cause SELinux to be disabled after you reboot into the patched kernel.
Reboot.
reboot
Verify that installation is complete:
uname -a
Code Block [root@rhel6vanilla
- Reboot
- Verify that the new kernel has been booted:
Code Block [root@client-10 ~]# uname -a Linux client-10.lab.whamcloud.comrhel6vanilla 2.6.1832-194279.1719.1.el5el6_lustre.gdfe90dcx86_64 #1 SMP Sun Mar 13 03:18:21 EET 2011 x86_64 x86_64 x86_64 GNU/Linux
Installing Ldiskfs.
- From the page that linked to the kernel, identify the Lustre modules RPM
The package will be named in the formlustre-ldiskfs-3.3.0-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm
- Download the Lustre modules RPM.
- Install the RPM.
Code Block [root@client-10 ~]# rpm -ivh ./lustre-ldiskfs-3.3.0-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm Preparing... ########################################### [100%] 1:lustre-ldiskfs ########################################### [100%] [root@client-10 ~]#
Install Lustre modules.
- From the page that linked to the kernel, identify the Lustre modules RPM
The package will be named in the formlustre-modules-2.0.59-2.6.18_194.17.1.el5_lustre.<buildid>.x86_64.rpm
- Download the Lustre modules RPM.
- Install the RPM.
Code Block [root@client-10 ~]# rpm --force -ivh ./lustre-modules-2.0.59-2.6.18_194.17.1.el5_lustre.gdfe90dc_g98bf9c9.x86_64.rpm Preparing... ########################################### [100%] 1:lustre-modules ########################################### [100%] [root@client-10 ~]#
Install e2fsprogs.
e2fsprogs is built in a project separate from the Lustre kernel builds. e2fsprogs is found at http://build.whamcloud.com/job/e2fsprogs-master/ on the Whamcloud build server.
- Identify the most recent build of e2fsprogs.
The RPM will have a name in the form:e2fsprogs-1.41.90.wc3-0redhat.x86_64.rpm
- Download the most recent RPM.
- Install the RPM
...
Wed Mar 20 16:37:18 PDT 2013 x86_64 x86_64 x86_64 GNU/Linux
Create a local Lustre filesystem with:
llmount.sh
Code Block [root@rhel6vanilla ~]# /usr/lib64/lustre/tests/llmount.sh Stopping clients: rhel6vanilla /mnt/lustre (opts:) Stopping clients: rhel6vanilla /mnt/lustre2 (opts:) Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6vanilla environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=2 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=2 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=2 Started lustre-OST0001 Starting client: rhel6vanilla: -o user_xattr,acl,flock rhel6vanilla@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=2 Using TIMEOUT=20 disable quota as required [root@rhel6vanilla ~]# mount /dev/mapper/vg_rhel6vanilla-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) /dev/vda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl) /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1) /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2) rhel6vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)
Testing
Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem