{{
}}
Purpose
Describe the steps you need to build and test a 2.1 Lustre system (MGS, MDT, MDS, OSS, OST, client) on a x86_64, RHEL 6.1 machine.
Prerequisite
- A newly installed RHEL 6.1 x86_64 machine with the hostname: rhel61_21.
- EPEL Repository: this is a convenient source for git.
NOTE the EPEL 5 repository is used because it includes quilt. - Have the system registered with Redhat Network.
- NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
Overview
Lustre 2.1 servers require a patched and compiled kernel. Patches are available in the Whamcloud git source repository. A test suite is included with the Lustre 2.1 source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system. Note that a patched kernel is NOT needed for the Lustre client.
Procedure
The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.
Provision machine and installing dependencies.
Once RHEL 6.1 is newly installed on rhel6_21 login as root.
- Install the kernel development tools.
# yum -y groupinstall "Development Tools"
- Install kernel build dependencies
yum -y install git xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed xmlto
- Install EPEL 5
NOTE EPEL5 is used because it contains quilt.# wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm # rpm -ivh ./epel-release-5-4.noarch.rpm
- Install quilt
# yum -y install quilt
Preparing the Lustre source.
- Create a user
build
with the home directory/build
# useradd -m build
- Switch to the user
build
and change to the build$HOME
directory.# su build # cd $HOME
- Get the MASTER branch from Whamcloud git.
# git clone git://git.whamcloud.com/fs/lustre-release.git # cd lustre-release
- Run
sh ./autogen.sh
- Resolve any outstanding dependencies until
autogen.sh
completes successfully. Success will look like:# sh ./autogen.sh Checking for a complete tree... checking for automake-1.9 >= 1.9... found 1.9.6 ... ... configure.ac:10: installing `./config.sub' configure.ac:12: installing `./install-sh' configure.ac:12: installing `./missing' Running autoconf
Prepare the kernel source
In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.
- Get the kernel source. First create the directory structure, then get the source from the RPM. Create a
.rpmmacros
file to install the kernel source in our user dir.# cd $HOME # mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} # cd kernel # echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros
- Install the kernel source:
# rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-131.2.1.el6.src.rpm 2>&1 | grep -v mockb
- Prepare the source using rpmbuild.
# cd ~/kernel/rpmbuild # rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
This will end with:... gpg: Total number processed: 1 gpg: imported: 1 + gpg --homedir . --export --keyring ./kernel.pub Red gpg: WARNING: unsafe permissions on homedir `.' + gcc -o scripts/bin2c scripts/bin2c.c + scripts/bin2c ksign_def_public_key __initdata + cd .. + exit 0
At this point, we now have kernel souce, with all the RHEL patches applied, residing in the directory /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
Patch the kernel source with the Lustre code.
- Add a unique build id so we can be certain our kernel is booted. Edit
/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/Makefile
and modify line 4, theEXTRAVERSION
to read:EXTRAVERSION = .lustre21
- enter the directory
/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
# cd /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
- overwrite the
.config
file with/home/build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config
# cp /home/build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config
- link the Lustre series and patches
# ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series # ln -s ~/lustre-release/lustre/kernel_patches/patches patches
- Apply the patches to the kernel source using quilt
# quilt push -av ... ... patching file fs/jbd2/transaction.c Hunk #3 succeeded at 1222 (offset 3 lines). Hunk #4 succeeded at 1357 (offset 3 lines). Now at patch patches/jbd2-jcberr-2.6-rhel6.patch
Build the new kernel as an RPM.
- Go into the kernel source directory and issue the following commands to build a kernel rpm.
# cd /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/ # make oldconfig || make menuconfig # make include/asm # make include/linux/version.h # make SUBDIRS=scripts # make include/linux/utsrelease.h # make rpm
- A successful build will return:
... ... Wrote: /build/kernel/rpmbuild/SRPMS/kernel-2.6.32lustre21-1.src.rpm Wrote: /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V + umask 022 + cd /build/kernel/rpmbuild/BUILD + cd kernel-2.6.32lustre21 + rm -rf /build/kernel/rpmbuild/BUILDROOT/kernel-2.6.32.lustre21-1.x86_64 + exit 0 rm ../kernel-2.6.32lustre21.tar.gz
NOTE If you receive a request to generate more entropy, you need to trigger some disk I/O or keyboard I/O. I would recommend (in another terminal):
# grep -Ri 'whamcloud' /usr
At this point, you should have a fresh kernel RPM /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm
Configure and build Lustre
- Configure Lustre source
# cd ~/lustre-release/ # ./configure --with-linux=/build/kernel/rpmbuild/BUILD/kernel-2.6.32.lustre21/ ... ... LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1 CFLAGS: -g -O2 -Werror EXTRA_KCFLAGS: -include /build/lustre-release/config.h -g -I/build/lustre-release/libcfs/include -I/build/lustre-release/lnet/include -I/build/lustre-release/lustre/include LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE Type 'make' to build Lustre.
- make rpms:
# make rpms ... ... Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD + umask 022 + cd /build/kernel/rpmbuild/BUILD + cd lustre-2.0.61 + rm -rf /build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustre21_g0533e7b.x86_64 + exit 0 make[1]: Leaving directory `/build/lustre-release'
- You should now have build the following, similarly named, rpms:
# ls ~build/kernel/rpmbuild/RPMS/x86_64/ kernel-2.6.32lustre21-1.x86_64.rpm lustre-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-debuginfo-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-ldiskfs-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-modules-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-source-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm lustre-tests-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
Installing the Lustre kernel and rebooting.
- As root, Install the kernel
# rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32lustre21-1.x86_64.rpm
- Create initrd using
dracut
# /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32lustre21
reboot
- view the login prompt with satisfaction:
Red Hat Enterprise Linux Server release 6.0 (Santiago) Kernel 2.6.32lustre21 on an x86_64 client-10 login:
Installing e2fsprogs
e2fsprogs is needed to run the test suite.
- Download e2fsprogs from http://downloads.whamcloud.com/public/e2fsprogs/latest/
- Install with
# rpm -Uvh ./e2fsprogs-1.41.90.wc2-7.el6.x86_64.rpm ./e2fsprogs-libs-1.41.90.wc2-7.el6.x86_64.rpm
Installing Lustre.
- Change to
root
and Change directory into/build/kernel/rpmbuild/RPMS/x86_64/
- Install modules
lustre-modules
and user space toolslustre-
# rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustre21* # rpm -ivh lustre-modules-2.0.61-2.6.32.lustre21* # rpm -ivh lustre-2.0.61-2.6.32.lustre21_* # rpm -ivh lustre-tests-*
Testing
- run
/usr/lib64/lustre/tests/llmount.sh
# /usr/lib64/lustre/tests/llmount.sh Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6_build environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel5_build: -o user_xattr,acl,flock rhel6_build@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required
- you will now have a Lustre filesystem available at
/mnt/lustre
- NOTE: if you receive an error:
mkfs.lustre: Can't parse NID 'rhel6_build@tcp'
you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.