Purpose

Describe the steps you need to build and test a 2.1 Lustre system (MGS, MDT, MDS, OSS, OST, client) on a RHEL6 machine.

Prerequisite

• A newly installed RHEL6 x86_64 machine with the hostname: rhel6_21.
• EPEL Repository: this is a convenient source for git.
  NOTE the EPEL 5 repository is used because it includes quilt.

Overview

Lustre 2.1 servers require a patched and compiled kernel. Patches are readily available in the Whamcloud git source repository. A test suite is included with the Lustre 2.1 source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.

Procedure

The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.

Provision Machine

Once RHEL 6 is newly installed on rhel6_21 login as root.

1. Install required kernel development tools.

   yum -y groupinstall "Development Tools"

   NOTE You may need to manually install the tools if the group install doesn't work:

   yum install -y make gcc patchutils elfutils-libelf-devel zlib-devel binutils-devel hmaccalc

2. Install EPEL 5
   NOTE EPEL5 is used because it contains quilt.

   wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm rpm -ivh ./epel-release-5-4.noarch.rpm

3. install rpm-build etc...

   yum -y install git yum -y install rpm-build redhat-rpm-config unifdef gnupg quilt automake

4. Create a user build with the home directory /build

   useradd -d /build build

5. Switch to the user build and change to the build $HOME directory.

   su build cd $HOME

6. Get the 1.8 branch from the Whamcloud git account.

   git clone git://git.whamcloud.com/fs/lustre-release.git cd lustre-release

7. Run sh ./autogen.sh
8. Resolve any outstanding dependencies until autogen.sh completes successfully. Success will look like:

   [root@client-10 lustre-release]# sh ./autogen.sh Checking for a complete tree... checking for automake-1.9 >= 1.9... found 1.9.6 ... ... configure.ac:10: installing `./config.sub' configure.ac:12: installing `./install-sh' configure.ac:12: installing `./missing' Running autoconf [root@client-10 lustre-release]#

Prepare the kernel source

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

1. Get the kernel source. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user dir.

   cd $HOME mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} cd kernel echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

2. Install the kernel source:

   rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-71.29.1.el6.src.rpm 2>&1 | grep -v mockb

3. Expand the source. Using rpmbuild will also apply CentOS patches.

   cd ~/kernel/rpmbuild/SPECS rpmbuild -bp --target=`uname -m` ./kernel.spec

   
   This will end with:

   ... gpg: Total number processed: 1 gpg: imported: 1 + gpg --homedir . --export --keyring ./kernel.pub Red gpg: WARNING: unsafe permissions on homedir `.' + gcc -o scripts/bin2c scripts/bin2c.c + scripts/bin2c ksign_def_public_key __initdata + cd .. + exit 0

At this point, we now have kernel souce, with all the RHEL patches applied, residing in the directory /build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64

Patch the kernel source with the Lustre code.

1. Add a unique build id so we can be certain our kernel is booted. Edit ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/Makefile and modify line 4, the EXTRAVERSION to read:

   EXTRAVERSION = -lustre21

2. enter the directory ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/

   cd ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/

3. overwrite the .config file with /build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

   cp /build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config

4. link the Lustre series and patches

   ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series ln -s ~/lustre-release/lustre/kernel_patches/patches patches

5. Apply the patches to the kernel source using quilt

   quilt push -av ... ... patching file fs/jbd2/journal.c Hunk #1 succeeded at 94 (offset 4 lines). patching file fs/jbd2/transaction.c Applying patch patches/jbd2-commit-timer-no-jiffies-rounding-rhel6.patch patching file fs/jbd2/transaction.c Now at patch patches/jbd2-commit-timer-no-jiffies-rounding-rhel6.patch

Build the new kernel as an RPM.

1. Go into the kernel source directory and issue the following commands to build a kernel rpm.

   cd /build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64 make oldconfig || make menuconfig make include/asm make include/linux/version.h make SUBDIRS=scripts make include/linux/utsrelease.h make rpm

2. A successful build will return:

   ... ... Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/kernel-2.6.18lustre18-root Wrote: /build/kernel/rpmbuild/SRPMS/kernel-2.6.18lustre18-1.src.rpm Wrote: /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.18lustre18-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.35163 + umask 022 + cd /build/kernel/rpmbuild/BUILD + cd kernel-2.6.18lustre18 + exit 0 rm ../kernel-2.6.18lustre18.tar.gz

NOTE If you receive a request to generate more entropy, you need to trigger some disk I/O or keyboard I/O. I would recommend (in another terminal):

grep -Ri 'whamcloud' /usr

At this point, you should have a fresh kernel RPM /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.18lustre18-1.x86_64.rpm

Configure and build Lustre

1. Configure Lustre source

   [build@client-10 linux-2.6.18.x86_64]$ cd ~/lustre-release/ [build@client-10 lustre-release]$ ./configure --with-linux=/build/kernel/rpmbuild/BUILD/kernel-2.6.18lustre18/ ... ... LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1 CFLAGS: -g -O2 -Werror EXTRA_KCFLAGS: -include /build/lustre-release/config.h -g -I/build/lustre-release/lnet/include -I/build/lustre-release/lnet/include -I/build/lustre-release/lustre/include LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE Type 'make' to build Lustre.

2. make rpms:

   [build@client-10 lustre-release]$ make rpms ... ... Wrote: /build/kernel/rpmbuild/RPMS/x86_64/lustre-debuginfo-1.8.5.54-2.6.18_194.32.1.el5.lustre18_201103071000.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.15638 + umask 022 + cd /build/kernel/rpmbuild/BUILD + cd lustre-1.8.5.54 + rm -rf /var/tmp/lustre-1.8.5.54-root + exit 0 make[1]: Leaving directory `/build/lustre-release'

3. You should now have build the following, similarly named, rpms:

   ls ~build/kernel/rpmbuild/RPMS/x86_64/ lustre-debuginfo-1.8.5.54-2.6.18_lustre18_201103081147.x86_64.rpm lustre-tests-1.8.5.54-2.6.18_lustre18_201103081147.x86_64.rpm lustre-source-1.8.5.54-2.6.18_lustre18_201103081147.x86_64.rpm lustre-modules-1.8.5.54-2.6.18_lustre18_201103081147.x86_64.rpm lustre-1.8.5.54-2.6.18_lustre18_201103081147.x86_64.rpm lustre-ldiskfs-3.1.5-2.6.18_lustre18_201103081148.x86_64.rpm lustre-ldiskfs-debuginfo-3.1.5-2.6.18_lustre18_201103081148.x86_64.rpm kernel-2.6.18lustre18-1.x86_64.rpm

Installing the Lustre kernel and rebooting.

1. As root, Install the kernel

   [root@client-10 ~]# rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.18lustre18-1.x86_64.rpm

2. Check that /boot/grub/menu.lst contains the correct default kernel to boot. This is typically 0:

   Default=0

3. reboot
4. connect with conman, and watch the machine come up
5. view the login prompt with satisfaction:

   CentOS release 5.5 (Final) Kernel 2.6.18-lustre18 on an x86_64 client-10.lab.whamcloud.com login:

Installing Lustre.

1. Change to root and Change directory into /build/kernel/rpmbuild/RPMS/x86_64/
2. Install modules lustre-modules and user space tools lustre-

   rpm -ivh /build/kernel/rpmbuild/RPMS/x86_64/lustre-modules-1.8.5.54-2.6.18_lustre18_*.x86_64.rpm /build/kernel/rpmbuild/RPMS/x86_64/lustre-1.8.5.54-2.6.18_lustre18_*.x86_64.rpm /build/kernel/rpmbuild/RPMS/x86_64/lustre-ldiskfs-3.1.5-2.6.18_lustre18_*.x86_64.rpm

Installing e2fsprogs

e2fsprogs is needed to run the test suite.

1. Download e2fsprogs from http://build.whamcloud.com/job/e2fsprogs-master/
2. Install with rpm -ivh e2fsprogs

ENDS~