Walk-thru- Build Lustre MASTER on RHEL 6.1 from Whamcloud git

{{

}}

Purpose

Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the Whamcloud MASTER branch on a x86_64, RHEL 6.1 machine.

Prerequisite

• A newly installed RHEL 6.1 x86_64 machine with the hostname: rhel61_21.
• EPEL Repository: this is a convenient source for git.
  NOTE the EPEL 5 repository is used because it includes quilt.
• Have the system registered with Redhat Network.
• NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.

Overview

Lustre 2.1 servers require a patched and compiled kernel. Patches are available in the Whamcloud git source repository. A test suite is included with the Lustre 2.1 source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system. Note that a patched kernel is NOT needed for the Lustre client.

Procedure

The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.

Provision machine and installing dependencies.

Once RHEL 6.1 is newly installed on rhel6_21 login as root.

1. Install the kernel development tools.

   # yum -y groupinstall "Development Tools"
   

2. Install kernel build dependencies

   yum -y install git automake xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed xmlto
   

3. Install EPEL 5
   NOTE EPEL5 is used because it contains quilt.

   # wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
   # rpm -ivh ./epel-release-5-4.noarch.rpm
   

4. Install quilt

   # yum -y install quilt
   

5. Install some additional dependencies that may not have been included with 'Development Tools'

   # yum -y install make gcc redhat-rpm-config patchutils asciidoc newt-devel
   

Preparing the Lustre source.

1. Create a user build with the home directory /build

   # useradd -m build
   

2. Switch to the user build and change to the build $HOME directory.

   # su build
   # cd $HOME
   

3. Get the MASTER branch from Whamcloud git.

   # git clone git://git.whamcloud.com/fs/lustre-release.git
   # cd lustre-release
   

4. Run sh ./autogen.sh
5. Resolve any outstanding dependencies until autogen.sh completes successfully. Success will look like:

   # sh ./autogen.sh 
   Checking for a complete tree...
   checking for automake-1.9 >= 1.9... found 1.9.6
   ...
   ...
   configure.ac:10: installing `./config.sub'
   configure.ac:12: installing `./install-sh'
   configure.ac:12: installing `./missing'
   
   Running autoconf
   
   

Prepare the kernel source

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

1. Get the kernel source. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user dir.

   # cd $HOME
   # mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
   # cd kernel
   # echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros
   

2. Install the kernel source:

   # rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-131.2.1.el6.src.rpm 2>&1 | grep -v mockb
   

   RHEL kernel versions

   RHEL periodically releases updates to the kernel. The Lustre Master branch tracks the most recent kernel from Red Hat. In the even that this page is not completely up-to-date, you should visit the Red Hat source RPM download site and manually ensure you are downloading the most recent kernel.

3. Prepare the source using rpmbuild.

   # cd ~/kernel/rpmbuild
   # rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
   

   
   This will end with:

   ...
   gpg: Total number processed: 1
   gpg:               imported: 1
   + gpg --homedir . --export --keyring ./kernel.pub Red
   gpg: WARNING: unsafe permissions on homedir `.'
   + gcc -o scripts/bin2c scripts/bin2c.c
   + scripts/bin2c ksign_def_public_key __initdata
   + cd ..
   + exit 0
   

At this point, we now have kernel souce, with all the RHEL patches applied, residing in the directory /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/

Patch the kernel source with the Lustre code.

1. Add a unique build id so we can be certain our kernel is booted. Edit /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/Makefile and modify line 4, the EXTRAVERSION to read:

   EXTRAVERSION = .lustre21
   

2. enter the directory /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/

   # cd /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
   

3. overwrite the .config file with /home/build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config

   # cp /home/build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config
   

4. link the Lustre series and patches

   # ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series
   # ln -s ~/lustre-release/lustre/kernel_patches/patches patches
   

5. Apply the patches to the kernel source using quilt

   # quilt push -av
   ...
   ...
   patching file fs/jbd2/transaction.c
   Hunk #3 succeeded at 1222 (offset 3 lines).
   Hunk #4 succeeded at 1357 (offset 3 lines).
   
   Now at patch patches/jbd2-jcberr-2.6-rhel6.patch
   

Build the new kernel as an RPM.

1. Go into the kernel source directory and issue the following commands to build a kernel rpm.

   # cd /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
   # make oldconfig || make menuconfig
   # make include/asm
   # make include/linux/version.h
   # make SUBDIRS=scripts
   # make include/linux/utsrelease.h
   # make rpm
   

2. A successful build will return:

   ...
   ...
   Wrote: /build/kernel/rpmbuild/SRPMS/kernel-2.6.32lustre21-1.src.rpm
   Wrote: /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm
   Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V
   + umask 022
   + cd /build/kernel/rpmbuild/BUILD
   + cd kernel-2.6.32lustre21
   + rm -rf /build/kernel/rpmbuild/BUILDROOT/kernel-2.6.32.lustre21-1.x86_64
   + exit 0
   rm ../kernel-2.6.32lustre21.tar.gz
   

# grep -Ri 'whamcloud' /usr

At this point, you should have a fresh kernel RPM /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm

Configure and build Lustre

1. Configure Lustre source

   # cd ~/lustre-release/
   # ./configure --with-linux=/build/kernel/rpmbuild/BUILD/kernel-2.6.32.lustre21/
   ...
   ...
   LLCPPFLAGS:    -D__arch_lib__ -D_LARGEFILE64_SOURCE=1
   CFLAGS:        -g -O2 -Werror
   EXTRA_KCFLAGS: -include /build/lustre-release/config.h  -g -I/build/lustre-release/libcfs/include -I/build/lustre-release/lnet/include -I/build/lustre-release/lustre/include
   LLCFLAGS:      -g -Wall -fPIC -D_GNU_SOURCE
   
   Type 'make' to build Lustre.
   

2. make rpms:

   # make rpms
   ...
   ...
   Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD
   + umask 022
   + cd /build/kernel/rpmbuild/BUILD
   + cd lustre-2.0.61
   + rm -rf /build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustre21_g0533e7b.x86_64
   + exit 0
   make[1]: Leaving directory `/build/lustre-release'
   

3. You should now have build the following, similarly named, rpms:

   # ls ~build/kernel/rpmbuild/RPMS/x86_64/
   kernel-2.6.32lustre21-1.x86_64.rpm
   lustre-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-debuginfo-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-ldiskfs-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-modules-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-source-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
   lustre-tests-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
   

Installing the Lustre kernel and rebooting.

1. As root, Install the kernel

   # rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32lustre21-1.x86_64.rpm
   

2. Create initrd using dracut

   # /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32lustre21
   

3. reboot
4. view the login prompt with satisfaction:

   Red Hat Enterprise Linux Server release 6.0 (Santiago)
   Kernel 2.6.32lustre21 on an x86_64
   
   client-10 login:
   

Installing e2fsprogs

e2fsprogs is needed to run the test suite.

1. Download e2fsprogs from http://downloads.whamcloud.com/public/e2fsprogs/latest/
2. Install with

   # rpm -Uvh ./e2fsprogs-1.41.90.wc2-7.el6.x86_64.rpm  ./e2fsprogs-libs-1.41.90.wc2-7.el6.x86_64.rpm
   

Installing Lustre.

1. Change to root and Change directory into /build/kernel/rpmbuild/RPMS/x86_64/
2. Install modules lustre-modules and user space tools lustre-

   # rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustre21*
   # rpm -ivh lustre-modules-2.0.61-2.6.32.lustre21*
   # rpm -ivh lustre-2.0.61-2.6.32.lustre21_*
   # rpm -ivh lustre-tests-*
   

Testing

1. run /usr/lib64/lustre/tests/llmount.sh

   # /usr/lib64/lustre/tests/llmount.sh
   Loading modules from /usr/lib64/lustre/tests/..
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   gss/krb5 is not supported
   Formatting mgs, mds, osts
   Format mds1: /tmp/lustre-mdt1
   Format ost1: /tmp/lustre-ost1
   Format ost2: /tmp/lustre-ost2
   Checking servers environments
   Checking clients rhel6_build environments
   Loading modules from /usr/lib64/lustre/tests/..
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   gss/krb5 is not supported
   Setup mgs, mdt, osts
   Starting mds1: -o loop,user_xattr,acl  /tmp/lustre-mdt1 /mnt/mds1
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   debug_mb=10
   Started lustre-MDT0000
   Starting ost1: -o loop  /tmp/lustre-ost1 /mnt/ost1
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   debug_mb=10
   Started lustre-OST0000
   Starting ost2: -o loop  /tmp/lustre-ost2 /mnt/ost2
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   debug_mb=10
   Started lustre-OST0001
   Starting client: rhel5_build: -o user_xattr,acl,flock rhel6_build@tcp:/lustre /mnt/lustre
   debug=0x33f0404
   subsystem_debug=0xffb7e3ff
   debug_mb=10
   Using TIMEOUT=20
   disable quota as required
   

2. you will now have a Lustre filesystem available at /mnt/lustre
3. NOTE: if you receive an error: mkfs.lustre: Can't parse NID 'rhel6_build@tcp' you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.

ENDS~