Walk-thru- Build Lustre 2.1 on RHEL6 from Whamcloud git

Purpose

Describe the steps you need to build and test a 2.1 Lustre system (MGS, MDT, MDS, OSS, OST, client) on a RHEL6 machine.

Prerequisite

A newly installed RHEL6 x86_64 machine with the hostname: rhel6_21.
EPEL Repository: this is a convenient source for git.
NOTE the EPEL 5 repository is used because it includes quilt.
NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.

Overview

Lustre 2.1 servers require a patched and compiled kernel. Patches are available in the Whamcloud git source repository. A test suite is included with the Lustre 2.1 source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.

Procedure

The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.

Provision Machine

Once RHEL 6 is newly installed on rhel6_21 login as root.

Install EPEL 5
NOTE EPEL5 is used because it contains quilt.

# wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
# rpm -ivh ./epel-release-5-4.noarch.rpm

Install required kernel development tools.

# yum -y groupinstall "Development Tools"

NOTE You may need to manually install the tools if the group install doesn't work:

# yum install -y make gcc patchutils elfutils-libelf-devel zlib-devel binutils-devel hmaccalc

install rpm-build etc...

# yum -y install rpm-build redhat-rpm-config unifdef gnupg quilt automake

Now install EPEL6, which provides git.

# wget http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
# rpm -Uvh ./epel-release-6-5.noarch.rpm

install git
```
# yum -y install git
```
Create a user build with the home directory /build
```
# useradd -d /build build
```
Switch to the user build and change to the build $HOME directory.
```
# su build
# cd $HOME
```

Get the 1.8 branch from the Whamcloud git account.

# git clone git://git.whamcloud.com/fs/lustre-release.git
# cd lustre-release

Run sh ./autogen.sh

Resolve any outstanding dependencies until autogen.sh completes successfully. Success will look like:

# sh ./autogen.sh 
Checking for a complete tree...
checking for automake-1.9 >= 1.9... found 1.9.6
...
...
configure.ac:10: installing `./config.sub'
configure.ac:12: installing `./install-sh'
configure.ac:12: installing `./missing'

Running autoconf

Prepare the kernel source

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

Get the kernel source. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user dir.
```
# cd $HOME
# mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
# cd kernel
# echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros
```

Install the kernel source:

# rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-71.29.1.el6.src.rpm 2>&1 | grep -v mockb

Expand the source. Using rpmbuild will also apply CentOS patches.

# cd ~/kernel/rpmbuild/SPECS
# rpmbuild -bp --target=`uname -m` ./kernel.spec

This will end with:

...
gpg: Total number processed: 1
gpg:               imported: 1
+ gpg --homedir . --export --keyring ./kernel.pub Red
gpg: WARNING: unsafe permissions on homedir `.'
+ gcc -o scripts/bin2c scripts/bin2c.c
+ scripts/bin2c ksign_def_public_key __initdata
+ cd ..
+ exit 0

At this point, we now have kernel souce, with all the RHEL patches applied, residing in the directory /build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64

Patch the kernel source with the Lustre code.

Add a unique build id so we can be certain our kernel is booted. Edit ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/Makefile and modify line 4, the EXTRAVERSION to read:
```
EXTRAVERSION = .lustre21
```
enter the directory ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/
```
# cd ~build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64/
```
overwrite the .config file with /build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
```
# cp /build/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config
```

link the Lustre series and patches

# ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series
# ln -s ~/lustre-release/lustre/kernel_patches/patches patches

Apply the patches to the kernel source using quilt

# quilt push -av
...
...
patching file fs/jbd2/journal.c
Hunk #1 succeeded at 94 (offset 4 lines).
patching file fs/jbd2/transaction.c

Applying patch patches/jbd2-commit-timer-no-jiffies-rounding-rhel6.patch
patching file fs/jbd2/transaction.c

Now at patch patches/jbd2-commit-timer-no-jiffies-rounding-rhel6.patch

Build the new kernel as an RPM.

Go into the kernel source directory and issue the following commands to build a kernel rpm.

# cd /build/kernel/rpmbuild/BUILD/kernel-2.6.32-71.29.1.el6/linux-2.6.32-71.29.1.el6.x86_64
# make oldconfig || make menuconfig
# make include/asm
# make include/linux/version.h
# make SUBDIRS=scripts
# make include/linux/utsrelease.h
# make rpm

A successful build will return:

...
...
Wrote: /build/kernel/rpmbuild/SRPMS/kernel-2.6.32lustre21-1.src.rpm
Wrote: /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V
+ umask 022
+ cd /build/kernel/rpmbuild/BUILD
+ cd kernel-2.6.32lustre21
+ rm -rf /build/kernel/rpmbuild/BUILDROOT/kernel-2.6.32.lustre21-1.x86_64
+ exit 0
rm ../kernel-2.6.32lustre21.tar.gz

NOTE If you receive a request to generate more entropy, you need to trigger some disk I/O or keyboard I/O. I would recommend (in another terminal):

# grep -Ri 'whamcloud' /usr

At this point, you should have a fresh kernel RPM /build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustre21-1.x86_64.rpm

Configure and build Lustre

Configure Lustre source

# cd ~/lustre-release/
# ./configure --with-linux=/build/kernel/rpmbuild/BUILD/kernel-2.6.32.lustre21/
...
...
LLCPPFLAGS:    -D__arch_lib__ -D_LARGEFILE64_SOURCE=1
CFLAGS:        -g -O2 -Werror
EXTRA_KCFLAGS: -include /build/lustre-release/config.h  -g -I/build/lustre-release/libcfs/include -I/build/lustre-release/lnet/include -I/build/lustre-release/lustre/include
LLCFLAGS:      -g -Wall -fPIC -D_GNU_SOURCE

Type 'make' to build Lustre.

make rpms:

# make rpms
...
...
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD
+ umask 022
+ cd /build/kernel/rpmbuild/BUILD
+ cd lustre-2.0.61
+ rm -rf /build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustre21_g0533e7b.x86_64
+ exit 0
make[1]: Leaving directory `/build/lustre-release'

You should now have build the following, similarly named, rpms:

# ls ~build/kernel/rpmbuild/RPMS/x86_64/
kernel-2.6.32lustre21-1.x86_64.rpm
lustre-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-debuginfo-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-ldiskfs-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-modules-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-source-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm
lustre-tests-2.0.61-2.6.32.lustre21_g0533e7b.x86_64.rpm

Installing the Lustre kernel and rebooting.

As root, Install the kernel

# rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32lustre21-1.x86_64.rpm

Create initrd using dracut

# /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32lustre21

reboot

view the login prompt with satisfaction:

Red Hat Enterprise Linux Server release 6.0 (Santiago)
Kernel 2.6.32lustre21 on an x86_64

client-10 login:

Installing e2fsprogs

e2fsprogs is needed to run the test suite.

Download e2fsprogs from http://downloads.whamcloud.com/public/e2fsprogs/latest/

Install with

# rpm -Uvh ./e2fsprogs-1.41.90.wc2-7.el6.x86_64.rpm  ./e2fsprogs-libs-1.41.90.wc2-7.el6.x86_64.rpm

Installing Lustre.

Change to root and Change directory into /build/kernel/rpmbuild/RPMS/x86_64/

Install modules lustre-modules and user space tools lustre-

# rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustre21*
# rpm -ivh lustre-modules-2.0.61-2.6.32.lustre21*
# rpm -ivh lustre-2.0.61-2.6.32.lustre21_*
# rpm -ivh lustre-tests-*

Testing

run /usr/lib64/lustre/tests/llmount.sh

# /usr/lib64/lustre/tests/llmount.sh
Loading modules from /usr/lib64/lustre/tests/..
debug=0x33f0404
subsystem_debug=0xffb7e3ff
gss/krb5 is not supported
Formatting mgs, mds, osts
Format mds1: /tmp/lustre-mdt1
Format ost1: /tmp/lustre-ost1
Format ost2: /tmp/lustre-ost2
Checking servers environments
Checking clients rhel6_build environments
Loading modules from /usr/lib64/lustre/tests/..
debug=0x33f0404
subsystem_debug=0xffb7e3ff
gss/krb5 is not supported
Setup mgs, mdt, osts
Starting mds1: -o loop,user_xattr,acl  /tmp/lustre-mdt1 /mnt/mds1
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-MDT0000
Starting ost1: -o loop  /tmp/lustre-ost1 /mnt/ost1
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-OST0000
Starting ost2: -o loop  /tmp/lustre-ost2 /mnt/ost2
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-OST0001
Starting client: rhel5_build: -o user_xattr,acl,flock rhel6_build@tcp:/lustre /mnt/lustre
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Using TIMEOUT=20
disable quota as required

you will now have a Lustre filesystem available at /mnt/lustre
NOTE: if you receive an error: mkfs.lustre: Can't parse NID 'rhel6_build@tcp' you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.

Page tree