This walk-thru is targeting developers who want to explore the bleading edge of Lustre. If you are evalulating Lustre for production, you should choose a Lustre Release. |
Purpose
Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the Whamcloud MASTER branch on a x86_64, RHEL 6.3 machine.
Prerequisite
<DIV class="panel" style="border-width: 1px; width:250px; float:right; text-align: left;padding:10px;"><DIV class="panelHeader;text-align: left;" mce_style="border-bottom-width: 1px;" style="border-bottom-width: 1px;"><B>Prerequsiste: Build dependencies</B></DIV><DIV class="panelContent;text-align: left;">
<P>
One of the challenges in writing this walk-thru is making it applicable across apparently small variations in distributions. For example, between RHEL and CentOS, additional tools may be packaged (like newt-devel). As a result, you may find you are able to complete this walk-thru on an OS other than RHEL6.3: please leave a comment and include your platform to alert future visitors if this is the case.</P>
</DIV></DIV>
|
- A newly installed RHEL 6.3 x86_64 machine connected to the internet. This walk-thru assumes the hostname:
rhel63-master
.
- EPEL Repository: this is a convenient source for git.
NOTE the EPEL 5 repository is used because it includes quilt.
- Have the system registered with Redhat Network.
- NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
- NOTE Verify that selinux is disabled.
Overview
Patches are available in the Whamcloud git source repository. A test suite is included with the Lustre 2.1 source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system. Note that a patched kernel is NOT needed for the Lustre client.
Procedure
The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.
Provision machine and installing dependencies.
Once RHEL 6.3 is newly installed on rhel6-master login as root.
- Install the kernel development tools.
# yum -y groupinstall "Development Tools"
|
It has been reported the Development Tools group is not available for some reason. If you experience this problem, you may find the following list if individual packages necessary to install.
# yum -y install automake xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed rpm-build make gcc redhat-rpm-config patchutils
|
|
- Install additional dependencies
# yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed
|
- Install EPEL 5
NOTE EPEL5 is used because it contains quilt.
# wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
# rpm -ivh ./epel-release-5-4.noarch.rpm
|
- Install quilt
newt-devel may not be available if you are using RHEL6. One option is to download newt-devel, slang-devel, and asciidoc RPMs from CentOS and install with:
yum --nogpgcheck localinstall ./newt-devel-0.52.11-3.el6.x86_64.rpm ./slang-devel-2.2.1-1.el6.x86_64.rpm ./asciidoc-8.4.5-4.1.el6.noarch.rpm
|
|
Preparing the Lustre source.
- Create a user
build
with the home directory /home/build
- Switch to the user
build
and change to the build $HOME
directory.
- Get the MASTER branch from Whamcloud git.
# git clone git://git.whamcloud.com/fs/lustre-release.git
# cd lustre-release
|
- Run
sh ./autogen.sh
- Resolve any outstanding dependencies until
autogen.sh
completes successfully. Success will look like:
# sh ./autogen.sh
Checking for a complete tree...
checking for automake-1.9 >= 1.9... found 1.9.6
...
...
configure.ac:10: installing `./config.sub'
configure.ac:12: installing `./install-sh'
configure.ac:12: installing `./missing'
Running autoconf
|
Prepare the kernel source
In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.
- Get the kernel source. First create the directory structure, then get the source from the RPM. Create a
.rpmmacros
file to install the kernel source in our user dir.
# cd $HOME
# mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
# cd kernel
# echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros
|
- Install the kernel source:
# rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-131.2.1.el6.src.rpm 2>&1 | grep -v mockb
|
Red Hat periodically release updates to the kernel. The Lustre Master branch tracks the most recent kernel from Red Hat. In the event that the link above is not completely up-to-date, you should visit the Red Hat source RPM download site and manually ensure you are downloading the most recent kernel. |
- Prepare the source using rpmbuild.
# cd ~/kernel/rpmbuild
# rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
|
This will end with:
...
gpg: Total number processed: 1
gpg: imported: 1
+ gpg --homedir . --export --keyring ./kernel.pub Red
gpg: WARNING: unsafe permissions on homedir `.'
+ gcc -o scripts/bin2c scripts/bin2c.c
+ scripts/bin2c ksign_def_public_key __initdata
+ cd ..
+ exit 0
|
At this point, we now have kernel souce, with all the RHEL patches applied, residing in the directory /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
Patch the kernel source with the Lustre code.
- Add a unique build id so we can be certain our kernel is booted. Edit
~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/Makefile
and modify line 4, the EXTRAVERSION
to read:
EXTRAVERSION = .lustremaster
|
- enter the directory
~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
# cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
|
- overwrite the
.config
file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config
# cp ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config
|
- link the Lustre series and patches
# ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series
# ln -s ~/lustre-release/lustre/kernel_patches/patches patches
|
- Apply the patches to the kernel source using quilt
# quilt push -av
...
...
patching file fs/jbd2/transaction.c
Hunk #3 succeeded at 1222 (offset 3 lines).
Hunk #4 succeeded at 1357 (offset 3 lines).
Now at patch patches/jbd2-jcberr-2.6-rhel6.patch
|
Build the new kernel as an RPM.
- Go into the kernel source directory and issue the following commands to build a kernel rpm.
# cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/
# make oldconfig || make menuconfig
# make include/asm
# make include/linux/version.h
# make SUBDIRS=scripts
# make include/linux/utsrelease.h
# make rpm
|
Whamcloud are currently tracking a bug against this section of the walk-thru. This bug means that you may have to issue make before make rpm to continue without error. The bug is LU-788. |
- A successful build will return:
...
...
Wrote: /home/build/kernel/rpmbuild/SRPMS/kernel-2.6.32lustremaster-1.src.rpm
Wrote: /home/build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustremaster-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V
+ umask 022
+ cd /home/build/kernel/rpmbuild/BUILD
+ cd kernel-2.6.32lustremaster
+ rm -rf /home/build/kernel/rpmbuild/BUILDROOT/kernel-2.6.32.lustremaster-1.x86_64
+ exit 0
rm ../kernel-2.6.32lustremaster.tar.gz
|
If you receive a request to generate more entropy, you need to trigger some disk I/O or keyboard I/O. In another terminal, you can execute the following command to generate entropy:
# grep -Ri 'whamcloud' /usr
|
|
At this point, you should have a fresh kernel RPM ~/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustremaster-1.x86_64.rpm
Configure and build Lustre
- Configure Lustre source
# cd ~/lustre-release/
# ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.lustremaster/
...
...
LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1
CFLAGS: -g -O2 -Werror
EXTRA_KCFLAGS: -include /home/build/lustre-release/config.h -g -I/home/build/lustre-release/libcfs/include -I/home/build/lustre-release/lnet/include -I/home/build/lustre-release/lustre/include
LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE
Type 'make' to build Lustre.
|
- make rpms:
# make rpms
...
...
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD
+ umask 022
+ cd /home/build/kernel/rpmbuild/BUILD
+ cd lustre-2.0.61
+ rm -rf /home/build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustremaster_g0533e7b.x86_64
+ exit 0
make[1]: Leaving directory `/home/build/lustre-release'
|
- You should now have build the following, similarly named, rpms:
# ls ~build/kernel/rpmbuild/RPMS/x86_64/
kernel-2.6.32lustremaster-1.x86_64.rpm
lustre-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-debuginfo-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-ldiskfs-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-modules-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-source-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
lustre-tests-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
|
Installing the Lustre kernel and rebooting.
- As root, Install the kernel
# rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32lustremaster-1.x86_64.rpm
|
- Create initrd using
dracut
# /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32lustremaster
|
- optional turn on lustre services, and specify net for lnet
* chkconfig lustre on
* vi /etc/modprobe.d/lustre.conf
|
reboot
- view the login prompt with satisfaction:
Red Hat Enterprise Linux Server release 6.0 (Santiago)
Kernel 2.6.32lustremaster on an x86_64
client-10 login:
|
Installing e2fsprogs
e2fsprogs is needed to run the test suite.
- Download e2fsprogs from http://downloads.whamcloud.com/public/e2fsprogs/latest/
- Install with
# rpm -Uvh ./e2fsprogs-1.41.90.wc2-7.el6.x86_64.rpm ./e2fsprogs-libs-1.41.90.wc2-7.el6.x86_64.rpm
|
Installing Lustre.
- Change to
root
and Change directory into /home/build/kernel/rpmbuild/RPMS/x86_64/
- Install modules
lustre-modules
and user space tools lustre-
# rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustremaster*
# rpm -ivh lustre-modules-2.0.61-2.6.32.lustremaster*
# rpm -ivh lustre-2.0.61-2.6.32.lustremaster_*
# rpm -ivh lustre-tests-*
|
Testing
- run
/usr/lib64/lustre/tests/llmount.sh
# /usr/lib64/lustre/tests/llmount.sh
Loading modules from /usr/lib64/lustre/tests/..
debug=0x33f0404
subsystem_debug=0xffb7e3ff
gss/krb5 is not supported
Formatting mgs, mds, osts
Format mds1: /tmp/lustre-mdt1
Format ost1: /tmp/lustre-ost1
Format ost2: /tmp/lustre-ost2
Checking servers environments
Checking clients rhel6-master environments
Loading modules from /usr/lib64/lustre/tests/..
debug=0x33f0404
subsystem_debug=0xffb7e3ff
gss/krb5 is not supported
Setup mgs, mdt, osts
Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-MDT0000
Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-OST0000
Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Started lustre-OST0001
Starting client: rhel5-build: -o user_xattr,acl,flock rhel6-master@tcp:/lustre /mnt/lustre
debug=0x33f0404
subsystem_debug=0xffb7e3ff
debug_mb=10
Using TIMEOUT=20
disable quota as required
|
- you will now have a Lustre filesystem available at
/mnt/lustre
- NOTE: if you receive an error:
mkfs.lustre: Can't parse NID 'rhel6-master@tcp'
you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.
ENDS~