Page History

...

Once RHEL 6.3 is newly installed on rhel6-master login as root.

1. Install the kernel development tools.

   Code Block
   # yum -y groupinstall "Development Tools"

   Info
   title Problem with installing 'Development Tools'
   It has been reported the Development Tools group is not available for some reason. If you experience this problem, you may find the following list if individual packages necessary to install. Code Block # yum -y install automake xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed rpm-build make gcc redhat-rpm-config patchutils

2. Install additional dependencies

   Code Block
   # yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed

3. Install EPEL 5
   NOTEEPEL5 is used because it contains quilt.

   Code Block
   # wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm # rpm -ivh ./epel-release-5-4.noarch.rpm

4. Install quilt

   Code Block
   # yum -y install quilt

   Note
   title newt-devel
   newt-devel may not be available if you are using RHEL6. One option is to download newt-devel, slang-devel, and asciidoc RPMs from CentOS and install with: Code Block yum --nogpgcheck localinstall ./newt-devel-0.52.11-3.el6.x86_64.rpm ./slang-devel-2.2.1-1.el6.x86_64.rpm ./asciidoc-8.4.5-4.1.el6.noarch.rpm

Preparing the Lustre source.

1. Create a user build with the home directory /home/build

   Code Block
   # useradd -m build

2. Switch to the user build and change to the build $HOMEdirectory.

   Code Block
   # su build # cd $HOME

3. Get the MASTER branch from Whamcloud git.

   Code Block
   # git clone git://git.whamcloud.com/fs/lustre-release.git # cd lustre-release

4. Run sh ./autogen.sh

5. Resolve any outstanding dependencies until autogen.shcompletes successfully. Success will look like:

   Code Block
   # sh ./autogen.sh Checking for a complete tree... checking for automake-1.9 >= 1.9... found 1.9.6 ... ... configure.ac:10: installing `./config.sub' configure.ac:12: installing `./install-sh' configure.ac:12: installing `./missing' Running autoconf

...

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

1. Get the kernel source. First create the directory structure, then get the source from the RPM. Create a .rpmmacrosfile to install the kernel source in our user dir.

   Code Block
   # cd $HOME # mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} # cd kernel # echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

2. Install the kernel source:

   Code Block
   # rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/kernel-2.6.32-131.2.1.el6.src.rpm 2>&1 | grep -v mockb

   Note
   title RHEL kernel versions
   Red Hat periodically release updates to the kernel. The Lustre Master branch tracks the most recent kernel from Red Hat. In the event that the link above is not completely up-to-date, you should visit the Red Hat source RPM download site and manually ensure you are downloading the most recent kernel.

3. Prepare the source using rpmbuild.

   Code Block
   # cd ~/kernel/rpmbuild # rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

   
   This will end with:

   Code Block
   ... gpg: Total number processed: 1 gpg: imported: 1 + gpg --homedir . --export --keyring ./kernel.pub Red gpg: WARNING: unsafe permissions on homedir `.' + gcc -o scripts/bin2c scripts/bin2c.c + scripts/bin2c ksign_def_public_key __initdata + cd .. + exit 0

...

Patch the kernel source with the Lustre code.

1. Add a unique build id so we can be certain our kernel is booted. Edit ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/Makefile and modify line 4, the EXTRAVERSIONto read:

   Code Block
   EXTRAVERSION = .lustremaster

2. enter the directory ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/

   Code Block
   # cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/

3. overwrite the .config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config

   Code Block
   # cp ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-2.6.32-2.6-rhel6-x86_64.config ./.config

4. link the Lustre series and patches

   Code Block
   # ln -s ~/lustre-release/lustre/kernel_patches/series/2.6-rhel6.series series # ln -s ~/lustre-release/lustre/kernel_patches/patches patches

5. Apply the patches to the kernel source using quilt

   Code Block
   # quilt push -av ... ... patching file fs/jbd2/transaction.c Hunk #3 succeeded at 1222 (offset 3 lines). Hunk #4 succeeded at 1357 (offset 3 lines). Now at patch patches/jbd2-jcberr-2.6-rhel6.patch

Build the new kernel as an RPM.

1. Go into the kernel source directory and issue the following commands to build a kernel rpm.

   Code Block
   # cd ~/kernel/rpmbuild/BUILD/kernel-2.6.32-131.2.1.el6/linux-2.6.32-131.2.1.el6.x86_64/ # make oldconfig || make menuconfig # make include/asm # make include/linux/version.h # make SUBDIRS=scripts # make include/linux/utsrelease.h # make rpm
   Note
   title BUG LU-788
   Whamcloud are currently tracking a bug against this section of the walk-thru. This bug means that you may have to issue make before make rpm to continue without error. The bug is LU-788.

2. A successful build will return:

   Code Block

   ... ... Wrote: /home/build/kernel/rpmbuild/SRPMS/kernel-2.6.32lustremaster-1.src.rpm Wrote: /home/build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustremaster-1.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.f73m1V + umask 022 + cd /home/build/kernel/rpmbuild/BUILD + cd kernel-2.6.32lustremaster + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/kernel-2.6.32.lustremaster-1.x86_64 + exit 0 rm ../kernel-2.6.32lustremaster.tar.gz

# grep -Ri 'whamcloud' /usr

...

Configure and build Lustre

1. Configure Lustre source

   Code Block

   # cd ~/lustre-release/ # ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.lustremaster/ ... ... LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1 CFLAGS: -g -O2 -Werror EXTRA_KCFLAGS: -include /home/build/lustre-release/config.h -g -I/home/build/lustre-release/libcfs/include -I/home/build/lustre-release/lnet/include -I/home/build/lustre-release/lustre/include LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE Type 'make' to build Lustre.

2. make rpms:

   Code Block
   # make rpms ... ... Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD + umask 022 + cd /home/build/kernel/rpmbuild/BUILD + cd lustre-2.0.61 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustremaster_g0533e7b.x86_64 + exit 0 make[1]: Leaving directory `/home/build/lustre-release'

3. You should now have build the following, similarly named, rpms:

   Code Block

   # ls ~build/kernel/rpmbuild/RPMS/x86_64/ kernel-2.6.32lustremaster-1.x86_64.rpm lustre-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-debuginfo-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-ldiskfs-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-modules-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-source-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-tests-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm

Installing the Lustre kernel and rebooting.

1. As root, Install the kernel

   Code Block
   # rpm -ivh ~build/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32lustremaster-1.x86_64.rpm

2. Create initrd using dracut

   Code Block
   # /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32lustremaster

3. optional turn on lustre services, and specify net for lnet

   Code Block
   * chkconfig lustre on * vi /etc/modprobe.d/lustre.conf

4. reboot

5. view the login prompt with satisfaction:

   Code Block
   Red Hat Enterprise Linux Server release 6.0 (Santiago) Kernel 2.6.32lustremaster on an x86_64 client-10 login:

...

1. Download e2fsprogs from http://downloads.whamcloud.com/public/e2fsprogs/latest/

2. Install with

   Code Block
   # rpm -Uvh ./e2fsprogs-1.41.90.wc2-7.el6.x86_64.rpm ./e2fsprogs-libs-1.41.90.wc2-7.el6.x86_64.rpm

...

1. Change to root and Change directory into /home/build/kernel/rpmbuild/RPMS/x86_64/

2. Install modules lustre-modules and user space tools lustre-

   Code Block
   # rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustremaster* # rpm -ivh lustre-modules-2.0.61-2.6.32.lustremaster* # rpm -ivh lustre-2.0.61-2.6.32.lustremaster_* # rpm -ivh lustre-tests-*

Testing

1. run /usr/lib64/lustre/tests/llmount.sh

   Code Block

   # /usr/lib64/lustre/tests/llmount.sh Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6-master environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel5-build: -o user_xattr,acl,flock rhel6-master@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required

2. you will now have a Lustre filesystem available at /mnt/lustre
3. NOTE: if you receive an error: mkfs.lustre: Can't parse NID 'rhel6-master@tcp' you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.

...