<<< In progress >>
This walk-thru is targeting developers who want to explore the bleeding edge of Lustre. If you are evaluating Lustre for production, you should choose a Lustre Release.
Purpose
Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the HPDD master branch on a x86_64, RHEL/CentOS 7.3 machine.
Prerequisite
- A newly installed RHEL/CentOS 7.3 x86_64 machine connected to the internet.
- EPEL Repository: this is a convenient source for git.
- NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
- NOTE Verify that selinux is disabled.
Overview
pre-built RPMs are available
Lustre servers require a patched and compiled kernel. A patched and compiled Lustre server kernel is available from Intel. A separate page is available to walk thru setting up Lustre with these pre-built RPMs. This document is for those who wish to build their Lustre system from source. Note that if you are not modifying the kernel patches on the server, it is possible to use the pre-built Lustre server kernel RPMs, and only build the Lustre code. Note that a patched kernel is NOT needed for the Lustre client.
Patches are available in the HPDD Git source repository. A test suite is included with the Lustre source. This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.
Procedure
The procedure requires that a OS is setup for development - this includes Lustre sources, kernel source and build tools. Once setup, a new kernel can be patched, compiled, run and tested. Further reading on building a RHEL RPM based kernel is available from, among other sources, the CentOS site.
Provision machine and installing dependencies.
Once RHEL 7.3 is newly installed on rhel6-master login as root.
Install the kernel development tools.
# yum -y groupinstall "Development Tools"
Problem with installing 'Development Tools'
If the Development Tools group is not be available for some reason, you may find the following list if individual packages necessary to install.
# yum -y install automake xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed rpm-build make gcc redhat-rpm-config patchutils git
Install additional dependencies
# yum -y install xmlto asciidoc elfutils-libelf-devel zlib-devel binutils-devel newt-devel python-devel hmaccalc perl-ExtUtils-Embed bison elfutils-devel audit-libs-devel
Install EPEL 7
# rpm -ivh http://download.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
Install quilt and libselinux-devel
# yum -y install pesign numactl-devel pciutils-devel ncurses-devel
Preparing the Lustre source.
Create a user
build
with the home directory/home/build
# useradd -m build
Switch to the user
build
and change to the build$HOME
directory.# su build $ cd $HOME
Get the MASTER branch from HPDD git.
$ git clone git://git.hpdd.intel.com/fs/lustre-release.git $ cd lustre-release
- Run
sh ./autogen.sh
Resolve any outstanding dependencies until
autogen.sh
completes successfully. Success will look like:$ sh ./autogen.sh configure.ac:10: installing 'config/config.guess' configure.ac:10: installing 'config/config.sub' configure.ac:12: installing 'config/install-sh' configure.ac:12: installing 'config/missing' libcfs/libcfs/autoMakefile.am: installing 'config/depcomp' $
Prepare a patched kernel for Lustre
You can have different ways to prepare a patched kernel for Lustre. The easier method is to download built RPM packages from build.hpdd.intel.com. For example, if you're running el7, you should download the packages from here: https://build.hpdd.intel.com/view/Non-Reviews/job/lustre-master/arch=x86_64,build_type=server,distro=el7,ib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/RPMS/x86_64/ at the time when this page is written. You're going to need the packages starting with 'kernel-'. After new kernel packages are downloaded, you can skip the following few steps and go to the section 'Installing the Lustre kernel and rebooting'.
If you want a more challenge life, you can patch the kernel by yourself, in that case, please follow the steps below.
Prepare the kernel source
In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.
Get the kernel source. First create the directory structure, then get the source from the RPM. Create a
.rpmmacros
file to install the kernel source in our user dir.$ cd $HOME $ mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} $ cd kernel $ echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros
Install the kernel source:
$ rpm -ivh http://vault.centos.org/7.3.1611/updates/Source/SPackages/kernel-3.10.0-514.2.2.el7.src.rpm
Prepare the source using rpmbuild.
$ cd ~/kernel/rpmbuild $ rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
This will end with:... + make ARCH=x86_64 oldnoconfig scripts/kconfig/conf --olddefconfig Kconfig # # configuration written to .config # + echo '# x86_64' + cat .config + find . '(' -name '*.orig' -o -name '*~' ')' -exec rm -f '{}' ';' + find . -name .gitignore -exec rm -f '{}' ';' + cd .. + exit 0
At this point, we now have kernel source, with all the RHEL/CentOS patches applied, residing in the directory ~/kernel/rpmbuild/BUILD/kernel-3.10.0-514.2.2.el7/linux-3.10.0-514.2.2.el7.x86_64/
Patch the kernel source with the Lustre code.
Gather all the patches from lustre tree into a single file
$ cd ~ $ rm -f ~/lustre-kernel-x86_64-lustre.patch $ cd ~/lustre-release/lustre/kernel_patches/series $ for patch in $(<"3.10-rhel7.series"); do \ patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}" \ cat "${patch_file}" >> $HOME/lustre-kernel-x86_64-lustre.patch" \ done $
Copy the lustre kernel patch into RPM build tree
# cp ~/lustre-kernel-x86_64-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-3.10.0-lustre.patch
Edit the kernel spec file by vi ~/kernel/rpmbuild/SPEC/kernel.spec
Find the line with 'find $RPM_BUILD_ROOT/lib/modules/$KernelVer' and insert following two lines below it
cp -a fs/ext3/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext3
cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4
Find the line with '# empty final patch to facilitate testing of kernel patches' and insert following two lines below it
# adds Lustre patches
Patch99995: patch-%{version}-lustre.patch
Find the line with 'ApplyOptionalPatch linux-kernel-test.patch' and insert following two lines below it
# lustre patch
ApplyOptionalPatch patch-%{version}-lustre.patch
Find the line with '%define listnewconfig_fail 1' and change 1 to 0
Save and close the spec file.
Overwrite the kernel
config
file with~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-3.10.0-3.10-rhel7-x86_64.config
echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-3.10.0-x86_64.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-3.10.0-3.10-rhel7-x86_64.config >> ~/kernel/rpmbuild/SOURCES/kernel-3.10.0-x86_64.config
Build the new kernel as an RPM.
Start building lustre kernel with rpmbuild
$ cd ~/kernel/rpmbuild $ buildid="_lustre" # Note: change to any string that identify your work $ rpmbuild -ba --with firmware --target x86_64 --with baseonly \ --define "buildid ${buildid}" \ ~/kernel/rpmbuild/SPECS/kernel.spec
A successful build will return
... ... Wrote: /mnt/home/build/kernel/rpmbuild/SRPMS/kernel-3.10.0-514.2.2.el7_lustre.src.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-headers-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-debuginfo-common-x86_64-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/perf-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/perf-debuginfo-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/python-perf-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/python-perf-debuginfo-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-tools-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-tools-libs-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-tools-libs-devel-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-tools-debuginfo-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-devel-3.10.0-514.2.2.el7_lustre.x86_64.rpm Wrote: /mnt/home/build/kernel/rpmbuild/RPMS/x86_64/kernel-debuginfo-3.10.0-514.2.2.el7_lustre.x86_64.rpm Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.F7X9cL + umask 022 + cd /mnt/home//build/kernel/rpmbuild/BUILD + cd kernel-3.10.0-514.2.2.el7 + rm -rf /mnt/home/build/kernel/rpmbuild/BUILDROOT/kernel-3.10.0-514.2.2.el7_lustre.x86_64 + exit 0
If you receive a request to generate more entropy, you need to trigger some disk I/O or keyboard I/O. In another terminal, you can either type randomly or execute the following command to generate entropy:
# grep -Ri 'intel' /usr
At this point, you should have a fresh kernel RPM ~/kernel/rpmbuild/RPMS/x86_64/kernel-2.6.32.lustremaster-1.x86_64.rpm
Installing the Lustre kernel and rebooting.
As root, Install the kernel
# rpm -ivh $PKG_PATH/kernel-*.rpm
Depending on how you got your kernel packages, the PKG_PATH should be ~build/kernel/rpmbuild/RPMS/x86_64 if you built the packages by yourself, or any other directory where you downloaded the packages from build.hpdd.intel.com.
Create initrd using
dracut(*This may not be required because initrd should have been created by installing new kernel*)
# /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --install 2.6.32.431.5.1.el6_lustre
optional turn on lustre services, and specify net for lnet
* chkconfig lustre on * vi /etc/modprobe.d/lustre.conf
If you don't know what should be written to this file, just leave it empty for now.
reboot
view the login prompt with satisfaction, and make sure that new kernel is running:
Red Hat Enterprise Linux Server release 6.0 (Santiago) Kernel 2.6.32l-'I'm new kernel' on an x86_64 client-10 login:
Configure and build Lustre
Configure Lustre source
# cd ~/lustre-release/ # ./configure --with-linux=/lib/modules/kernel-2.6.32_lustremaster/build ... ... LLCPPFLAGS: -D__arch_lib__ -D_LARGEFILE64_SOURCE=1 CFLAGS: -g -O2 -Werror EXTRA_KCFLAGS: -include /home/build/lustre-release/config.h -g -I/home/build/lustre-release/libcfs/include -I/home/build/lustre-release/lnet/include -I/home/build/lustre-release/lustre/include LLCFLAGS: -g -Wall -fPIC -D_GNU_SOURCE Type 'make' to build Lustre.
make rpms:
# make rpms ... ... Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.TsLWpD + umask 022 + cd /home/build/kernel/rpmbuild/BUILD + cd lustre-2.0.61 + rm -rf /home/build/kernel/rpmbuild/BUILDROOT/lustre-2.0.61-2.6.32_lustremaster_g0533e7b.x86_64 + exit 0 make[1]: Leaving directory `/home/build/lustre-release'
You should now have build the following, similarly named, rpms:
# ls *.rpm kernel-2.6.32lustremaster-1.x86_64.rpm lustre-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-debuginfo-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-ldiskfs-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-ldiskfs-debuginfo-3.3.0-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-modules-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-source-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm lustre-tests-2.0.61-2.6.32.lustremaster_g0533e7b.x86_64.rpm
Installing e2fsprogs
e2fsprogs is needed to run the test suite.
- Download e2fsprogs from http://downloads.hpdd.intel.com/public/e2fsprogs/latest/
Install with
# rpm -Uvh ./e2fsprogs-1.42.6.wc2-7.el6.x86_64.rpm ./e2fsprogs-libs-1.42.6.wc2-7.el6.x86_64.rpm
Installing Lustre.
- Change to
root
and Change directory into ~
lustre-release/build/
Install modules
lustre-modules
and user space toolslustre-
# rpm -ivh lustre-ldiskfs-3.3.0-2.6.32.lustremaster* # rpm -ivh lustre-modules-2.0.61-2.6.32.lustremaster* # rpm -ivh lustre-2.0.61-2.6.32.lustremaster_* # rpm -ivh lustre-tests-*
Disable SELinux (Lustre Servers)
SELinux, which is on by default in RHEL/CentOS, will prevent the format commands for the various Lustre targets from completing. Therefore you must either disable it or adjust the settings. These instructions explain how to disable it.
- Run getenforce to see if SELinux is enabled. It should return 'Enforcing' or 'Disabled'.
- To disable it, vi /etc/selinux/config and change the line 'selinux=enforcing' to 'selinux=disabled'.
Finally, reboot your system.
# vi /etc/selinux/config ---- # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of these two values: # targeted - Only targeted network daemons are protected. # strict - Full SELinux protection. SELINUXTYPE=targeted --- # shutdown -r now
Testing
run
/usr/lib64/lustre/tests/llmount.sh
# /usr/lib64/lustre/tests/llmount.sh Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients rhel6-master environments Loading modules from /usr/lib64/lustre/tests/.. debug=0x33f0404 subsystem_debug=0xffb7e3ff gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o loop,user_xattr,acl /tmp/lustre-mdt1 /mnt/mds1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-MDT0000 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Started lustre-OST0001 Starting client: rhel5-build: -o user_xattr,acl,flock rhel6-master@tcp:/lustre /mnt/lustre debug=0x33f0404 subsystem_debug=0xffb7e3ff debug_mb=10 Using TIMEOUT=20 disable quota as required
- you will now have a Lustre filesystem available at
/mnt/lustre
- NOTE: if you receive an error:
mkfs.lustre: Can't parse NID 'rhel6-master@tcp'
you'll need to associate the IP address of a non-loopback interface with name of your machine into the /etc/hosts file.