Versions Compared

Old Version 29

changes.mady.by.user Andreas Dilger

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Dilger

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Update to Ubuntu 18.04
Note

This walk-through is targeting developers who want to explore the bleeding edge of Lustre or build on CPUs/kernel combinations not currently supported by the automated builders.

If you are evaluating Lustre for production, you should choose a stable Lustre Release.

Do note Note that as of the writing of this page, a pre-compiled Lustre Release is only available for Ubuntu as a Lustre Client, not Server.

...

Prerequisites

  • A newly installed Ubuntu 1416.04 or 1618.04 machine connected to the internet.
  • NOTE: The build instructions below assume that SE-Linux is disabled. If you enable it, this guide should only be used as a source of inspiration, not of authoritative information.

...

The main difference in this guide is the versions of ZFS that will be compiled against:

Yes, 14.04 indeed uses a newer version than 16.04. Make of that what you will.

 

Other than that, the following package versions will be used in this guide for both Ubuntu releases:

  • Linux kernel 4.415.0-45.66
  • Lustre 2.9.x (or later) with Ubuntu-Server Enablement patches integrated:
  • 32

1.) 1.) Installing the build dependency packages

...

sudo apt-get build-dep linux-image-$(uname -r)
sudo apt-get install build-essential debhelper devscripts fakeroot kernel-wedge libudev-dev pciutils-dev
sudo apt-get install texinfo xmlto libelf-dev python-dev liblzma-dev libaudit-dev dh-systemd libyaml-dev
sudo apt-get install module-assistant libreadline-dev dpatch libsnmp-dev quilt

...


To enable ZFS -server support for Ubuntu 14.04., you need to add the Ubuntu-ZFS PPA.
...
Following that, you need to run the following on both Ubuntu 1416.04. as well as 1618.04:
sudo apt-get build-dep ubuntu-zfs spl-linuxdpkg zfs-linux
...
dpkg
To enable LDISKFS server support, you need to add the following packages:
...
NOTE: You only have to install these on your compile node if you are building Ubuntu server . Your support. If you are only building Lustre for client nodes, and the final servers and clients do not need any of these packages. 
The following instructions assume that you run in a directory that has at least 20 GB free disk space and is writable by your user.
It will be assumed, that this path is available as ${BUILDPATH}, such as /usr/src/lustre.
2.) Building the Linux Kernel
Before you can build Lustre server suppor, you will always need to have a compiled kernel source directory available, for your intended kernel version.  If you are building only a client, then you do NOT need to build a kernel, only install the kernel headers and config file.
Depending on which backing filesystem you intend to use, you must then do the Depending on which backing filesystem you intend to use, you must then do the following:
  • If you intend to use LDISKFS, you need to use a specific kernel version to generate the LDISKFS modules, and optionally patch it with performance and functionality improvements.
  • If you are only using ZFS, you do not need to apply any changes to the kernel and can use any version you want, but you will still need to compile it.
...
In either case, you will have to download the correct kernel for your distribution. You can choose between using the GIT or HTTP protocol for each.
 
2.1.) (ZFS, optionally LDISKFS) Building an Unmodified Kernel
...
As such, first figure out the version of the kernel on your target servers by executing:
[bash-4.3]$ uname -r
4.415.0-4532-generic
You can also target any other kernel version, but in the following we will assume you target "4.415.0-4535"
The first thing to do is to go into the kernel source directory and check out that version:
[bash-4.3]$ cd ${BUILDPATH}/ubuntu-kernel
[bash-4.3]$ git tag | grep 4.415.0-4532
Ubuntu-lts-4.415.0-4532.66_1418.04.1
[bash-4.3]$ git checkout Ubuntu-lts-4.415.0-4532.66_1418.04.1
Do note that this tag will be different for Ubuntu 1416.04 and 1618.04. As you can see here, we're using Ubuntu 1418.04. 

The next thing you need is the kernel configuration for that version. The easiest way to get it, is by grabbing it from a server running that version.
Assuming the current compile server already runs that kernel (which is not necessary, but certainly makes things easier):
cp /boot/config-4.415.0-4532-generic .config
An alternative source of this config file is the Debian package for that kernel, which you can download over the Internet.
The file you seek is in the linux-image-*.deb file – where the value of "*" depends on the kernel version and whether you use the lowlatency kernel or not.
For example:

Once you have the config file in place, you need to make sure that the same version number as used by Ubuntu is embedded in the kernel:
touch .scmversion
sed -i 's/^VERSION = .*/VERSION = 4/' Makefile
sed -i 's/^PATCHLEVEL = .*/PATCHLEVEL = 4/' Makefile
sed -i 's/^SUBLEVEL = .*/SUBLEVEL = 0/' Makefile
sed -i 's/^EXTRAVERSION =.*/EXTRAVERSION = -4535-generic/' Makefile
Finally, you can compile the kernel source so that it is usable by Lustre:
...
Note: "nproc" returns the number of CPUs. If you don't have that tool or wish to use a different parallelism, just enter a number of parallel processes there.
 
...
3.) (
...
Optional) Building 
...
The Lustre build system needs to apply two kinds of patches against the kernel to enable LDISKFS:
  1. General performance improvement patches
  2. Conversion Patches that turn EXT4 into LDISKFS
...
ZFS packages
If you wish to compile Lustre so that it can use ZFS as a backing storage, the current compile server does not need to have the ZFS modules installed.
All that you need to do is, it to have the ZFS source code compiled, so that Lustre can link against its modules.
While it should be possible to compile against a subtly different ZFS version than what you will use later on the target system, the safest way is to use the same version.
First, create a directory that will hold both SPL and ZFS.
cd ${BUILDPATH}
mkdir zfs
cd zfs
Then, download the correct ZFS source code for your target Ubuntu version:
Ubuntu 16.04:
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.7.9.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.7.9-3ubuntu6.debian.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.7.9.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.7.9-3ubuntu2.debian.tar.xz
Ubuntu 16.04:
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.7.12.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.7.12-1ubuntu3.debian.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.7.12.orig.tar.gz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.7.12-1ubuntu1.debian.tar.xz
The rest of the guide assumes you chose the Ubuntu 18.04. version. Adjust the file- and pathnames accordingly, if you are targeting 16.04.

Extract, patch and compile SPL (adjust the ZFSVERSION parameter accordingly):
ZFSVERSION=0.7.12
tar -xf spl-linux_${ZFSVERSION}.orig.tar.bz2
cd spl-linux-${ZFSVERSION}
tar -xf ../spl-linux_${ZFSVERSION}-1ubuntu1.debian.tar.gz
for f in debian/patches/*.patch; do patch -Np1 -i $f; done
./autogen.sh &&
./configure --with-linux=${BUILDPATH}/ubuntu-kernel &&
make -j $(nproc)
Then, do the same for ZFS:
cd ${BUILDPATH}/zfs
tar -xf zfs-linux_${ZFSVERSION}.orig.tar.bz2
cd zfs-linux-${ZFSVERSION}
tar -xf ../zfs-linux_${ZFSVERSION}-1~trusty.debian.tar.gz
for f in debian/patches/*.patch; do patch -Np1 -i $f; done
./autogen.sh &&
./configure --with-linux=${BUILDPATH}/ubuntu-kernel --with-spl=${BUILDPATH}/zfs/spl-linux-${ZFSVERSION} &&
make -j $(nproc)

4.) (Optional) Building modified e2fsprogs for LDISKFS support
If you are building an LDISKFS server, you need to compile and install a modified version of the e2fsprogs package, which provides the virtual ldiskfsprogs package.  This is NOT needed if you only intend to build a client, or use ZFS on the server.
To compile the needed package, execute the following steps:
cd ${BUILDPATH}
git clone git://git.whamcloud.com/tools/e2fsprogs.git
cd e2fsprogs
git checkout master-lustre
If you can't use the GIT protocol (e.g. if you're behind a corporate proxy), download a snapshot from:
Then, generate the Ubuntu Debian packages:
wget -P ../ http://archive.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/e2fsprogs_1.44.4-2.debian.tar.xz
tar --exclude "debian/changelog" -xf ../e2fsprogs_1.44.4-2.debian.tar.xz
sudo apt-get install libfuse-dev libattr1-dev libblkid-dev uuid-dev
./configure
dpkg-buildpackage -b -us -uc
This will create several ".deb" files in ${BUILDPATH}.
Save them somewhere sane, so that you can install the modified e2fsprogs on the target servers later.
Note 1: The downloading of the Ubuntu Debian install files is necessary, as the ones form the GIT repo are not Ubuntu specific.
Please make sure to use the same version numbers for the git checkout as well as the Ubuntu download.
Note 2: The 'sed' call works around the build scripts not properly identifying a build output artifact.
This has no effect on running the e2fsprogs, but might cause issues if you use something that compiles against the e2fsprogs, as you will miss that particular header file.
5.) Building Lustre Client & Server Packages
5.1) Downloading Lustre Sources
The first step is to download the most recent Lustre sources from the HPDD Git repo:
cd ${BUILDPATH}
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout b2_10
This checks out the latest 2.10 code. If you need a newer version, adjust the checked-out branch accordingly.
All releases up to (and including) 2.10 do not come with Ubuntu Server support enabled. Later versions should have the needed changes already.

In either case though, you must be aware that enabling LDISKFS is very sensitive to kernel changes, so only a handful of kernel versions are supported.
For Ubuntu only a narrow kernel version range is supported directly by the patches:
  • 4.4.0-45.66 to 4.4.0-85.108
  • This means that on Ubuntu 14.04, you need to use the Xenial (16.04.) HWE kernel line if you need LDISKFS support
Newer versions might work – as long as the kernel patches apply cleanly.
Most 4.4.0 versions should work with only minor adjustments to the patches.
Note: Future versions of Lustre might add additional patch series and support other kernels, too.
 
So, if you do not just need LDISKFS, but also need perfect performance, read on and produce a patched set of DEB packages:
 
For simplicity's sake, the following assumes that you will build for Ubuntu 14.04. and kernel 4.4.0-45.66.
First, enter the kernel directory and check out the correct code:
[bash-4.3]$ cd ${BUILDPATH}/ubuntu-kernel
[bash-4.3]$ git tag | grep 4.4.0-45
Ubuntu-lts-4.4.0-45.66_14.04.1
[bash-4.3]$ git checkout Ubuntu-lts-4.4.0-45.66_14.04.1
 
Now, you need to apply the initial set of performance improvement patches. This is not strictly needed, but will ensure optimum LDISKFS performance.
Extract the patchset from the Lustre source code:
cd ${BUILDPATH}
PDIR="lustre-source/lustre/kernel_patches/"
for p in $(cat $PDIR/series/4.4-ubuntu14+16.series); do cat $PDIR/patches/$p; done > kernel.patch
cd ubuntu-kernel
patch -Np1 -i ../kernel.patch
This set of patches should apply cleanly.  If they do not (in case you used a different kernel version), you must fix the patches by manually merging the ".rej" files.
 
Next, you need to create a new version for this kernel, so the generated debian packages are different.
To do that, add the following to the top of the "debian.master/changelog" file.
linux (4.4.0-45.66lustre) xenial; urgency=medium

  * Added Lustre Patches

 -- YOUR NAME <YOUR@EMAIL.COM>  Mon, 08 May 2017 18:16:59 +0200

You need to insert your name and E-Mail address, alter the date and (in case you did that) set your chosen version.
Note, that you can automatically create this entry via "dch -i -c debian.master/changelog" – if you have that tool installed.
If you are using Ubuntu 14.04., you also need to add the same prefix to "debian.xenial/changelog".
 
Now, you must save this by creating a GIT commit
git commit -a -s
As the message, use:
UBUNTU: Ubuntu-lts-4.4.0-45.66~14.04.1-lustre
Added Lustre performance improvement patches on top of LTS release kernel
Signed-off-by: YOUR NAME <YOUR@EMAIL.COM>
 
Now, you can create the Debian packages to deploy the kernel on your target servers.
To do so, please execute:
chmod a+x debian/rules

chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*
fakeroot debian/rules clean
fakeroot debian/rules editconfigs
When executing this last command, you need to go through all configs, or you will get a complain later.
Simply hit enter when asked "[Y/n]" and save the kernel config unmodified (Exit → Hit Enter)
After the config files are created, you can build all kernel packages
fakeroot debian/rules binary
If you only need the main modules (linux-image-*, linux-header-*), you can save time by running:
fakeroot debian/rules binary-headers binary-generic binary-perarch
After that is done, the ${BUILDPATH} should contain the needed Debian packages. Save them for later deployment on the target servers.
 
Now, as a last step, you need to build the kernel again, in a way that Lustre can understand:
git clean -dfx
git reset --hard
cp /boot/config-4.4.0-45-generic .config
touch .scmversion
sed -i 's/VERSION = .*/VERSION = 4/' Makefile
sed -i 's/PATCHLEVEL = .*/PATCHLEVEL = 4/' Makefile
sed -i 's/SUBLEVEL = .*/SUBLEVEL = 0/' Makefile
sed -i 's/EXTRAVERSION =.*/EXTRAVERSION = -45-generic/' Makefile
make -j $(nproc)
Please note the comments from section (2.1) regarding the kernel config. If you do not have it under "/boot", you must extract it from the appropriate .deb package.
The same applies to the version replacement. Alter the number as needed for your target kernel version.
3.) (Optional) Building ZFS packages
If you wish to compile Lustre so that it can use ZFS as a backing storage, the current compile server does not need to have the ZFS modules installed.
All that you need to do is, it to have the ZFS source code compiled, so that Lustre can link against its modules.
While it should be possible to compile against a subtly different ZFS version than what you will use later on the target system, the safest way is to use the same version.
First, create a directory that will hold both SPL and ZFS.
cd ${BUILDPATH}
mkdir zfs
cd zfs
Then, download the correct ZFS source code for your target Ubuntu version:
Ubuntu 14.04:
wget https://launchpad.net/~zfs-native/+archive/ubuntu/stable/+files/spl-linux_0.6.5.10.orig.tar.bz2
wget https://launchpad.net/~zfs-native/+archive/ubuntu/stable/+files/spl-linux_0.6.5.10-1~trusty.debian.tar.gz
wget https://launchpad.net/~zfs-native/+archive/ubuntu/stable/+files/zfs-linux_0.6.5.10.orig.tar.bz2
wget https://launchpad.net/~zfs-native/+archive/ubuntu/stable/+files/zfs-linux_0.6.5.10-1~trusty.debian.tar.gz
Note: It is very likely that these links no longer exist. The ZFS team on Launchpad is in the habit of only keeping the most recently published version.
If you get a 404 error when downloading, go to the following page and find the updated links:
Ubuntu 16.04:
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.6.5.6.orig.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/main/z/zfs-linux/zfs-linux_0.6.5.6-0ubuntu17.debian.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.6.5.6.orig.tar.xz
wget http://archive.ubuntu.com/ubuntu/pool/universe/s/spl-linux/spl-linux_0.6.5.6-0ubuntu2.debian.tar.xz
The rest of the guide assumes you chose the Ubuntu 14.04. version. Adjust the file- and pathnames accordingly, if you are targeting 16.04.
 
Extract, patch and compile SPL (adjust the ZFSVERSION parameter accordingly):
ZFSVERSION=0.6.5.10
tar -xf spl-linux_${ZFSVERSION}.orig.tar.bz2
cd spl-linux-${ZFSVERSION}
tar -xf ../spl-linux_${ZFSVERSION}-1~trusty.debian.tar.gz
for f in debian/patches/*.patch; do patch -Np1 -i $f; done
./autogen.sh &&
./configure --with-linux=${BUILDPATH}/ubuntu-kernel &&
make -j $(nproc)
Then, do the same for ZFS:
cd ${BUILDPATH}/zfs
tar -xf zfs-linux_${ZFSVERSION}.orig.tar.bz2
cd zfs-linux-${ZFSVERSION}
tar -xf ../zfs-linux_${ZFSVERSION}-1~trusty.debian.tar.gz
for f in debian/patches/*.patch; do patch -Np1 -i $f; done
./autogen.sh &&
./configure --with-linux=${BUILDPATH}/ubuntu-kernel --with-spl=${BUILDPATH}/zfs/spl-linux-${ZFSVERSION} &&
make -j $(nproc)
 
4.) (Optional) Building modified e2fsprogs for LDISKFS support
If you wish to use LDISKFS, you need to compile and install a modified version of the e2fsprogs package.
This is not needed, if you only intend to use ZFS.
To compile the needed package, execute the following steps:
cd ${BUILDPATH}
git clone git://git.whamcloud.com/tools/e2fsprogs.git
cd e2fsprogs
git checkout v1.42.13.wc6 -b v1.42.13.wc6
If you can't use the GIT protocol (e.g. if you're behind a corporate proxy), download a snapshot from:
 
Then, generate the Ubuntu Debian packages:
wget -P ../ http://archive.ubuntu.com/ubuntu/pool/main/e/e2fsprogs/e2fsprogs_1.42.13-1ubuntu1.debian.tar.xz
tar --exclude "debian/changelog" -xf ../e2fsprogs_1.42.13-1ubuntu1.debian.tar.xz
sed -i 's/ext2_types-wrapper.h$//g' lib/ext2fs/Makefile.in
dpkg-buildpackage -b -us -uc
This will create several ".deb" files in ${BUILDPATH}.
Save them somewhere sane, so that you can install the modified e2fsprogs on the target servers later.
Note 1: The downloading of the Ubuntu Debian install files is necessary, as the ones form the GIT repo are not Ubuntu specific.
Please make sure to use the same version numbers for the git checkout as well as the Ubuntu download.
Note 2: The 'sed' call works around the build scripts not properly identifying a build output artifact.
This has no effect on running the e2fsprogs, but might cause issues if you use something that compiles against the e2fsprogs, as you will miss that particular header file.
5.) Building Lustre Client & Server Packages
5.1) Downloading Lustre Sources
The first step is to download the most recent Lustre sources from the HPDD Git repo:
cd ${BUILDPATH}
git clone git://git.whamcloud.com/fs/lustre-release.git
cd lustre-release
git checkout b2_10
This checks out the latest 2.10 code. If you need a newer version, adjust the checked-out branch accordingly.
All releases up to (and including) 2.10 do not come with Ubuntu Server support enabled. Later versions should have the needed changes already.
To make the versions <= 2.10 buildable for Ubuntu servers, you have to download and apply the following change:
git fetch https://review.whamcloud.com/fs/lustre-release refs/changes/23/27323/9
git cherry-pick FETCH_HEAD
Unless GIT reports a merge-conflict, you should now have a Ubuntu-Server enabled Lustre source code base.
Versions later than 2.10 most likely have this change integrated already and do not need having that change applied manually.
...
Building the Client packages has the least demands and only needs the kernel sources headers being available.
You don't even need to have a 4.415.0-series kernel present as for the server build. The clients can run under any 2.6.32 or later kernel, as long as the kernel headers installed match the running kernel version.
Unless the build system for those has already been made compatible with DKMS (Dynamic Kernel Module Support), it will be necessary to generate a suitable package for each kernel version your clients run under.
...
Of special note is, that if you want LDISKFS support, you must use a 4.415.0-series Linux kernel. Other kernel versions might need significant alterations to the EXT4-to-LDISKFS patches. If you wish to only use ZFS, you do not need to care about that. Similar to the Client packages, the source code will then be compatible with any Linux kernel version >= 2.6
...
If the compilation succeeded, you should find the server modules again under "lustre-release/debs".
Copy these to a sane location, so that you can use them for installing Lustre Servers.