Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky linux 8.7 machine.
RHEL/Rocky Linux 8.7 aarch64 can be used inside a VM (tested on UTM 4.0.9) on an M1 Macbook Pro host. The "Use Hypervisor" option should be disabled on the QEMU page of preference in this case.
This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.
Once RHEL/Rocky 8.7 is newly installed on an x86_64 or aarch64 machine login as user root.
Install the kernel development tools:
yum -y groupinstall "Development Tools" |
Install additional dependencies:
yum config-manager --set-enabled powertools dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists dnf install -y epel-release dnf install -y ccache |
Install tools for kernel RPM build:
dnf install -y bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync # May only be needed on RHEL9 derivatives: dnf install -y python3-devel |
Install e2fsprogs packages:
git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.0-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8 sudo make install cd .. |
Or download and install the following e2fsprogs packages from https://build.whamcloud.com/
Create a user build with the home directory /home/build and switch to this user:
useradd -m build su build cd $HOME |
Prepare Lustre src
git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh |
In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.
First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:
cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros |
Install the kernel source:
rpm -ivh https://download.rockylinux.org/pub/rocky/8/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm |
Prepare the source using rpmbuild:
cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec |
Copy the kernel config file into lustre tree:
cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.`uname -m`/configs/kernel-4.18.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config |
Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config::
Find the line with '# IO Schedulers' and insert following two lines below it:CONFIG_IOSCHED_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
Or use cmd:
sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config |
Gather all the patches from lustre tree into a single file:
cd ~/lustre-release/lustre/kernel_patches/series && \
for patch in $(<"4.18-rhel8.7.series"); do \
patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; \
done |
Copy the kernel patch into RPM build tree:
cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-4.18.0-lustre.patch |
Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:
sed -i.inst -e '/^ find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec |
Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change x86_64 with arm64 in the first command for aarch64 architecture.
echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-x86_64.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-`uname -m`.config |
Start building the kernel with rpmbuild:
cd ~/kernel/rpmbuild && buildid="_lustre" && \
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
--without kabichk --define "buildid ${buildid}" \
~/kernel/rpmbuild/SPECS/kernel.spec |
Install the kernel packages:
cd ~/kernel/rpmbuild/RPMS/`uname -m`/ sudo rpm -Uvh --replacepkgs --force kernel-*.rpm sudo reboot |
Login system after reboot, run name -r and see:
or 4.18.0-425.19.2.el8_lustre.aarch64 |
Configure Lustre source:
cd lustre-release ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8_lustre.`uname -m`/ --disable-gss --disable-shared --disable-crypto |
Build and install Lustre:
make -j8 sudo make install sudo depmod -a |
Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:
/usr/lib64/lustre/tests/llmount.sh |
|
/mnt/lustreEnd