Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky linux 8.7 machine.
RHEL/Rocky Linux 8.7 aarch64 can be used inside a VM (tested on UTM 4.0.9) on an M1 Macbook Pro host. The "Use Hypervisor" option should be disabled on the QEMU page of preference in this case. A full walk-thru on installing a aarch64 VM on an Apple Silicon Mac is available here.
This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.
Once RHEL/Rocky 8.7 is newly installed on an x86_64 or aarch64 machine login as user root.
Install the kernel development tools:
yum -y groupinstall "Development Tools" |
Install additional dependencies:
yum config-manager --set-enabled powertools dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists opencsd-devel dnf install -y epel-release dnf install -y ccache pdsh dnf --enablerepo=ha install resource-agents |
Install tools for kernel RPM build:
dnf install -y bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync # May only be needed on RHEL9 derivatives: dnf install -y python3-devel |
Install e2fsprogs packages:
git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.0-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8 sudo make install cd .. |
Or download and install the following e2fsprogs packages from https://build.whamcloud.com/
Create a user build with the home directory /home/build and switch to this user:
useradd -m build su build cd $HOME |
Prepare Lustre src
git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh |
In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.
First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:
cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros |
Install the kernel source:
rpm -ivh https://dl.rockylinux.org/vault/rocky/8.7/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm |
Prepare the source using rpmbuild:
cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec |
Copy the kernel config file into lustre tree:
cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.`uname -m`/configs/kernel-4.18.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config |
Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config:
Find the line with '# IO Schedulers' and insert following two lines below it:CONFIG_IOSCHED_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"
Or use cmd:
sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config |
Gather all the patches from lustre tree into a single file:
cd ~/lustre-release/lustre/kernel_patches/series && \
for patch in $(<"4.18-rhel8.7.series"); do \
patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; \
done |
Copy the kernel patch into RPM build tree:
cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-4.18.0-lustre.patch |
Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:
sed -i.inst -e '/^ find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec |
Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.
echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config |
4K or 16K . ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config uses 64K and needs to be modified. Otherwise the Lustre kernel won't boot. Set the page size to 4K either via make menuconfig or manually. Diff before and after the change: 14,18c13,17 < CONFIG_ARM64_PAGE_SHIFT=16 < CONFIG_ARM64_CONT_SHIFT=5 < CONFIG_ARCH_MMAP_RND_BITS_MIN=14 < CONFIG_ARCH_MMAP_RND_BITS_MAX=29 < CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=7 --- > CONFIG_ARM64_PAGE_SHIFT=12 > CONFIG_ARM64_CONT_SHIFT=4 > CONFIG_ARCH_MMAP_RND_BITS_MIN=18 > CONFIG_ARCH_MMAP_RND_BITS_MAX=33 > CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=11 35c34 < CONFIG_PGTABLE_LEVELS=3 --- > CONFIG_PGTABLE_LEVELS=4 301c300 < CONFIG_ARCH_MMAP_RND_BITS=14 --- > CONFIG_ARCH_MMAP_RND_BITS=18 389a389 > # 622c622 < # CONFIG_ARM64_4K_PAGES is not set --- > CONFIG_ARM64_4K_PAGES=y 624,625c624,625 < CONFIG_ARM64_64K_PAGES=y < # CONFIG_ARM64_VA_BITS_42 is not set --- > # CONFIG_ARM64_64K_PAGES is not set > # CONFIG_ARM64_VA_BITS_39 is not set 627d626 < # CONFIG_ARM64_VA_BITS_52 is not set 629,631c628,629 < # CONFIG_ARM64_PA_BITS_48 is not set < CONFIG_ARM64_PA_BITS_52=y < CONFIG_ARM64_PA_BITS=52 --- > CONFIG_ARM64_PA_BITS_48=y > CONFIG_ARM64_PA_BITS=48 658a657 > CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y 727c726 < CONFIG_FORCE_MAX_ZONEORDER=14 --- > CONFIG_FORCE_MAX_ZONEORDER=11 793a793 > # CONFIG_COMPAT is not set 2625a2626 > # CONFIG_VMXNET3 is not set |
Start building the kernel with rpmbuild:
cd ~/kernel/rpmbuild && buildid="_lustre" && \
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
--without kabichk --define "buildid ${buildid}" \
~/kernel/rpmbuild/SPECS/kernel.spec |
Install the kernel packages:
cd ~/kernel/rpmbuild/RPMS/`uname -m`/ sudo rpm -Uvh --replacepkgs --force kernel-*.rpm sudo reboot |
Login system after reboot, run name -r and see:
or 4.18.0-425.19.2.el8_lustre.aarch64 |
Configure Lustre source:
cd lustre-release ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8_lustre.`uname -m`/ --disable-gss --disable-shared --disable-crypto |
Build and install Lustre:
make -j8 sudo make install sudo depmod -a |
Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:
/usr/lib64/lustre/tests/llmount.sh |
|
/mnt/lustreEnd