Walk-thru- Build Lustre MASTER on RHEL/Rocky Linux 9.4 from Git

Purpose

Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky Linux 9.4 machine.

RHEL/Rocky Linux 9.4 aarch64 can be used inside a VM on an M1 Macbook Pro host with several hypervisors such as Virtualbox/Vagrant/UTM/Lima.

Prerequisite

A newly installed RHEL/Rocky linux 9.4 x86_64 or aarch64 machine connected to the internet.
NOTE It is suggested that you have at least 2GB of memory on the machine you are using for the build.
This manual uses the Rocky linux 9.4 distribution available at x86_64 or aarch64.
It is assumed that Rocky linux 9.4 was installed as is, and nothing else was installed on it.

There is also the option of a Lima VM with a provided configuration file which creates and installs all the required packages.
lustre_vm.yaml

Overview

This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.

Process

Provision machine and installing dependencies.

Once RHEL/Rocky 9 is newly installed on an x86_64 or aarch64 machine login as user root.

Install all the required packages:

dnf -y groupinstall "Development Tools"
dnf -y config-manager --set-enabled crb
dnf -y install vim git libuuid-devel libblkid-devel wget rpmdevtools dnf-plugins-core redhat-rpm-config kernel-srpm-macros python3-devel epel-release keyutils-libs-devel libnl3-devel libyaml-devel libmount-devel
dnf -y install bc dwarves gcc-plugin-devel glibc-static kernel-rpm-macros net-tools perl-devel rsync pesign rocky-sb-certs htop
dnf -y install krb5-devel

Add a build user and make it sudoer:

useradd -m builder
echo "builder ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/builder

Install e2fsprogs packages:

git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.1-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs
make -j8
sudo make install
cd ..

Or download and install the following e2fsprogs packages from https://build.whamcloud.com/

If you want to use the provided configuration file for Lima VM:

Create and start the VM:

limactl start lustre_vm.yaml....
limactl shell lustre_vm

and just go to the Install e2fsprogs package step.

Preparing the Lustre source.

Login as builder user:
```
sudo su builder
```

Prepare Lustre source:

git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh

Create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:

cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

Install the kernel source:

rpm -ivh https://dl.rockylinux.org/vault/rocky/9.4/BaseOS/source/tree/Packages/k/kernel-5.14.0-427.31.1.el9_4.src.rpm

Prepare the source using rpmbuild:

cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

Copy the kernel config file into lustre tree:

cp ~/kernel/rpmbuild/BUILD/kernel-5.14.0-427.31.1.el9_4/linux-5.14.0-427.31.1.el9.`uname -m`/configs/kernel-5.14.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.4-`uname -m`.config

Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.4-`uname -m`.config:
Find the line with '# IO Schedulers' and insert following two lines below it:
CONFIG_IOSCHED_DEADLINE=y
CONFIG_DEFAULT_IOSCHED="deadline"

Or use cmd:
```
sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.4-`uname -m`.config
```

Gather all the patches from lustre tree into a single file:

cd ~/lustre-release/lustre/kernel_patches/series
for patch in $(<"5.14-rhel9.4.series"); do \
    patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
    cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; 
done

Copy the kernel patch into RPM build tree:

cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-5.14.0-lustre.patch

Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:

sed -i.inst -e '/^    find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
    cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
    rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec

Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.4-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.

echo '# arm64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
or 
echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.4-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config

Start building the kernel with rpmbuild:

cd ~/kernel/rpmbuild && buildid="_lustre"
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
             --without kabichk --define "buildid ${buildid}" \
             ~/kernel/rpmbuild/SPECS/kernel.spec

Install the kernel packages:

cd ~/kernel/rpmbuild/RPMS/`uname -m`/
sudo rpm -Uvh --replacepkgs --force kernel-*.rpm
sudo reboot

Login system after reboot, run name -r and see:
# uname -r
5.14.0-427.31.1_lustre.el9.x86_64
or
5.14.0-427.31.1_lustre.el9.aarch64
Configure and Build Lustre

Configure Lustre source:

cd ~/lustre-release/
./configure --with-linux=/home/builder/kernel/rpmbuild/BUILD/kernel-5.14.0-427.31.1.el9_4/linux-5.14.0-427.31.1_lustre.el9.`uname -m`/

Build and install Lustre:

make -j8
sudo make install
sudo depmod -a

Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:

/usr/lib64/lustre/tests/llmount.sh

mgs: Rocky Linux release 9.4 (Blue Onyx) MGS_OS_VERSION_ID=9.4 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=151257088 MGS_OS_ID_LIKE=rhel centos fedora rocky mds1: Rocky Linux release 9.4 (Blue Onyx) MDS1_OS_ID=rocky MDS1_OS_VERSION_CODE=151257088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_VERSION_ID=9.4 ost1: Rocky Linux release 9.4 (Blue Onyx) OST1_OS_VERSION_ID=9.4 OST1_OS_VERSION_CODE=151257088 OST1_OS_ID=rocky OST1_OS_ID_LIKE=rhel centos fedora rocky client: Rocky Linux release 9.4 (Blue Onyx) CLIENT_OS_VERSION_ID=9.4 CLIENT_OS_ID_LIKE=rhel centos fedora rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=151257088 Stopping clients: lima-lustre-vm /mnt/lustre (opts:-f) Stopping clients: lima-lustre-vm /mnt/lustre2 (opts:-f) lima-lustre-vm: executing set_hostid Loading modules from /usr/lib64/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients lima-lustre-vm environments Loading modules from /usr/lib64/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Commit the device label on /tmp/lustre-mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 Commit the device label on /tmp/lustre-ost1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 Commit the device label on /tmp/lustre-ost2 Started lustre-OST0001 Starting client: lima-lustre-vm: -o user_xattr,flock 192.168.5.15@tcp:/lustre /mnt/lustre Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff000141b4c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff000141b4c800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 9s: want 'procname_uid' got 'procname_uid' disable quota as required

You will now have a Lustre filesystem available at /mnt/lustre

Page tree