Purpose

Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky Linux 9.6 machine.

RHEL/Rocky Linux 9.6 aarch64 can be used inside a VM on an M1 Macbook Pro host with several hypervisors such as Virtualbox/Vagrant/UTM/Lima.

Prerequisite

  • A newly installed RHEL/Rocky linux 9.6 x86_64 or aarch64 machine connected to the internet.
  • NOTE It is suggested that you have at least 2GB of memory on the machine you are using for the build.
  • This manual uses the Rocky linux 9.6 distribution available at x86_64 or aarch64.
  • It is assumed that Rocky linux 9.6 was installed as is, and nothing else was installed on it.

There is also the option of a Lima VM or Vagrant with a provided configuration files respectively which create and install all the required packages.

Overview

This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system. 

lustre_vm.yamlVagrantfile

Process

Provision machine and installing dependencies.

Once RHEL/Rocky 9.6 is newly installed on an x86_64 or aarch64 machine login as user root.

  1. Install all the required packages:

    dnf config-manager --set-enabled crb
dnf -y install vim git libuuid-devel libblkid-devel wget rpmdevtools dnf-plugins-core redhat-rpm-config kernel-srpm-macros python3-devel epel-release keyutils libs-devel libnl3-devel libyaml-devel libmount-devel
dnf -y install bc dwarves gcc-plugin-devel glibc-static kernel-rpm-macros net-tools perl-devel rsync pesign rocky-sb-certs htop
dnf -y install krb5-devel
dnf config-manager --set-enabled rt
nf -y install realtime-setup
  2. Add a build user and make it sudoer:
    useradd -m builder
echo "builder ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/builder
  3. Install e2fsprogs packages:
    git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.1-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs
make -j8
sudo make install
cd ..

    Or download and install the following e2fsprogs packages from https://build.whamcloud.com/


If you want to use the provided configuration file for Lima VM or Vagrant:

Create and start the VM:

Lima VM:

limactl start lustre_vm.yaml....
limactl shell lustre_vm

Vagrant:

vagrant up
vagrant ssh

and just go to the Install e2fsprogs package step.

Preparing the Lustre source.


  1. Login as builder user:
    sudo su builder

  2. Prepare Lustre source:

    git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh

  3. Create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:

    cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

  4. Install the kernel source:

    rpm -ivh https://dl.rockylinux.org/vault/rocky/9.6/BaseOS/source/tree/Packages/k/kernel-5.14.0-570.58.1.el9_6.src.rpm

  5. Prepare the source using rpmbuild:

    cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec
  6. Copy the kernel config file into lustre tree:
    cp ~/kernel/rpmbuild/BUILD/kernel-5.14.0-570.58.1.el9_6/linux-5.14.0-570.58.1.el9.`uname -m`/configs/kernel-5.14.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.6-`uname -m`.config

  7. Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.6-`uname -m`.config:

    Find the line with '# IO Schedulers' and insert following two lines below it:

    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_DEFAULT_IOSCHED="deadline"

    Or use cmd:

    sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.6-`uname -m`.config

  8. Gather all the patches from lustre tree into a single file:

    cd ~/lustre-release/lustre/kernel_patches/series
for patch in $(<"5.14-rhel9.6.series"); do \
    patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
    cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; 
done
  9. Copy the kernel patch into RPM build tree:
    cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-5.14.0-lustre.patch
  10. Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:

    sed -i.inst -e '/^    find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
    cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
    rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec
  11. Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.6-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.

    echo '# arm64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
or 
echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-5.14.0-5.14-rhel9.6-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config

  12. Start building the kernel with rpmbuild:

    cd ~/kernel/rpmbuild && buildid="_lustre"
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
             --without kabichk --define "buildid ${buildid}" \
             ~/kernel/rpmbuild/SPECS/kernel.spec

  13. Install the kernel packages:

    cd ~/kernel/rpmbuild/RPMS/`uname -m`/
sudo rpm -Uvh --replacepkgs --force kernel-*.rpm
sudo reboot

  14. Login system after reboot, run name -r and see:

    # uname -r
    5.14.0-570.58.1_lustre.el9.x86_64

    or

    5.14.0-570.58.1_lustre.el9.aarch64

    Configure and Build Lustre

  15. Configure Lustre source:

    cd ~/lustre-release/
./configure --with-linux=/home/builder/kernel/rpmbuild/BUILD/kernel-5.14.0-570.58.1.el9_6/linux-5.14.0-570.58.1_lustre.el9.`uname -m`/

  16. Build and install Lustre:

    make -j8
sudo make install
sudo depmod -a

  17. Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:

    /usr/lib64/lustre/tests/llmount.sh

    mgs: Rocky Linux release 9.6 (Blue Onyx)
    MGS_OS_VERSION_ID=9.6
    MGS_OS_ID=rocky
    MGS_OS_VERSION_CODE=151388160
    MGS_OS_ID_LIKE=rhel centos fedora rocky
    mds1: Rocky Linux release 9.6 (Blue Onyx)
    MDS1_OS_ID=rocky
    MDS1_OS_VERSION_CODE=151388160
    MDS1_OS_ID_LIKE=rhel centos fedora rocky
    MDS1_OS_VERSION_ID=9.6
    ost1: Rocky Linux release 9.6 (Blue Onyx)
    OST1_OS_VERSION_ID=9.6
    OST1_OS_VERSION_CODE=151388160
    OST1_OS_ID=rocky
    OST1_OS_ID_LIKE=rhel centos fedora rocky
    client: Rocky Linux release 9.6 (Blue Onyx)
    CLIENT_OS_VERSION_ID=9.6
    CLIENT_OS_ID_LIKE=rhel centos fedora rocky
    CLIENT_OS_ID=rocky
    CLIENT_OS_VERSION_CODE=151388160
    Stopping clients: lustre-build /mnt/lustre (opts:-f)
    Stopping client lustre-build /mnt/lustre opts:-f
    Stopping clients: lustre-build /mnt/lustre2 (opts:-f)
    Stopping /mnt/lustre-mds1 (opts:-f) on lustre-build
    Stopping /mnt/lustre-ost1 (opts:-f) on lustre-build
    Stopping /mnt/lustre-ost2 (opts:-f) on lustre-build
    lustre-build: executing set_hostid
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    MODOPTS_LIBCFS=
    libcfs will create CPU partition based on online CPUs
    Formatting mgs, mds, osts
    Format mds1: /tmp/lustre-mdt1
    Format ost1: /tmp/lustre-ost1
    Format ost2: /tmp/lustre-ost2
    Checking servers environments
    Checking clients lustre-build environments
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    MODOPTS_LIBCFS=
    libcfs will create CPU partition based on online CPUs
    Setup mgs, mdt, osts
    Start mds1: mount -t lustre -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
    Commit the device label on /tmp/lustre-mdt1
    Started lustre-MDT0000
    Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
    seq.cli-lustre-OST0000-super.width=65536
    Commit the device label on /tmp/lustre-ost1
    Started lustre-OST0000
    /mnt/lustre-ost1: 0 B (0 bytes) trimmed
    Start ost2: mount -t lustre -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
    seq.cli-lustre-OST0001-super.width=65536
    Commit the device label on /tmp/lustre-ost2
    Started lustre-OST0001
    /mnt/lustre-ost2: 0 B (0 bytes) trimmed
    Starting client: lustre-build:  -o user_xattr,flock 10.0.2.15@tcp,192.168.0.1@tcp:/lustre /mnt/lustre
    Using TIMEOUT=20

  18. You will now have a Lustre filesystem available at /mnt/lustre
  • No labels