Purpose

Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky linux 8.7 machine.

RHEL/Rocky Linux 8.7 aarch64 can be used inside a VM (tested on UTM 4.0.9) on an M1 Macbook Pro host. The "Use Hypervisor" option should be disabled on the QEMU page of preference in this case. A full walk-thru on installing a aarch64 VM on an Apple Silicon Mac is available here.

Prerequisite

  • A newly installed RHEL/Rocky linux 8 x86_64 or aarch64 machine connected to the internet.
  • NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
  • NOTE Verify that SElinux is disabled.
  • This manual uses the Rocky linux 8 distribution available at x86_64 or aarch64.
  • It is assumed that Rocky linux 8 was installed as is, and nothing else was installed on it.

Overview

This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system. 

If you prefer to skip the kernel build steps and get right to building Lustre, you can download and install pre-patched server kernel RPMs from a recent lustre-master, and can then jump straight to the Configure and build Lustre section below.  You should navigate to the el8.x server results (green checkmarked circle), then Build Artifacts→Artifacts→RPMs→x86_64, and get at least the kernel, kernel-devel, kernel-modules RPMs to install.


Process

Provision machine and installing dependencies.

Once RHEL/Rocky 8.7 is newly installed on an x86_64 or aarch64 machine login as user root.

  1. Install the kernel development tools: 

    yum -y groupinstall "Development Tools"

  2. Install additional dependencies:

    yum config-manager --set-enabled powertools
dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex
yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists opencsd-devel
dnf install -y epel-release
dnf install -y ccache pdsh
dnf --enablerepo=ha install resource-agents

  3. Install tools for kernel RPM build:

    dnf install -y bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync
# May only be needed on RHEL9 derivatives:
dnf install -y python3-devel

  4. Install e2fsprogs packages:

    git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.1-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8
sudo make install
cd ..

    Or download and install the following e2fsprogs packages from https://build.whamcloud.com/

Preparing the Lustre source.

  1. Create a user build with the home directory /home/build  and switch to this user:

    useradd -m build
su build
cd $HOME

  2. Prepare Lustre src

    git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh

Prepare a patched kernel for Lustre

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

  1. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:

    cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

  2. Install the kernel source:

    rpm -ivh https://dl.rockylinux.org/vault/rocky/8.7/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm

  3. Prepare the source using rpmbuild:

    cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

  4. Copy the kernel config file into lustre tree:

    cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.`uname -m`/configs/kernel-4.18.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

  5. Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config:

    Find the line with '# IO Schedulers' and insert following two lines below it:

    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_DEFAULT_IOSCHED="deadline"

    Or use cmd:

    sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

  6. Gather all the patches from lustre tree into a single file:

    cd ~/lustre-release/lustre/kernel_patches/series && \
for patch in $(<"4.18-rhel8.7.series"); do \
     patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
     cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; \
done

  7. Copy the kernel patch into RPM build tree:

    cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-4.18.0-lustre.patch

  8. Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:

    sed -i.inst -e '/^    find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
    cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
    rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec

  9. Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.

    echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
  10. (optional - only for aarch64 VMs on Apple Silicon Macs) As outlined in the Apple Silicon guide the kernel page size must be set to 4K  or 16K . ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config uses 64K  and needs to be modified.  Otherwise the Lustre kernel won't boot. Set the page size to 4K either via make menuconfig or manually. Diff before and after the change: 
    14,18c13,17
< CONFIG_ARM64_PAGE_SHIFT=16
< CONFIG_ARM64_CONT_SHIFT=5
< CONFIG_ARCH_MMAP_RND_BITS_MIN=14
< CONFIG_ARCH_MMAP_RND_BITS_MAX=29
< CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=7
---
> CONFIG_ARM64_PAGE_SHIFT=12
> CONFIG_ARM64_CONT_SHIFT=4
> CONFIG_ARCH_MMAP_RND_BITS_MIN=18
> CONFIG_ARCH_MMAP_RND_BITS_MAX=33
> CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=11
35c34
< CONFIG_PGTABLE_LEVELS=3
---
> CONFIG_PGTABLE_LEVELS=4
301c300
< CONFIG_ARCH_MMAP_RND_BITS=14
---
> CONFIG_ARCH_MMAP_RND_BITS=18
389a389
> #
622c622
< # CONFIG_ARM64_4K_PAGES is not set
---
> CONFIG_ARM64_4K_PAGES=y
624,625c624,625
< CONFIG_ARM64_64K_PAGES=y
< # CONFIG_ARM64_VA_BITS_42 is not set
---
> # CONFIG_ARM64_64K_PAGES is not set
> # CONFIG_ARM64_VA_BITS_39 is not set
627d626
< # CONFIG_ARM64_VA_BITS_52 is not set
629,631c628,629
< # CONFIG_ARM64_PA_BITS_48 is not set
< CONFIG_ARM64_PA_BITS_52=y
< CONFIG_ARM64_PA_BITS=52
---
> CONFIG_ARM64_PA_BITS_48=y
> CONFIG_ARM64_PA_BITS=48
658a657
> CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
727c726
< CONFIG_FORCE_MAX_ZONEORDER=14
---
> CONFIG_FORCE_MAX_ZONEORDER=11
793a793
> # CONFIG_COMPAT is not set
2625a2626
> # CONFIG_VMXNET3 is not set

  11. Start building the kernel with rpmbuild:

    cd ~/kernel/rpmbuild && buildid="_lustre" && \
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
           --without kabichk --define "buildid ${buildid}" \
           ~/kernel/rpmbuild/SPECS/kernel.spec

  12. Install the kernel packages:

    cd ~/kernel/rpmbuild/RPMS/`uname -m`/
sudo rpm -Uvh --replacepkgs --force kernel-*.rpm
sudo reboot

  13. Login system after reboot, run name -r and see:

    # uname -r
    4.18.0-425.19.2.el8_lustre.x86_64

    or

    4.18.0-425.19.2.el8_lustre.aarch64

Configure and build Lustre

  1. Configure Lustre source:

    cd lustre-release
./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8_lustre.`uname -m`/ --disable-gss --disable-crypto

  2. Build and install Lustre:

    make -j8
sudo make install
sudo depmod -a

  3. Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:

    /usr/lib64/lustre/tests/llmount.sh

    Stopping clients: sr050 /mnt/lustre (opts:-f)
    Stopping clients: sr050 /mnt/lustre2 (opts:-f)
    sr050: executing set_hostid
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    libcfs will create CPU partition based on online CPUs
    Formatting mgs, mds, osts
    Format mds1: /tmp/lustre-mdt1
    Format ost1: /tmp/lustre-ost1
    Format ost2: /tmp/lustre-ost2
    Checking servers environments
    Checking clients sr050 environments
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    libcfs will create CPU partition based on online CPUs
    Setup mgs, mdt, osts
    Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
    Commit the device label on /tmp/lustre-mdt1
    Started lustre-MDT0000
    Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
    Commit the device label on /tmp/lustre-ost1
    Started lustre-OST0000
    Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
    Commit the device label on /tmp/lustre-ost2
    Started lustre-OST0001
    Starting client: sr050:  -o user_xattr,flock sr050@tcp:/lustre /mnt/lustre
    UUID                   1K-blocks        Used   Available Use% Mounted on
    lustre-MDT0000_UUID       125056        1644      112176   2% /mnt/lustre[MDT:0]
    lustre-OST0000_UUID       313104        1244      284700   1% /mnt/lustre[OST:0]
    lustre-OST0001_UUID       313104        1244      284700   1% /mnt/lustre[OST:1]
     
    filesystem_summary:       626208        2488      569400   1% /mnt/lustre
     
    Using TIMEOUT=20
    osc.lustre-OST0000-osc-ffffce9052651800.idle_timeout=debug
    osc.lustre-OST0001-osc-ffffce9052651800.idle_timeout=debug
    setting jobstats to procname_uid
    Setting lustre.sys.jobid_var from disable to procname_uid
    Waiting 90s for 'procname_uid'
    Updated after 4s: want 'procname_uid' got 'procname_uid'
    disable quota as required
    lod.lustre-MDT0000-mdtlov.mdt_hash=crush

  4. You will now have a Lustre filesystem available at /mnt/lustre

End

  • No labels