Page History

...

Describe the steps you need to build and test a Lustre system (MGS, MDT, MDS, OSS, OST, client) from the master branch on an x86_64 or aarch64, RHEL/Rocky linux 8.7 machine.

...

...

RHEL/Rocky Linux 8.7 aarch64 can be used inside a VM (tested on UTM 4.0.9) on an M1 Macbook Pro host. The "Use Hypervisor" option should be disabled on the QEMU page of preference in this case. A full walk-thru on installing a aarch64 VM on an Apple Silicon Mac is available here.

Prerequisite

• A newly installed RHEL/Rocky linux 8 x86_64 or aarch64 machine connected to the internet.
• NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
• NOTE Verify that SElinux is disabled.
• This manual uses the Rocky linux 8 distribution available at x86_64 or aarch64
linux 8 x86_64 machine connected to the internet.
• NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
• NOTE Verify that SElinux is disabled.
• This manual uses the Rocky linux 8 distribution available at the link.
• It is assumed that Rocky linux 8 was installed as is, and nothing else was installed on it.

...

This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.

Process

Provision machine and installing dependencies.

Once RHEL/Rocky 8.7 is newly installed on an x86_64 machine login as user root.

...

Install the kernel development tools: 

yum -y groupinstall "Development Tools"

...

Install additional dependencies:

yum config-manager --set-enabled powertools
dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex
yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists
dnf install -y epel-release
dnf install -y ccache

If you prefer to skip the kernel build steps and get right to building Lustre, you can download and install pre-patched server kernel RPMs from a recent lustre-master, and can then jump straight to the Configure and build Lustre section below.  You should navigate to the el8.x server results (green checkmarked circle), then Build Artifacts→Artifacts→RPMs→x86_64, and get at least the kernel, kernel-devel, kernel-modules RPMs to install.

...

Process

Provision machine and installing dependencies.

Once RHEL/Rocky 8.7 is newly installed on an x86_64 or aarch64 machine login as user root.

1. Install the kernel development tools: 

   Code Block
   language bash theme Eclipse
   yum -y groupinstall "Development Tools"

2. Install additional dependencies:
   

   Code Block
   language bash theme Eclipse
   yum config-manager --set-enabled powertools dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists opencsd-devel dnf install -y epel-release dnf install -y ccache pdsh dnf --enablerepo=ha install resource-agents

3. Install tools for kernel RPM build:

   Code Block
   language bash theme Eclipse
   dnf install -y bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync # May only be needed on RHEL9 derivatives: dnf install -y python3-devel

4. Install e2fsprogs packages:
   

   Code Block
   language bash theme Eclipse
   git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.1-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8 sudo make install cd ..

   Or download and install the following e2fsprogs packages from https://build.whamcloud.com/

Preparing the Lustre source.

1. Create a user build with the home directory /home/build  and switch to this user:

   Code Block
   language bash theme Eclipse
   useradd -m build su build cd $HOME

2. Prepare Lustre src

   Code Block
   language bash theme Eclipse
   git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh

Prepare a patched kernel for Lustre

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

1. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:

   Code Block
   language bash theme Eclipse
   cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

2. Install the kernel source:

   Code Block
   language bash theme Eclipse
   rpm -ivh https://dl.rockylinux.org/vault/rocky/8.7/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm

3. Prepare the source using rpmbuild:

   Code Block
   language bash theme Eclipse
   cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

4. Copy the kernel config file into lustre tree

...

1. :

   Code Block
   language bash theme Eclipse

...

dnf install bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync

1. cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.`uname -m`/configs/kernel-4.18.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

2. Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config:

   Find the line with '# IO Schedulers' and insert following two lines below it:

   CONFIG_IOSCHED_DEADLINE=y
   CONFIG_DEFAULT_IOSCHED="deadline"
   
   Or use cmd:

   Code Block
   language bash theme Eclipse
   sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

3. Gather all the patches from lustre tree into a single file:
   

   Code Block
   enableddl true language bash theme Eclipse lang Bash globaltitle 7. Copy the kernel patch into RPM build tree:
   cd ~/lustre-release/lustre/kernel_patches/series && \ for patch in $(<"4.18-rhel8.7.series"); do \      patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \      cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; \ done

4. Copy the kernel patch into RPM build tree

...

Install e2fsprogs packages:

git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.0-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8
sudo make install
cd ..

Or download and install the following e2fsprogs packages from https://build.whamcloud.com/

Preparing the Lustre source.

1. Create a user build with the home directory /home/build  and switch to this user:

   Code Block
   language bash theme Eclipse
   useradd -m build su build cd $HOME

2. Prepare Lustre src

   Code Block
   language bash theme Eclipse
   git clone "https://review.whamcloud.com/fs/lustre-release" && cd lustre-release && sh ./autogen.sh

Prepare a patched kernel for Lustre

In this walk-thru, the kernel is built using rpmbuild - a tool specific to RPM based distributions.

1. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:
   

   Code Block
   language bash theme Eclipse
   cd $HOME && mkdir -p cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS} cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

2. Install the kernel source:

   Code Block
   language bash theme Eclipse
   rpm -ivh https://download.rockylinux.org/pub/rocky/8/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm

3. Prepare the source using rpmbuild:

   Code Block
   language bash theme Eclipse
   cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

4. SOURCES/patch-4.18.0-lustre.patch

5. Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:
   

   Code Block
   language bash theme Eclipse
   sed -i.inst -e '/^ find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\ cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\ rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \ -e '/^# empty final patch to facilitate testing of kernel patches/i\ Patch99995: patch-%{version}-lustre.patch' \ -e '/^ApplyOptionalPatch linux-kernel-test.patch/i\ ApplyOptionalPatch patch-%{version}-lustre.patch' \ ~/kernel/rpmbuild/SPECS/kernel.spec

6. Overwrite the kernel config file with

   Copy the kernel config file into lustre tree:

   Code Block

   language	bash
   theme	Eclipse

   cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.x86_64/configs/kernel-4.18.0-x86_64.config

   ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change "x86_64

   .config

   Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-x86_64.config::

   Find the line with '# IO Schedulers' and insert following two lines below it:CONFIG_IOSCHED_DEADLINE=y
   CONFIG_DEFAULT_IOSCHED="deadline"
   Or use cmd:

   Code Block
   language bash theme Eclipse
   sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-x86_64.config

   Gather all the patches from lustre tree into a single file:
   

   " with "arm64" in the first command for aarch64 architecture.

   Code Block
   language bash theme Eclipse
   echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config

7. (optional - only for aarch64 VMs on Apple Silicon Macs) As outlined in the Apple Silicon guide the kernel page size must be set to 4K  or 16K . ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config uses 64K  and needs to be modified.  Otherwise the Lustre kernel won't boot. Set the page size to 4K either via make menuconfig or manually. Diff before and after the change: true

   Code Block
   Code Block
   enableddl
   language bash theme Eclipse langcollapse Bash globaltitle 7. Copy the kernel patch into RPM build tree:
   cd ~/lustre-release/lustre/kernel_patches/series && \ for patch in $(<"4.18-rhel8.7.series"); do \      patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \      cat "${patch_file}" >> "$HOME/lustre-kernel-x86_64-lustre.patch"; \ done

8. Copy the kernel patch into RPM build tree:
   

   Code Block
   language bash theme Eclipse
   cp ~/lustre-kernel-x86_64-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-4.18.0-lustre.patch

9. Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:
   

   Code Block
   language bash theme Eclipse
   sed -i.inst -e '/^ find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\ cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\ rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \ -e '/^# empty final patch to facilitate testing of kernel patches/i\ Patch99995: patch-%{version}-lustre.patch' \ -e '/^ApplyOptionalPatch linux-kernel-test.patch/i\ ApplyOptionalPatch patch-%{version}-lustre.patch' \ ~/kernel/rpmbuild/SPECS/kernel.spec

10. Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-x86_64.config:

    Code Block
    language bash theme Eclipse
    echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-x86_64.config cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-x86_64.config >> ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-x86_64.config

11. true

    14,18c13,17 < CONFIG_ARM64_PAGE_SHIFT=16 < CONFIG_ARM64_CONT_SHIFT=5 < CONFIG_ARCH_MMAP_RND_BITS_MIN=14 < CONFIG_ARCH_MMAP_RND_BITS_MAX=29 < CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=7 --- > CONFIG_ARM64_PAGE_SHIFT=12 > CONFIG_ARM64_CONT_SHIFT=4 > CONFIG_ARCH_MMAP_RND_BITS_MIN=18 > CONFIG_ARCH_MMAP_RND_BITS_MAX=33 > CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=11 35c34 < CONFIG_PGTABLE_LEVELS=3 --- > CONFIG_PGTABLE_LEVELS=4 301c300 < CONFIG_ARCH_MMAP_RND_BITS=14 --- > CONFIG_ARCH_MMAP_RND_BITS=18 389a389 > # 622c622 < # CONFIG_ARM64_4K_PAGES is not set --- > CONFIG_ARM64_4K_PAGES=y 624,625c624,625 < CONFIG_ARM64_64K_PAGES=y < # CONFIG_ARM64_VA_BITS_42 is not set --- > # CONFIG_ARM64_64K_PAGES is not set > # CONFIG_ARM64_VA_BITS_39 is not set 627d626 < # CONFIG_ARM64_VA_BITS_52 is not set 629,631c628,629 < # CONFIG_ARM64_PA_BITS_48 is not set < CONFIG_ARM64_PA_BITS_52=y < CONFIG_ARM64_PA_BITS=52 --- > CONFIG_ARM64_PA_BITS_48=y > CONFIG_ARM64_PA_BITS=48 658a657 > CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y 727c726 < CONFIG_FORCE_MAX_ZONEORDER=14 --- > CONFIG_FORCE_MAX_ZONEORDER=11 793a793 > # CONFIG_COMPAT is not set 2625a2626 > # CONFIG_VMXNET3 is not set

12. Start building the kernel with Start building the kernel with rpmbuild:
    

    Code Block
    language bash theme Eclipse
    cd ~/kernel/rpmbuild && buildid="_lustre" && \ rpmbuild -ba --with firmware --target x86_64 `uname -m` --with baseonly \ --without kabichk --define "buildid ${buildid}" \ ~/kernel/rpmbuild/SPECS/kernel.spec

13. Install the kernel packages:
    

    Code Block
    language bash theme Eclipse
    cd ~/kernel/rpmbuild/RPMS/x86_64`uname -m`/ sudo rpm -Uvh --replacepkgs --force kernel-*.rpm sudo reboot

14. Login system after reboot, run name -r and see:
    

    # uname -r 4.18.0-425.19.2.el8_lustre.x86_64 or 4.18.0-425.19.2.el8_lustre.aarch64

Configure and build Lustre

1. Configure Lustre source:
   

   Code Block
   language bash theme Eclipse
   cd lustre-release ./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8_lustre.x86_64`uname -m`/ --disable-gss --disable-shared --disable-crypto

2. Build and install Lustre:

   Code Block
   language bash theme Eclipse
   make -j8 sudo make install sudo depmod -a

3. Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:

   Code Block
   language bash theme Eclipse
   /usr/lib64/lustre/tests/llmount.sh

   Stopping clients: sr050 /mnt/lustre (opts:-f) Stopping clients: sr050 /mnt/lustre2 (opts:-f) sr050: executing set_hostid Loading modules from /usr/lib64/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Checking servers environments Checking clients sr050 environments Loading modules from /usr/lib64/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs Setup mgs, mdt, osts Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1 Commit the device label on /tmp/lustre-mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1 Commit the device label on /tmp/lustre-ost1 Started lustre-OST0000 Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2 Commit the device label on /tmp/lustre-ost2 Started lustre-OST0001 Starting client: sr050:  -o user_xattr,flock sr050@tcp:/lustre /mnt/lustre UUID                   1K-blocks        Used   Available Use% Mounted on lustre-MDT0000_UUID       125056        1644      112176   2% /mnt/lustre[MDT:0] lustre-OST0000_UUID       313104        1244      284700   1% /mnt/lustre[OST:0] lustre-OST0001_UUID       313104        1244      284700   1% /mnt/lustre[OST:1]   filesystem_summary:       626208        2488      569400   1% /mnt/lustre   Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffce9052651800.idle_timeout=debug osc.lustre-OST0001-osc-ffffce9052651800.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 4s: want 'procname_uid' got 'procname_uid' disable quota as required lod.lustre-MDT0000-mdtlov.mdt_hash=crush

4. You will now have a Lustre filesystem available at /mnt/lustre

...