Versions Compared

Old Version 7

changes.mady.by.user Vitaliy Kuznetsov

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Dilger

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: add shortcut to bypass kernel build

...

RHEL/Rocky Linux 8.7 aarch64 can be used inside a VM (tested on UTM 4.0.9) on an M1 Macbook Pro host.  The The "Use Hypervisor" option should be disabled on the QEMU page of preference in this case. A full walk-thru on installing a aarch64 VM on an Apple Silicon Mac is available here.

Prerequisite

  • A newly installed RHEL/Rocky linux 8 x86_64 or aarch64 machine connected to the internet.
  • NOTE It is suggested that you have at least 1GB of memory on the machine you are using for the build.
  • NOTE Verify that SElinux is disabled.
  • This manual uses the Rocky linux 8 distribution available at x86_64 or aarch64.
  • It is assumed that Rocky linux 8 was installed as is, and nothing else was installed on it.

...

This document walks through the steps of patching the kernel, building Lustre and running a basic test of the complete system.

Process

Provision machine and installing dependencies.

 

If you prefer to skip the kernel build steps and get right to building Lustre, you can download and install pre-patched server kernel RPMs from a recent lustre-master, and can then jump straight to the Configure and build Lustre section below.  You should navigate to the el8.x server results (green checkmarked circle), then Build Artifacts→Artifacts→RPMs→x86_64, and get at least the kernel, kernel-devel, kernel-modules RPMs to install.


...

Process

Provision machine and installing dependencies.

Once RHEL/Rocky 8.7 is newly installed on an x86_64 Once RHEL/Rocky 8.7 is newly installed on an x86_64 or aarch64 machine login as user root.

  1. Install the kernel development tools: 

    Code Block
    languagebash
    themeEclipse
    yum -y groupinstall "Development Tools"

  2. Install additional dependencies:

    Code Block
    languagebash
    themeEclipse
    yum config-manager --set-enabled powertools
dnf install -y gcc autoconf libtool which make patch diffutils file binutils-devel python38 python3-devel elfutils-devel libselinux-devel libaio-devel dnf-plugins-core bc bison flex git libyaml-devel libnl3-devel libmount-devel json-c-devel redhat-lsb libssh-devel libattr-devel libtirpc-devel libblkid-devel openssl-devel libuuid-devel texinfo texinfo-tex
yum -y install audit-libs-devel binutils-devel elfutils-devel kabi-dw ncurses-devel newt-devel numactl-devel openssl-devel pciutils-devel perl perl-devel python2 python3-docutils xmlto xz-devel elfutils-libelf-devel libcap-devel libcap-ng-devel llvm-toolset libyaml libyaml-devel kernel-rpm-macros kernel-abi-whitelists opencsd-devel
dnf install -y epel-release
dnf install -y ccache pdsh
dnf --enablerepo=ha install resource-agents

  3. Install tools for kernel RPM build:

    Code Block
    languagebash
    themeEclipse
    dnf install -y bpftool dwarves java-devel libbabeltrace-devel libbpf-devel libmnl-devel net-tools rsync
# May only be needed on RHEL9 derivatives:
dnf install -y python3-devel

  4. Install e2fsprogs packages:

    Code Block
    languagebash
    themeEclipse
    git clone "https://review.whamcloud.com/tools/e2fsprogs" e2fsprogs && cd e2fsprogs && git checkout v1.47.01-wc1 && ./configure --with-root-prefix=/usr --enable-elf-shlibs --disable-uuidd --disable-fsck --disable-e2initrd-helper --disable-libblkid --disable-libuuid --enable-quota --disable-fuse2fs && make -j8
sudo make install
cd ..

    Or download and install the following e2fsprogs packages from https://build.whamcloud.com/

...

  1. First create the directory structure, then get the source from the RPM. Create a .rpmmacros file to install the kernel source in our user directory:

    Code Block
    languagebash
    themeEclipse
    cd $HOME && mkdir -p kernel/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd kernel && echo '%_topdir %(echo $HOME)/kernel/rpmbuild' > ~/.rpmmacros

  2. Install the kernel source:

    Code Block
    languagebash
    themeEclipse
    rpm -ivh https://dl.rockylinux.org/vault/rocky/8.7/BaseOS/source/tree/Packages/k/kernel-4.18.0-425.19.2.el8_7.src.rpm

  3. Prepare the source using rpmbuild:

    Code Block
    languagebash
    themeEclipse
    cd ~/kernel/rpmbuild && rpmbuild -bp --target=`uname -m` ./SPECS/kernel.spec

  4. Copy the kernel config file into lustre tree:

    Code Block
    languagebash
    themeEclipse
    cp ~/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8.`uname -m`/configs/kernel-4.18.0-`uname -m`.config ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

  5. Edit the kernel config file ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config:

    Find the line with '# IO Schedulers' and insert following two lines below it:

    CONFIG_IOSCHED_DEADLINE=y
    CONFIG_DEFAULT_IOSCHED="deadline"

    Or use cmd:

    Code Block
    languagebash
    themeEclipse
    sed -i '/# IO Schedulers/a CONFIG_IOSCHED_DEADLINE=y\nCONFIG_DEFAULT_IOSCHED="deadline"' ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config

  6. Gather all the patches from lustre tree into a single file:

    Code Block
    enableddltrue
    languagebash
    themeEclipse
    langBash
    globaltitle7. Copy the kernel patch into RPM build tree:
    cd ~/lustre-release/lustre/kernel_patches/series && \
for patch in $(<"4.18-rhel8.7.series"); do \
     patch_file="$HOME/lustre-release/lustre/kernel_patches/patches/${patch}"; \
     cat "${patch_file}" >> "$HOME/lustre-kernel-`uname -m`-lustre.patch"; \
done

  7. Copy the kernel patch into RPM build tree:

    Code Block
    languagebash
    themeEclipse
    cp ~/lustre-kernel-`uname -m`-lustre.patch ~/kernel/rpmbuild/SOURCES/patch-4.18.0-lustre.patch

  8. Edit the kernel spec file ~/kernel/rpmbuild/SPECS/kernel.spec:

    Code Block
    languagebash
    themeEclipse
    sed -i.inst -e '/^    find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
    cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
    rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec

  9. Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.

    Code Block
    languagebash
    themeEclipse
    echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-`uname -m`.config
cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-4.18.0-`uname -m`.config
  10. sed -i.inst -e '/^    find $RPM_BUILD_ROOT\/lib\/modules\/$KernelVer/a\
    cp -a fs/ext4/* $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4\
    rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/fs/ext4/ext4-inode-test*' \
-e '/^# empty final patch to facilitate testing of kernel patches/i\
Patch99995: patch-%{version}-lustre.patch' \
-e '/^ApplyOptionalPatch linux-kernel-test.patch/i\
ApplyOptionalPatch patch-%{version}-lustre.patch' \
~/kernel/rpmbuild/SPECS/kernel.spec

  11. Overwrite the kernel config file with ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config. Change "x86_64" with "arm64" in the first command for aarch64 architecture.

    Code Block
    languagebash
    themeEclipse
    echo '# x86_64' > ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
cat ~/lustre-release/lustre/kernel_patches/kernel_configs/kernel-4.18.0-4.18-rhel8.7-`uname -m`.config >> ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config
  12. (optional - only for aarch64 VMs on Apple Silicon Macs) As outlined in the Apple Silicon guide the kernel page size must be set to 4K  or 16K . ~/kernel/rpmbuild/SOURCES/kernel-`uname -m`.config uses 64K  and needs to be modified.  Otherwise the Lustre kernel won't boot. Set the page size to 4K either via make menuconfig or manually. Diff before and after the change: 
    Code Block
    languagebash
    themeEclipse
    collapsetrue
    14,18c13,17
< CONFIG_ARM64_PAGE_SHIFT=16
< CONFIG_ARM64_CONT_SHIFT=5
< CONFIG_ARCH_MMAP_RND_BITS_MIN=14
< CONFIG_ARCH_MMAP_RND_BITS_MAX=29
< CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=7
---
> CONFIG_ARM64_PAGE_SHIFT=12
> CONFIG_ARM64_CONT_SHIFT=4
> CONFIG_ARCH_MMAP_RND_BITS_MIN=18
> CONFIG_ARCH_MMAP_RND_BITS_MAX=33
> CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=11
35c34
< CONFIG_PGTABLE_LEVELS=3
---
> CONFIG_PGTABLE_LEVELS=4
301c300
< CONFIG_ARCH_MMAP_RND_BITS=14
---
> CONFIG_ARCH_MMAP_RND_BITS=18
389a389
> #
622c622
< # CONFIG_ARM64_4K_PAGES is not set
---
> CONFIG_ARM64_4K_PAGES=y
624,625c624,625
< CONFIG_ARM64_64K_PAGES=y
< # CONFIG_ARM64_VA_BITS_42 is not set
---
> # CONFIG_ARM64_64K_PAGES is not set
> # CONFIG_ARM64_VA_BITS_39 is not set
627d626
< # CONFIG_ARM64_VA_BITS_52 is not set
629,631c628,629
< # CONFIG_ARM64_PA_BITS_48 is not set
< CONFIG_ARM64_PA_BITS_52=y
< CONFIG_ARM64_PA_BITS=52
---
> CONFIG_ARM64_PA_BITS_48=y
> CONFIG_ARM64_PA_BITS=48
658a657
> CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
727c726
< CONFIG_FORCE_MAX_ZONEORDER=14
---
> CONFIG_FORCE_MAX_ZONEORDER=11
793a793
> # CONFIG_COMPAT is not set
2625a2626
> # CONFIG_VMXNET3 is not set

  13. Start building the kernel with Start building the kernel with rpmbuild:

    Code Block
    languagebash
    themeEclipse
    cd ~/kernel/rpmbuild && buildid="_lustre" && \
rpmbuild -ba --with firmware --target `uname -m` --with baseonly \
           --without kabichk --define "buildid ${buildid}" \
           ~/kernel/rpmbuild/SPECS/kernel.spec

  14. Install the kernel packages:

    Code Block
    languagebash
    themeEclipse
    cd ~/kernel/rpmbuild/RPMS/`uname -m`/
sudo rpm -Uvh --replacepkgs --force kernel-*.rpm
sudo reboot

  15. Login system after reboot, run name -r and see:

    # uname -r
    4.18.0-425.19.2.el8_lustre.x86_64

    or

    4.18.0-425.19.2.el8_lustre.aarch64

...

  1. Configure Lustre source:

    Code Block
    languagebash
    themeEclipse
    cd lustre-release
./configure --with-linux=/home/build/kernel/rpmbuild/BUILD/kernel-4.18.0-425.19.2.el8_7/linux-4.18.0-425.19.2.el8_lustre.`uname -m`/ --disable-gss --disable-shared --disable-crypto

  2. Build and install Lustre:

    Code Block
    languagebash
    themeEclipse
    make -j8
sudo make install
sudo depmod -a

  3. Run /usr/lib64/lustre/tests/llmount.sh and tests lustre:

    Code Block
    languagebash
    themeEclipse
    /usr/lib64/lustre/tests/llmount.sh

    Stopping clients: sr050 /mnt/lustre (opts:-f)
    Stopping clients: sr050 /mnt/lustre2 (opts:-f)
    sr050: executing set_hostid
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    libcfs will create CPU partition based on online CPUs
    Formatting mgs, mds, osts
    Format mds1: /tmp/lustre-mdt1
    Format ost1: /tmp/lustre-ost1
    Format ost2: /tmp/lustre-ost2
    Checking servers environments
    Checking clients sr050 environments
    Loading modules from /usr/lib64/lustre/tests/..
    detected 8 online CPUs by sysfs
    libcfs will create CPU partition based on online CPUs
    Setup mgs, mdt, osts
    Starting mds1: -o localrecov  /dev/mapper/mds1_flakey /mnt/lustre-mds1
    Commit the device label on /tmp/lustre-mdt1
    Started lustre-MDT0000
    Starting ost1: -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
    Commit the device label on /tmp/lustre-ost1
    Started lustre-OST0000
    Starting ost2: -o localrecov  /dev/mapper/ost2_flakey /mnt/lustre-ost2
    Commit the device label on /tmp/lustre-ost2
    Started lustre-OST0001
    Starting client: sr050:  -o user_xattr,flock sr050@tcp:/lustre /mnt/lustre
    UUID                   1K-blocks        Used   Available Use% Mounted on
    lustre-MDT0000_UUID       125056        1644      112176   2% /mnt/lustre[MDT:0]
    lustre-OST0000_UUID       313104        1244      284700   1% /mnt/lustre[OST:0]
    lustre-OST0001_UUID       313104        1244      284700   1% /mnt/lustre[OST:1]
     
    filesystem_summary:       626208        2488      569400   1% /mnt/lustre
     
    Using TIMEOUT=20
    osc.lustre-OST0000-osc-ffffce9052651800.idle_timeout=debug
    osc.lustre-OST0001-osc-ffffce9052651800.idle_timeout=debug
    setting jobstats to procname_uid
    Setting lustre.sys.jobid_var from disable to procname_uid
    Waiting 90s for 'procname_uid'
    Updated after 4s: want 'procname_uid' got 'procname_uid'
    disable quota as required
    lod.lustre-MDT0000-mdtlov.mdt_hash=crush

  4. You will now have a Lustre filesystem available at /mnt/lustre

...