Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

Lustre server machines are simple to setup with CentOS 6.2. Pre-built kernel images are available from Whamclouds build server. A number of different kernels and tools are continually built and tested. For this Walk-thru the lustre-master build will be used.

Whamcloud Lustre GA

Whamcloud produce Maintenance Releases and Feature Releases. This walk through covers running Lustre from the Master release. The Master release is subject to continuous integration testing but may have known or unknown issues. If you are evaluating Lustre for your production environment, you are advised to contact Whamcloud and substitute http://downloads.whamcloud.com/public/ for the RPMs you need for the walk-thru below.

Prerequisite.

  • A x86_64 machine with CentOS 6.2 installed. The machine has the hostname rhel6_vanilla.

Procedure

Install Lustre patched kernel.

  1. From lustre-master build, download the correct kernel RPMs for your platform
    • kernel
    • kernel-firmware
  2. Install these RPMs with yum localinstall ./kernel*
    [root@rhel6_vanilla ~]# yum localinstall ./kernel*
    Loaded plugins: fastestmirror, security
    Setting up Local Package Process
    Examining ./kernel-2.6.32-220.el6_lustre.x86_64.rpm: kernel-2.6.32-220.el6_lustre.x86_64
    Marking ./kernel-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-220.el6.x86_64
    base                                                                                  | 3.7 kB     00:00
    base/primary_db                                                                       | 4.5 MB     00:02
    extras                                                                                | 3.5 kB     00:00
    extras/primary_db                                                                     | 6.3 kB     00:00
    updates                                                                               | 3.5 kB     00:00
    updates/primary_db                                                                    | 1.2 MB     00:00
    Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64
    Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch
    Resolving Dependencies
    --> Running transaction check
    ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed
    --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64
    --> Running transaction check
    ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated
    ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =============================================================================================================
     Package              Arch        Version                    Repository                                 Size
    =============================================================================================================
    Installing:
     kernel               x86_64      2.6.32-220.el6_lustre      /kernel-2.6.32-220.el6_lustre.x86_64      112 M
    Updating for dependencies:
     kernel-firmware      noarch      2.6.32-220.4.1.el6         updates                                   6.2 M
    
    Transaction Summary
    =============================================================================================================
    Install       1 Package(s)
    Upgrade       1 Package(s)
    
    Total size: 118 M
    Total download size: 6.2 M
    Is this ok [y/N]: y
    Downloading Packages:
    kernel-firmware-2.6.32-220.4.1.el6.noarch.rpm                                         | 6.2 MB     00:01
    warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
    Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    Importing GPG key 0xC105B9DE:
     Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>
     Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2)
     From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    Is this ok [y/N]: y
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Updating   : kernel-firmware-2.6.32-220.4.1.el6.noarch                                                 1/3
      Installing : kernel-2.6.32-220.el6_lustre.x86_64                                                       2/3
      Cleanup    : kernel-firmware-2.6.32-220.el6.noarch                                                     3/3
    
    Installed:
      kernel.x86_64 0:2.6.32-220.el6_lustre
    
    Dependency Updated:
      kernel-firmware.noarch 0:2.6.32-220.4.1.el6
    
    Complete!
    [root@rhel6_vanilla ~]#
    

Download e2fsprogs and Lustre server RPMSs.

  1. Download the following e2fsprogs (and dependencies) RPMs from http://build.whamcloud.com/job/e2fsprogs-master
    • e2fsprogs
    • e2fsprogs-libs
    • libcom_err
    • libss
  2. Download the following Lustre server RPMs from lustre-master build.
    • lustre
    • lustre-ldiskfs
    • lustre-modules
    • lustre-tests

Install Lustre.

  1. yum localinstall ./lustre-* ./e2fsprogs-* ./lib* should do it:
    [root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-* ./lib*
    Loaded plugins: fastestmirror, refresh-packagekit, security
    Setting up Local Package Process
    Examining ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64
    Marking ./lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed
    Loading mirror speeds from cached hostfile
     * base: centos-distro.cavecreek.net
     * extras: mirror.raystedman.net
     * updates: centosv.centos.org
    Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64
    Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed
    Examining ./lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64
    Marking ./lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed
    Examining ./lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm: lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64
    Marking ./lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64.rpm to be installed
    Examining ./e2fsprogs-1.42.3.wc1-7.el6.x86_64.rpm: e2fsprogs-1.42.3.wc1-7.el6.x86_64
    Marking ./e2fsprogs-1.42.3.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-11.el6.x86_64
    Examining ./e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64.rpm: e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64
    Marking ./e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-11.el6.x86_64
    Examining ./libcom_err-1.42.3.wc1-7.el6.x86_64.rpm: libcom_err-1.42.3.wc1-7.el6.x86_64
    Marking ./libcom_err-1.42.3.wc1-7.el6.x86_64.rpm as an update to libcom_err-1.41.12-11.el6.x86_64
    Examining ./libss-1.42.3.wc1-7.el6.x86_64.rpm: libss-1.42.3.wc1-7.el6.x86_64
    Marking ./libss-1.42.3.wc1-7.el6.x86_64.rpm as an update to libss-1.41.12-11.el6.x86_64
    Resolving Dependencies
    --> Running transaction check
    ---> Package e2fsprogs.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package e2fsprogs.x86_64 0:1.42.3.wc1-7.el6 will be an update
    ---> Package e2fsprogs-libs.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package e2fsprogs-libs.x86_64 0:1.42.3.wc1-7.el6 will be an update
    ---> Package libcom_err.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package libcom_err.x86_64 0:1.42.3.wc1-7.el6 will be an update
    ---> Package libss.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package libss.x86_64 0:1.42.3.wc1-7.el6 will be an update
    ---> Package lustre.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed
    ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed
    ---> Package lustre-modules.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed
    ---> Package lustre-tests.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5 will be installed
    --> Processing Dependency: libmpi.so.1()(64bit) for package: lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64
    --> Running transaction check
    ---> Package openmpi.x86_64 0:1.5.3-3.el6 will be installed
    --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libnuma.so.1(libnuma_1.2)(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: environment-modules for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libesmtp.so.5()(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libibverbs.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: libnuma.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Processing Dependency: librdmacm.so.1()(64bit) for package: openmpi-1.5.3-3.el6.x86_64
    --> Running transaction check
    ---> Package environment-modules.x86_64 0:3.2.7b-6.el6 will be installed
    --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.7b-6.el6.x86_64
    ---> Package libesmtp.x86_64 0:1.0.4-15.el6 will be installed
    ---> Package libibverbs.x86_64 0:1.1.5-3.el6 will be installed
    ---> Package librdmacm.x86_64 0:1.0.14.1-3.el6 will be installed
    ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed
    --> Running transaction check
    ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =============================================================================================================================
     Package             Arch   Version           Repository                                                                Size
    =============================================================================================================================
    Installing:
     lustre              x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5
                                                  /lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64          26 M
     lustre-ldiskfs      x86_64 3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5
                                                  /lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64  8.8 M
     lustre-modules      x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5
                                                  /lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64 154 M
     lustre-tests        x86_64 2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5
                                                  /lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64    13 M
    Updating:
     e2fsprogs           x86_64 1.42.3.wc1-7.el6  /e2fsprogs-1.42.3.wc1-7.el6.x86_64                                       2.6 M
     e2fsprogs-libs      x86_64 1.42.3.wc1-7.el6  /e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64                                  344 k
     libcom_err          x86_64 1.42.3.wc1-7.el6  /libcom_err-1.42.3.wc1-7.el6.x86_64                                       56 k
     libss               x86_64 1.42.3.wc1-7.el6  /libss-1.42.3.wc1-7.el6.x86_64                                            67 k
    Installing for dependencies:
     environment-modules x86_64 3.2.7b-6.el6      base                                                                      96 k
     libesmtp            x86_64 1.0.4-15.el6      base                                                                      57 k
     libibverbs          x86_64 1.1.5-3.el6       base                                                                      44 k
     librdmacm           x86_64 1.0.14.1-3.el6    base                                                                      27 k
     numactl             x86_64 2.0.3-9.el6       base                                                                      55 k
     openmpi             x86_64 1.5.3-3.el6       base                                                                     2.1 M
     tcl                 x86_64 1:8.5.7-6.el6     base                                                                     1.9 M
    
    Transaction Summary
    =============================================================================================================================
    Install      11 Package(s)
    Upgrade       4 Package(s)
    
    Total size: 210 M
    Total download size: 4.3 M
    Is this ok [y/N]: y
    Downloading Packages:
    (1/7): environment-modules-3.2.7b-6.el6.x86_64.rpm                                                    |  96 kB     00:00     
    (2/7): libesmtp-1.0.4-15.el6.x86_64.rpm                                                               |  57 kB     00:00     
    (3/7): libibverbs-1.1.5-3.el6.x86_64.rpm                                                              |  44 kB     00:00     
    (4/7): librdmacm-1.0.14.1-3.el6.x86_64.rpm                                                            |  27 kB     00:00     
    (5/7): numactl-2.0.3-9.el6.x86_64.rpm                                                                 |  55 kB     00:00     
    (6/7): openmpi-1.5.3-3.el6.x86_64.rpm                                                                 | 2.1 MB     00:01     
    (7/7): tcl-8.5.7-6.el6.x86_64.rpm                                                                     | 1.9 MB     00:01     
    -----------------------------------------------------------------------------------------------------------------------------
    Total                                                                                        849 kB/s | 4.3 MB     00:05     
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Updating   : libcom_err-1.42.3.wc1-7.el6.x86_64                                                                       1/19
      Updating   : e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64                                                                   2/19
      Installing : libibverbs-1.1.5-3.el6.x86_64                                                                            3/19
      Installing : librdmacm-1.0.14.1-3.el6.x86_64                                                                          4/19
      Updating   : libss-1.42.3.wc1-7.el6.x86_64                                                                            5/19
      Updating   : e2fsprogs-1.42.3.wc1-7.el6.x86_64                                                                        6/19
      Installing : lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                   7/19
      Installing : lustre-modules-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                  8/19
      Installing : lustre-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                          9/19
      Installing : 1:tcl-8.5.7-6.el6.x86_64                                                                                10/19
      Installing : environment-modules-3.2.7b-6.el6.x86_64                                                                 11/19
      Installing : libesmtp-1.0.4-15.el6.x86_64                                                                            12/19
      Installing : numactl-2.0.3-9.el6.x86_64                                                                              13/19
      Installing : openmpi-1.5.3-3.el6.x86_64                                                                              14/19
      Installing : lustre-tests-2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5.x86_64                                   15/19
      Cleanup    : e2fsprogs-1.41.12-11.el6.x86_64                                                                         16/19
      Cleanup    : e2fsprogs-libs-1.41.12-11.el6.x86_64                                                                    17/19
      Cleanup    : libss-1.41.12-11.el6.x86_64                                                                             18/19
      Cleanup    : libcom_err-1.41.12-11.el6.x86_64                                                                        19/19
    
    Installed:
      lustre.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                          
      lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                   
      lustre-modules.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                  
      lustre-tests.x86_64 0:2.2.54-2.6.32_220.17.1.el6_lustre.x86_64_g4bd9bf5                                                    
    
    Dependency Installed:
      environment-modules.x86_64 0:3.2.7b-6.el6       libesmtp.x86_64 0:1.0.4-15.el6       libibverbs.x86_64 0:1.1.5-3.el6      
      librdmacm.x86_64 0:1.0.14.1-3.el6               numactl.x86_64 0:2.0.3-9.el6         openmpi.x86_64 0:1.5.3-3.el6         
      tcl.x86_64 1:8.5.7-6.el6                       
    
    Updated:
      e2fsprogs.x86_64 0:1.42.3.wc1-7.el6    e2fsprogs-libs.x86_64 0:1.42.3.wc1-7.el6    libcom_err.x86_64 0:1.42.3.wc1-7.el6   
      libss.x86_64 0:1.42.3.wc1-7.el6       
    
    Complete!
    [root@rhel6_vanilla ~]#
    

Disable SELinux for Lustre patched kernel.

Lustre does not play well with SELinux (at least, the llmount.sh script fails when SELinux is enabled), so SELinux must be disabled. One option is to disable SELinux with GRUB. With your editor of choice, open /boot/grub/grub.conf. Append the option selinux=0 to the end of the kernel line for the Lustre patched kernel entry and save. This will cause SELinux to be disabled after you reboot into the patched kernel.

Reboot.

  1. reboot

Verify that installation is complete:

  1. uname -a
    [root@rhel6_vanilla ~]# uname -a
    Linux rhel6_vanilla 2.6.32-220.el6_lustre.x86_64 #1 SMP Wed Jan 25 12:21:36 PST 2012 x86_64 x86_64 x86_64 GNU/Linux
    
  2. Create a local Lustre filesystem with: llmount.sh
    [root@rhel6_vanilla ~]# /usr/lib64/lustre/tests/llmount.sh
    Stopping clients: rhel6_vanilla /mnt/lustre (opts:)
    Stopping clients: rhel6_vanilla /mnt/lustre2 (opts:)
    Loading modules from /usr/lib64/lustre/tests/..
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    gss/krb5 is not supported
    quota/lquota options: 'hash_lqs_cur_bits=3'
    Formatting mgs, mds, osts
    Format mds1: /tmp/lustre-mdt1
    Format ost1: /tmp/lustre-ost1
    Format ost2: /tmp/lustre-ost2
    Checking servers environments
    Checking clients rhel6_vanilla environments
    Loading modules from /usr/lib64/lustre/tests/..
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    gss/krb5 is not supported
    Setup mgs, mdt, osts
    Starting mds1: -o loop,user_xattr,acl  /tmp/lustre-mdt1 /mnt/mds1
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-MDT0000
    Starting ost1: -o loop  /tmp/lustre-ost1 /mnt/ost1
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-OST0000
    Starting ost2: -o loop  /tmp/lustre-ost2 /mnt/ost2
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-OST0001
    Starting client: rhel6_vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcp:/lustre /mnt/lustre
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Using TIMEOUT=20
    disable quota as required
    [root@rhel6_vanilla ~]# mount
    /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw)
    /dev/sda1 on /boot type ext4 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl)
    /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1)
    /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2)
    rhel6_vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)
    

Testing

Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem

  • No labels