Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Next »

Lustre server machines are simple to setup with CentOS 6.2. Pre-built kernel images are available from Whamclouds build server. A number of different kernels and tools are continually built and tested. For this Walk-thru the lustre-master build will be used.

Whamcloud Lustre GA

Whamcloud produce Maintenance Releases and Feature Releases. This walk through covers running Lustre from the Master release. The Master release is subject to continuous integration testing but may have known or unknown issues. If you are evaluating Lustre for your production environment, you are advised to contact Whamcloud and substitute http://downloads.wham1cloud.com/public/ for the RPMs you need for the walk-thru below.

Prerequisite.

  • A x86_64 machine with CentOS 6.2 installed. The machine has the hostname rhel6_vanilla.

Procedure

Install Lustre patched kernel.

  1. From lustre-master build, download the correct kernel RPMs for your platform
    • kernel
    • kernel-firmware
  2. Install these RPMs with yum localinstall ./kernel*
    [root@rhel6_vanilla ~]# yum localinstall ./kernel*
    Loaded plugins: fastestmirror, security
    Setting up Local Package Process
    Examining ./kernel-2.6.32-220.el6_lustre.x86_64.rpm: kernel-2.6.32-220.el6_lustre.x86_64
    Marking ./kernel-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-2.6.32-220.el6.x86_64
    base                                                                                  | 3.7 kB     00:00     
    base/primary_db                                                                       | 4.5 MB     00:02     
    extras                                                                                | 3.5 kB     00:00     
    extras/primary_db                                                                     | 6.3 kB     00:00     
    updates                                                                               | 3.5 kB     00:00     
    updates/primary_db                                                                    | 1.2 MB     00:00     
    Examining ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm: kernel-firmware-2.6.32-220.el6_lustre.x86_64
    Marking ./kernel-firmware-2.6.32-220.el6_lustre.x86_64.rpm as an update to kernel-firmware-2.6.32-220.el6.noarch
    Resolving Dependencies
    --> Running transaction check
    ---> Package kernel.x86_64 0:2.6.32-220.el6_lustre will be installed
    --> Processing Dependency: kernel-firmware >= 2.6.32-220.el6_lustre for package: kernel-2.6.32-220.el6_lustre.x86_64
    --> Running transaction check
    ---> Package kernel-firmware.noarch 0:2.6.32-220.el6 will be updated
    ---> Package kernel-firmware.noarch 0:2.6.32-220.4.1.el6 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =============================================================================================================
     Package              Arch        Version                    Repository                                 Size
    =============================================================================================================
    Installing:
     kernel               x86_64      2.6.32-220.el6_lustre      /kernel-2.6.32-220.el6_lustre.x86_64      112 M
    Updating for dependencies:
     kernel-firmware      noarch      2.6.32-220.4.1.el6         updates                                   6.2 M
    
    Transaction Summary
    =============================================================================================================
    Install       1 Package(s)
    Upgrade       1 Package(s)
    
    Total size: 118 M
    Total download size: 6.2 M
    Is this ok [y/N]: y
    Downloading Packages:
    kernel-firmware-2.6.32-220.4.1.el6.noarch.rpm                                         | 6.2 MB     00:01     
    warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
    Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    Importing GPG key 0xC105B9DE:
     Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>
     Package: centos-release-6-2.el6.centos.7.x86_64 (@anaconda-CentOS-201112102333.x86_64/6.2)
     From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
    Is this ok [y/N]: y
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Updating   : kernel-firmware-2.6.32-220.4.1.el6.noarch                                                 1/3 
      Installing : kernel-2.6.32-220.el6_lustre.x86_64                                                       2/3 
      Cleanup    : kernel-firmware-2.6.32-220.el6.noarch                                                     3/3 
    
    Installed:
      kernel.x86_64 0:2.6.32-220.el6_lustre                                                                      
    
    Dependency Updated:
      kernel-firmware.noarch 0:2.6.32-220.4.1.el6                                                                
    
    Complete!
    [root@rhel6_vanilla ~]#
    

Download e2fsprogs and Lustre server RPMSs.

  1. Download the following e2fsprogs RPMs from http://build.whamcloud.com/job/e2fsprogs-master
    • e2fsprogs
    • e2fsprogs-libs
  2. Download the following Lustre server RPMs from lustre-master build.
    • lustre
    • lustre-ldiskfs
    • lustre-modules
    • lustre-tests

Install Lustre.

  1. yum localinstall ./lustre-* ./e2fsprogs-* should do it:
    [root@rhel6_vanilla ~]# yum localinstall ./lustre-* ./e2fsprogs-*
    Loaded plugins: fastestmirror, security
    Setting up Local Package Process
    Examining ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    Marking ./lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed
    Loading mirror speeds from cached hostfile
     * base: ftp.usf.edu
     * extras: mirror.raystedman.net
     * updates: mirror.raystedman.net
    Examining ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    Marking ./lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed
    Examining ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    Marking ./lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed
    Examining ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    Marking ./lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64.rpm to be installed
    Examining ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-1.41.90.wc4-7.el6.x86_64
    Marking ./e2fsprogs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-1.41.12-11.el6.x86_64
    Examining ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm: e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64
    Marking ./e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64.rpm as an update to e2fsprogs-libs-1.41.12-11.el6.x86_64
    Resolving Dependencies
    --> Running transaction check
    ---> Package e2fsprogs.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package e2fsprogs.x86_64 0:1.41.90.wc4-7.el6 will be an update
    ---> Package e2fsprogs-libs.x86_64 0:1.41.12-11.el6 will be updated
    ---> Package e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6 will be an update
    ---> Package lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed
    --> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    ---> Package lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed
    ---> Package lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed
    ---> Package lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b will be installed
    --> Processing Dependency: libmpi.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Processing Dependency: libopen-pal.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Processing Dependency: libopen-rte.so.0()(64bit) for package: lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
    --> Running transaction check
    ---> Package compat-openmpi.x86_64 0:1.4.3-1.el6 will be installed
    --> Processing Dependency: libibverbs.so.1(IBVERBS_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libnuma.so.1(libnuma_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: librdmacm.so.1(RDMACM_1.0)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libibverbs.so.1(IBVERBS_1.1)(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: environment-modules for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libibcm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: librdmacm.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libplpa.so.0()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libibverbs.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libgfortran.so.3()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    --> Processing Dependency: libnuma.so.1()(64bit) for package: compat-openmpi-1.4.3-1.el6.x86_64
    ---> Package net-snmp-libs.x86_64 1:5.5-37.el6_2.1 will be installed
    --> Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-37.el6_2.1.x86_64
    --> Running transaction check
    ---> Package environment-modules.x86_64 0:3.2.7b-6.el6 will be installed
    --> Processing Dependency: libtcl8.5.so()(64bit) for package: environment-modules-3.2.7b-6.el6.x86_64
    ---> Package libgfortran.x86_64 0:4.4.6-3.el6 will be installed
    ---> Package libibcm.x86_64 0:1.0.5-2.el6 will be installed
    ---> Package libibverbs.x86_64 0:1.1.5-3.el6 will be installed
    ---> Package librdmacm.x86_64 0:1.0.14.1-3.el6 will be installed
    ---> Package lm_sensors-libs.x86_64 0:3.1.1-10.el6 will be installed
    ---> Package numactl.x86_64 0:2.0.3-9.el6 will be installed
    ---> Package plpa-libs.x86_64 0:1.3.2-2.1.el6 will be installed
    --> Running transaction check
    ---> Package tcl.x86_64 1:8.5.7-6.el6 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =============================================================================================================
     Package             Arch   Version                                      Repository                     Size
    =============================================================================================================
    Installing:
     lustre              x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
                                                                                                            24 M
     lustre-ldiskfs      x86_64 3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b  /lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
                                                                                                           8.8 M
     lustre-modules      x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
                                                                                                           139 M
     lustre-tests        x86_64 2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b /lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64
                                                                                                            12 M
    Updating:
     e2fsprogs           x86_64 1.41.90.wc4-7.el6                            /e2fsprogs-1.41.90.wc4-7.el6.x86_64
                                                                                                           2.3 M
     e2fsprogs-libs      x86_64 1.41.90.wc4-7.el6                            /e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64
                                                                                                           308 k
    Installing for dependencies:
     compat-openmpi      x86_64 1.4.3-1.el6                                  base                          1.3 M
     environment-modules x86_64 3.2.7b-6.el6                                 base                           96 k
     libgfortran         x86_64 4.4.6-3.el6                                  base                          263 k
     libibcm             x86_64 1.0.5-2.el6                                  base                           19 k
     libibverbs          x86_64 1.1.5-3.el6                                  base                           44 k
     librdmacm           x86_64 1.0.14.1-3.el6                               base                           27 k
     lm_sensors-libs     x86_64 3.1.1-10.el6                                 base                           37 k
     net-snmp-libs       x86_64 1:5.5-37.el6_2.1                             updates                       1.5 M
     numactl             x86_64 2.0.3-9.el6                                  base                           55 k
     plpa-libs           x86_64 1.3.2-2.1.el6                                base                           25 k
     tcl                 x86_64 1:8.5.7-6.el6                                base                          1.9 M
    
    Transaction Summary
    =============================================================================================================
    Install      15 Package(s)
    Upgrade       2 Package(s)
    
    Total size: 192 M
    Total download size: 5.3 M
    Is this ok [y/N]: y
    Downloading Packages:
    (1/11): compat-openmpi-1.4.3-1.el6.x86_64.rpm                                         | 1.3 MB     00:00     
    (2/11): environment-modules-3.2.7b-6.el6.x86_64.rpm                                   |  96 kB     00:00     
    (3/11): libgfortran-4.4.6-3.el6.x86_64.rpm                                            | 263 kB     00:00     
    (4/11): libibcm-1.0.5-2.el6.x86_64.rpm                                                |  19 kB     00:00     
    (5/11): libibverbs-1.1.5-3.el6.x86_64.rpm                                             |  44 kB     00:00     
    (6/11): librdmacm-1.0.14.1-3.el6.x86_64.rpm                                           |  27 kB     00:00     
    (7/11): lm_sensors-libs-3.1.1-10.el6.x86_64.rpm                                       |  37 kB     00:00     
    (8/11): net-snmp-libs-5.5-37.el6_2.1.x86_64.rpm                                       | 1.5 MB     00:03     
    (9/11): numactl-2.0.3-9.el6.x86_64.rpm                                                |  55 kB     00:00     
    (10/11): plpa-libs-1.3.2-2.1.el6.x86_64.rpm                                           |  25 kB     00:00     
    (11/11): tcl-8.5.7-6.el6.x86_64.rpm                                                   | 1.9 MB     00:00     
    -------------------------------------------------------------------------------------------------------------
    Total                                                                        1.0 MB/s | 5.3 MB     00:05     
    Running rpm_check_debug
    Running Transaction Test
    Transaction Test Succeeded
    Running Transaction
      Installing : libibverbs-1.1.5-3.el6.x86_64                                                            1/19 
      Updating   : e2fsprogs-libs-1.41.90.wc4-7.el6.x86_64                                                  2/19 
      Updating   : e2fsprogs-1.41.90.wc4-7.el6.x86_64                                                       3/19 
      Installing : lustre-ldiskfs-3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64                        4/19 
      Installing : lustre-modules-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64                       5/19 
      Installing : libibcm-1.0.5-2.el6.x86_64                                                                                                                                             6/19 
      Installing : librdmacm-1.0.14.1-3.el6.x86_64                                                                                                                                        7/19 
      Installing : lm_sensors-libs-3.1.1-10.el6.x86_64                                                                                                                                    8/19 
      Installing : 1:net-snmp-libs-5.5-37.el6_2.1.x86_64                                                                                                                                  9/19 
      Installing : lustre-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64                                                                                                            10/19 
      Installing : libgfortran-4.4.6-3.el6.x86_64                                                                                                                                        11/19 
      Installing : 1:tcl-8.5.7-6.el6.x86_64                                                                                                                                              12/19 
      Installing : environment-modules-3.2.7b-6.el6.x86_64                                                                                                                               13/19 
      Installing : numactl-2.0.3-9.el6.x86_64                                                                                                                                            14/19 
      Installing : plpa-libs-1.3.2-2.1.el6.x86_64                                                                                                                                        15/19 
      Installing : compat-openmpi-1.4.3-1.el6.x86_64                                                                                                                                     16/19 
      Installing : lustre-tests-2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b.x86_64                                                                                                      17/19 
      Cleanup    : e2fsprogs-1.41.12-11.el6.x86_64                                                                                                                                       18/19 
      Cleanup    : e2fsprogs-libs-1.41.12-11.el6.x86_64                                                                                                                                  19/19 
    
    Installed:
      lustre.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b                                 lustre-ldiskfs.x86_64 0:3.3.0-2.6.32_220.el6_lustre.x86_64_g7d1e63b                        
      lustre-modules.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b                         lustre-tests.x86_64 0:2.1.55-2.6.32_220.el6_lustre.x86_64_g7d1e63b                         
    
    Dependency Installed:
      compat-openmpi.x86_64 0:1.4.3-1.el6  environment-modules.x86_64 0:3.2.7b-6.el6  libgfortran.x86_64 0:4.4.6-3.el6       libibcm.x86_64 0:1.0.5-2.el6  libibverbs.x86_64 0:1.1.5-3.el6  
      librdmacm.x86_64 0:1.0.14.1-3.el6    lm_sensors-libs.x86_64 0:3.1.1-10.el6      net-snmp-libs.x86_64 1:5.5-37.el6_2.1  numactl.x86_64 0:2.0.3-9.el6  plpa-libs.x86_64 0:1.3.2-2.1.el6 
      tcl.x86_64 1:8.5.7-6.el6            
    
    Updated:
      e2fsprogs.x86_64 0:1.41.90.wc4-7.el6                                                      e2fsprogs-libs.x86_64 0:1.41.90.wc4-7.el6                                                     
    
    Complete!
    

Reboot

  1. reboot

Verify that installation is complete:

  1. uname -a
    [root@rhel6_vanilla ~]# uname -a
    Linux rhel6_vanilla 2.6.32-220.el6_lustre.x86_64 #1 SMP Wed Jan 25 12:21:36 PST 2012 x86_64 x86_64 x86_64 GNU/Linux
    
  2. Create a local Lustre filesystem with: llmount.sh
    [root@rhel6_vanilla ~]# /usr/lib64/lustre/tests/llmount.sh 
    Stopping clients: rhel6_vanilla /mnt/lustre (opts:)
    Stopping clients: rhel6_vanilla /mnt/lustre2 (opts:)
    Loading modules from /usr/lib64/lustre/tests/..
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    gss/krb5 is not supported
    quota/lquota options: 'hash_lqs_cur_bits=3'
    Formatting mgs, mds, osts
    Format mds1: /tmp/lustre-mdt1
    Format ost1: /tmp/lustre-ost1
    Format ost2: /tmp/lustre-ost2
    Checking servers environments
    Checking clients rhel6_vanilla environments
    Loading modules from /usr/lib64/lustre/tests/..
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    gss/krb5 is not supported
    Setup mgs, mdt, osts
    Starting mds1: -o loop,user_xattr,acl  /tmp/lustre-mdt1 /mnt/mds1
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-MDT0000
    Starting ost1: -o loop  /tmp/lustre-ost1 /mnt/ost1
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-OST0000
    Starting ost2: -o loop  /tmp/lustre-ost2 /mnt/ost2
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Started lustre-OST0001
    Starting client: rhel6_vanilla: -o user_xattr,acl,flock rhel6_vanilla@tcp:/lustre /mnt/lustre
    debug=0x33f0404
    subsystem_debug=0xffb7e3ff
    debug_mb=10
    Using TIMEOUT=20
    disable quota as required
    [root@rhel6_vanilla ~]# mount
    /dev/mapper/VolGroup-lv_root on / type ext4 (rw)
    proc on /proc type proc (rw)
    sysfs on /sys type sysfs (rw)
    devpts on /dev/pts type devpts (rw,gid=5,mode=620)
    tmpfs on /dev/shm type tmpfs (rw)
    /dev/sda1 on /boot type ext4 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    /dev/loop0 on /mnt/mds1 type lustre (rw,loop=/dev/loop0,user_xattr,acl)
    /dev/loop1 on /mnt/ost1 type lustre (rw,loop=/dev/loop1)
    /dev/loop2 on /mnt/ost2 type lustre (rw,loop=/dev/loop2)
    rhel6_vanilla@tcp:/lustre on /mnt/lustre type lustre (rw,user_xattr,acl,flock)
    

Testing

Now that you have all the components of a Lustre system installed, you can test them.
This is covered in a separate wiki page: Testing a Lustre filesystem

  • No labels