You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

As part of work funded by OpenSFS, a plugin for Clang has been developed. This plugin, together with gcov, can be used  to identify code in the Lustre source tree that is never called. In addition, these tools can be used to more generally illuminate code usage. For example, hot and cold areas of data and functions. The purpose of this document is to enable developer to begin working with these tools.

Setting up the environment

  1. Setup a RHEL 6.3 (or 6.4, or 6.5) image capable of building the kernel and Lustre source code. Use a disk image with at least 24 GB.
    1. gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4). gcc that is packaged with EL 6.5 also works.
    2. install git
  2. Install Clang and LLVM. RPMs are available from elrepo. Clang/LLVM version 3.0 has good parity with gcc-4.4. Parity with gcc-4.4 is important as gcc is used during the analysis.

    • llvm-devel-3.0-5.el6.elrepo.x86_64

    • llvm-libs-3.0-5.el6.elrepo.x86_64
    • llvm-doc-3.0-5.el6.elrepo.noarch
    • llvm-3.0-5.el6.elrepo.x86_64
    • clang-devel-3.0-5.el6.elrepo.x86_64
    • clang-3.0-5.el6.elrepo.x86_64
    • clang-analyzer-3.0-5.el6.elrepo.x86_64

    • clang-doc-3.0-5.el6.elrepo.noarch

      elrepo-extras

      Once elrepo is available as a repo, only the main channel is available by default. To install llvm and clang you need the elrepo-extras repo. This can be enabled by including

           --enablerepo=elrepo-extras

      in your yum command.

  3. Download lustre-static-analysis Clang plugin: 
         git clone git://review.whamcloud.com:29418/tools/lustre-static-analysis/
  4. Make the contents of Endian, DeclUse and Utils directories.

Building Lustre for static analysis

  1. Build a Lustre patched kernel with ggov enabled. This document assumes a Lustre patched kernel version: 2.6.32-279.19.1.el6_lustre_gcov.x86_64.

    1. To enable gcov set CONFIG_GCOV_KERNEL=y. It is not necessary to set CONFIG_GCOV_PROFILE_ALL. With the first option set, profiling can be enable on a per-module basis during the Lustre build.

    2. Build the kernel as usual (i.e. using GCC). Trying to build it with Clang is unlikely to succeed.

  2. Build SPL and ZFS if necessary.

  3. Build Lustre software using a script included in lustre-static-analysis for assistance:

    # cd lustre-release
# git clean -fxd
# sh autogen.sh && ./configure
# ~/lustre-static-analysis/DeclUse/decl_use_make GCOV_PROFILE=y

    For each .c to .o compilation performed by GCC, decl_use_make will also run Clang with the lustre-static-analysis plugin. When compiling file.c the output file.o is produced by GCC and should be identical to a normal Lustre build. Simultaneously decl_use_make instructs Clang to read file.c and produce corresponding file.o._Ref and file.o._Err. Note: Clang does not compile file.c to an object file. Clang parses file.c and produces an AST in memory. The file.o._Err file records the errors Clang discovered during processing file.c.

    In practice, file.o is first build as .tmp_file.o then renamed to file.o. This happens in the Makefile so GCC and Clang are unaware of the file.o.

Products from Clang static analysis

For the purposes of static analysis provided by the Clang plugin, the file.o._Ref file is of primary interest. For example:

# find lustre/llite -name '*._Ref'
./lustre/llite/.tmp_rw.o._Ref
./lustre/llite/.tmp_dir.o._Ref
./lustre/llite/.tmp_vvp_lock.o._Ref
./lustre/llite/.tmp_remote_perm.o._Ref
...
#
# ls -lh ./lustre/llite/.tmp_file.o._Ref
-rw-r--r-- 1 root root 7.9M May 16 10:47 ./lustre/llite/.tmp_file.o._Ref
# grep ll_file_read ./lustre/llite/.tmp_file.o._Ref
Decl  Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:1062:16
Def   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:1062:16
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3026:27
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3046:27
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3069:27
#
# grep lu_fid ./lustre/llite/.tmp_file.o._Ref
Decl  Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:118:8
Def   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:118:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq  lustre/include/lustre/lustre_user.h:125:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_oid  lustre/include/lustre/lustre_user.h:127:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_ver  lustre/include/lustre/lustre_user.h:133:8
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:137:16
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:142:16
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:170:17
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:189:17
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:379:16
#
# grep lu_fid:f_seq ./lustre/llite/.tmp_file.o._Ref
Decl  Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_user.h:125:8
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:395:21
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:548:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:650:14
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:747:8
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:771:6
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:772:3
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:820:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:835:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:846:7
#

In each ._Ref files, the columns from left to right are:

Column 1Column 2Column 3Column 4

The type of operation declared on the symbol. Either:

Decl = declared

Def = defined

Method

Use = used

The type of the symbol. Either:

Record

Field

Function

 

Canonical declaration of the symbol referencedLocation of the reference

Automatic processing of analysis output

Executing the script useage-post.sh will create a directory called Unused-2.4.50 (or more generally: Unused-$(git describe))

# ls -lh Unused-2.4.50/# ls -lh Unused-2.4.50/
total 97M
-rw-r--r-- 1 root root 2.7M May 16 15:48 Decl
-rw-r--r-- 1 root root 1.7M May 16 15:48 Def
-rw-r--r-- 1 root root 295K May 16 15:48 Err
-rw-r--r-- 1 root root 423K May 16 15:48 MacroDef
-rw-r--r-- 1 root root  76K May 16 15:48 MacroUnused
-rw-r--r-- 1 root root 349K May 16 15:48 MacroUse
-rw-r--r-- 1 root root  88M May 16 15:48 Ref
-rw-r--r-- 1 root root 1.1M May 16 15:48 Undefined
-rw-r--r-- 1 root root 152K May 16 15:48 Unused
-rw-r--r-- 1 root root 2.6M May 16 15:48 Use
# wc -l Unused-2.4.50/*
    33789 Unused-2.4.50/Decl
    20778 Unused-2.4.50/Def
     2322 Unused-2.4.50/Err
     5878 Unused-2.4.50/MacroDef
     1072 Unused-2.4.50/MacroUnused
     4830 Unused-2.4.50/MacroUse
   639812 Unused-2.4.50/Ref
    13011 Unused-2.4.50/Undefined
     1859 Unused-2.4.50/Unused
    32070 Unused-2.4.50/Use
   755421 total

The file Ref is the sorted and uniqed concatenation of the ._Ref files produced by usage-make.sh.  The remaining files are produced by filtering and comparing various parts of Ref. For example, there are the unused declarations in lustre/llite:

# grep /llite/ Unused-2.4.50/Unused
EnumConstant  lustre/llite/llite_internal.h:121:9:LLIF_CONTENDED
EnumConstant  lustre/llite/llite_internal.h:123:9:LLIF_SRVLOCK
EnumConstant  lustre/llite/llite_internal.h:379:9:STATS_TRACK_LAST
Field         lustre/llite/llite_internal.h:1020:8:vvp_thread_info:vti_lvb
Field         lustre/llite/llite_internal.h:1020:8:vvp_thread_info:vti_queue
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_fd
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_locked_list
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_root
Field         lustre/llite/llite_internal.h:1611:8:if_quotactl_18:obd_type
Field         lustre/llite/llite_internal.h:638:8:it_cb_data:hash
Field         lustre/llite/llite_internal.h:71:8:ll_dentry_data:lld_cwd_och
Field         lustre/llite/llite_internal.h:71:8:ll_dentry_data:lld_mnt_och
Field         lustre/llite/llite_internal.h:920:8:vvp_io:cui_partpage
Function      lustre/llite/llite_internal.h:1086:28:ll_node_from_inode
Function      lustre/llite/llite_internal.h:1154:19:ll_mds_max_easize
Function      lustre/llite/llite_internal.h:1216:5:ll_is_file_contended
Function      lustre/llite/llite_internal.h:663:20:ll_ra_read_get
Function      lustre/llite/llite_internal.h:713:6:ll_removepage
Function      lustre/llite/llite_internal.h:716:5:ll_file_punch
Function      lustre/llite/llite_internal.h:717:9:ll_file_lockless_io
Function      lustre/llite/llite_internal.h:718:6:ll_clear_file_contended
Function      lustre/llite/llite_internal.h:719:5:ll_sync_page_range
Function      lustre/llite/llite_internal.h:830:7:ll_read_opt
Function      lustre/llite/llite_internal.h:836:15:ll_inode_from_lock
Function      lustre/llite/llite_internal.h:864:6:lustre_dump_inode
Function      lustre/llite/llite_lib.c:1085:6:lu_context_keys_dump
Function      lustre/llite/llite_mmap.c:577:29:file_to_user
Function      lustre/llite/llite_mmap.c:58:14:ll_nopage
Function      lustre/llite/lloop.c:901:1:__inittest
Function      lustre/llite/lloop.c:902:1:__exittest
Function      lustre/llite/lloop.c:904:1:__check_max_loop
Function      lustre/llite/namei.c:78:5:ll_unlock
Function      lustre/llite/super25.c:232:1:__inittest
Function      lustre/llite/super25.c:233:1:__exittest
Function      lustre/llite/vvp_dev.c:172:5:vvp_global_init
Function      lustre/llite/vvp_dev.c:185:6:vvp_global_fini
Record        lustre/llite/llite_internal.h:1078:8:ll_lock_tree
Typedef       lustre/llite/llite_internal.h:1075:25:rb_node_t
Var           lustre/llite/llite_internal.h:658:25:ll_async_page_slab
Var           lustre/llite/llite_internal.h:659:15:ll_async_page_slab_size
Var           lustre/llite/llite_internal.h:84:31:ll_pgcache_seq_fops
Var           lustre/llite/llite_internal.h:884:31:ll_special_chr_inode_fops
Var           lustre/llite/llite_internal.h:885:31:ll_special_chr_file_fops
Var           lustre/llite/llite_internal.h:886:31:ll_special_blk_inode_fops
Var           lustre/llite/llite_internal.h:887:31:ll_special_fifo_inode_fops
Var           lustre/llite/llite_internal.h:888:31:ll_special_fifo_file_fops
Var           lustre/llite/llite_internal.h:889:31:ll_special_sock_inode_fops
Var           lustre/llite/rw26.c:559:33:ll_aops
Var           lustre/llite/vvp_dev.c:544:24:vvp_dump_pgcache_file_ops

Usage != declaration

Some of the variables listed above maybe recognised as in use. It is important to note that this is a list of variable declaration usage. The ._Err file (described later) is useful for correcting this situation.

Profiling

In order to perform profiling, Lustre must be up and running. Once Lustre is mounted, a suitable workload must be performed to drive the profiling tools. For example, a general work-load might-be:

# TESTS="sanity sanityn sanity-benchmark metadata-updates racer lnet-selftest replay-single conf-sanity recovery-small replay-ost-single replay-dual replay-vbr insanity sanity-quota sanity-sec sanity-gss lustre-rsync-test ost-pools mmp obdfilter-survey sgpdd-survey sanity-scrub lfsck sanity-hsm"
# export FSTYPE=ldiskfs
# export MDSCOUNT=2
# for test in $TESTS; do sh ./lustre/tests/$test.sh; done
...
# export FSTYPE=zfs
# export MDSCOUNT=2
# for test in $TESTS; do sh ./lustre/tests/$test.sh; done
...
# export SAHRED_DIRECTORY=/tmp/will-this-work
# export MDSCOUNT=1
# sh ./lustre/tests/lfsck.sh
#
...

To collect the profiling data first mount debugfs then run gcov-collect. GCOV helper scripts to be linked when available. Example output is:

# mount -t debugfs none /sys/kernel/debug
# ls -l /sys/kernel/debug/gcov/root/lustre-release/lustre/llite
-rw------- 1 root root 0 May 17 13:54 dcache.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 dcache.gcno -> /root/lustre-release/lustre/llite/.tmp_dcache.gcno
-rw------- 1 root root 0 May 17 13:54 dir.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 dir.gcno -> /root/lustre-release/lustre/llite/.tmp_dir.gcno
-rw------- 1 root root 0 May 17 13:54 file.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 file.gcno -> /root/lustre-release/lustre/llite/.tmp_file.gcno
-rw------- 1 root root 0 May 17 13:54 llite_capa.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 llite_capa.gcno -> /root/lustre-release/lustre/llite/.tmp_llite_capa.gcno
-rw------- 1 root root 0 May 17 13:54 llite_close.gcda
...
# mkdir /tmp/gcov-2.4.50
# gcov-collect /tmp/gcov-2.4.50
# ls -lh /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov
-rw------- 1 root root 164K May 16 21:15 /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov
# cat /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov
        -:    0:Source:/root/lustre-release/lustre/llite/file.c
        -:    0:Graph:/sys/kernel/debug/gcov/root/lustre-release/lustre/llite/file.gcno
        -:    0:Data:/sys/kernel/debug/gcov/root/lustre-release/lustre/llite/file.gcda
        -:    0:Runs:0
        -:    0:Programs:0
...
   588675: 1062:static ssize_t ll_file_read(struct file *file, char *buf, size_t count,
        -: 1063:                            loff_t *ppos)
        -: 1064:{
        -: 1065:        struct lu_env *env;
        -: 1066:        struct iovec  *local_iov;
        -: 1067:        struct kiocb  *kiocb;
        -: 1068:        ssize_t        result;
        -: 1069:        int            refcheck;
   588675: 1070:        ENTRY;
        -: 1071:
   588675: 1072:        env = cl_env_get(&refcheck);
   588734: 1073:        if (IS_ERR(env))
    #####: 1074:                RETURN(PTR_ERR(env));
        -: 1075:
   588734: 1076:        local_iov = &vvp_env_info(env)->vti_local_iov;
   588740: 1077:        kiocb = &vvp_env_info(env)->vti_kiocb;
   588740: 1078:        local_iov->iov_base = (void __user *)buf;
   588740: 1079:        local_iov->iov_len = count;
  1177480: 1080:        init_sync_kiocb(kiocb, file);
   588740: 1081:        kiocb->ki_pos = *ppos;
   588740: 1082:        kiocb->ki_left = count;
        -: 1083:
   588740: 1084:        result = ll_file_aio_read(kiocb, local_iov, 1, kiocb->ki_pos);
   588716: 1085:        *ppos = kiocb->ki_pos;
        -: 1086:
   588716: 1087:        cl_env_put(env, &refcheck);
   588741: 1088:        RETURN(result);
        -: 1089:}
...

The output from gcov reads as follows

Column 1Column 2Column3
A count of times the line was executed. '-' means no code was generated. '#####' means count=0.The number of the line of interest in the source file.The line of interest.

The output from this profiling phase can be used to prune the ._Ref data to provide an enhanced usage picture. To do this, we eliminate references form dead lines ('#####').

# export LC_ALL=C# cd Unused-2.4.50# gcov-filter.py /tmp/gcov-2.4.50 Ref > LiveRef# awk '$1 == "Use" && $2 != "Macro" { print $2, $3; }' LiveRef | sort | uniq > LiveUse# comm -23 Decl LiveUse > Dead# wc -l Dead3277 Dead## comm -23 Dead Unused | grep /llite/Field     lustre/llite/llite_internal.h:992:9:-:sendfileField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_access_permField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_fsgidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_fsuidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_gidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_listField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_uidField     lustre/llite/llite_nfs.c:118:8:lustre_nfs_fid:lnf_childField     lustre/llite/llite_nfs.c:118:8:lustre_nfs_fid:lnf_parentFunction  lustre/llite/dir.c:919:12:ll_ioc_copy_startFunction  lustre/llite/dir.c:992:12:ll_ioc_copy_endFunction  lustre/llite/file.c:1261:12:ll_lov_recreateFunction  lustre/llite/file.c:1307:12:ll_lov_recreate_objFunction  lustre/llite/file.c:1325:12:ll_lov_recreate_fidFunction  lustre/llite/file.c:1523:12:ll_lov_getstripeFunction  lustre/llite/file.c:1760:12:ll_ioctl_fiemap...Record    lustre/llite/llite_nfs.c:118:8:lustre_nfs_fidVar       lustre/llite/llite_capa.c:67:27:ll_capa_renewedVar       lustre/llite/llite_capa.c:68:27:ll_capa_renewal_noentVar       lustre/llite/llite_capa.c:69:27:ll_capa_renewal_failedVar       lustre/llite/llite_capa.c:70:27:ll_capa_renewal_retriesVar       lustre/llite/llite_internal.h:725:31:ll_file_operations

._Err output file

Explain _Err file.

Err Function /root/lustre-release/lustre/fid/fid_request.c:155:5:seq_client_alloc_super first declared without linkage in c file 
Err Function /root/lustre-release/lustre/fid/fid_store.c:69:6:seq_update_cb first declared without linkage in c file 
Err Function /root/lustre-release/lustre/fid/fid_store.c:82:5:seq_update_cb_add first declared without linkage in c file 
Err Function /root/lustre-release/lustre/fld/fld_request.c:420:5:fld_client_rpc first declared without linkage in c file 
Err Function /root/lustre-release/lustre/include/lclient.h:388:17:ccc_vmpage_page_transient was already declared 
Err Function /root/lustre-release/lustre/include/liblustre.h:165:6:cfs_get_random_bytes was already declared 
Err Function /root/lustre-release/lustre/include/lu_target.h:308:5:tgt_hpreq_handler was already declared 
Err Function /root/lustre-release/lustre/include/lustre_dlm.h:1414:5:ldlm_handle_enqueue0 was already declared 
Err Function /root/lustre-release/lustre/include/lustre_mdc.h:200:5:it_disposition was already declared 
Err Function /root/lustre-release/lustre/include/lustre_mdc.h:201:6:it_clear_disposition was already declared 
Err Function /root/lustre-release/lustre/include/lustre_mdc.h:202:6:it_set_disposition was already declared 
Err Function /root/lustre-release/lustre/include/obd_class.h:1927:5:class_add_uuid was already declared 
Err Function /root/lustre-release/lustre/include/obd_class.h:87:20:class_conn2export was already declared 
Err Function /root/lustre-release/lustre/ldlm/ldlm_flock.c:72:5:ldlm_flock_blocking_ast first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_internal.h:163:5:ldlm_lock_remove_from_lru was already declared 
Err Function /root/lustre-release/lustre/ldlm/ldlm_internal.h:244:5:ldlm_init was already declared 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lib.c:2129:6:target_recovery_fini first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lib.c:2150:6:target_recovery_init first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lib.c:2398:5:target_send_reply_msg first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lock.c:2214:5:ldlm_cancel_locks_for_export_cb first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lock.c:361:5:ldlm_lock_destroy_internal first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lock.c:711:6:ldlm_add_bl_work_item first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lock.c:732:6:ldlm_add_cp_work_item first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lockd.c:2420:5:ldlm_revoke_lock_cb first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_lockd.c:77:19:round_timeout first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_resource.c:318:6:ldlm_namespace_proc_unregister first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_resource.c:330:5:ldlm_namespace_proc_register first declared without linkage in c file 
Err Function /root/lustre-release/lustre/ldlm/ldlm_resource.c:960:5:ldlm_namespace_get_return first declared without linkage in c file

 

Gotchas

  • The static usage tracker is confused by disconnected declarations.

 

TODO:

  • No labels