Static analysis of the Lustre source code with Clang/LLVM

As part of work funded by OpenSFS in 2013, a plugin for Clang has been developed. This plugin, together with gcov, can be used to identify dead code in the Lustre source tree. In addition, these tools can be used to more generally illuminate code usage. For example, hot and cold spots in functions and data and identifying suspicious patterns all become possible using these tools. The purpose of this document is to enable developer to begin working with these tools.

Setting up the environment

Setup a RHEL 6.3 (or 6.4) image capable of building the kernel and Lustre source code. Use a disk image with at least 16 GB.
1. gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)
Install Clang and LLVM. RPMs are avaliable from elrepo. Clang/LLVM version 3.0 has good parity with gcc-4.4. Parity wiht gcc-4.4 is important as gcc is used during the analysis.
- llvm-devel-3.0-5.el6.elrepo.x86_64
- llvm-libs-3.0-5.el6.elrepo.x86_64
- llvm-doc-3.0-5.el6.elrepo.noarch
- llvm-3.0-5.el6.elrepo.x86_64
- clang-devel-3.0-5.el6.elrepo.x86_64
- clang-3.0-5.el6.elrepo.x86_64
- clang-analyzer-3.0-5.el6.elrepo.x86_64
- clang-doc-3.0-5.el6.elrepo.noarch
Download ll-clang and build the LL.so plugin. NOT AVAILABLE YET.

Building Lustre for static analysis

Build a Lustre patched kernel with ggov enabled. This document assumes a Lustre patched kernel version: 2.6.32-279.19.1.el6_lustre_gcov.x86_64.
1. To enable gcov set CONFIG_GCOV_KERNEL=y. It is not necessary to set CONFIG_GCOV_PROFILE_ALL. With the first option set, profiling can be enable on a per-module basis by adding GCOV_PROFILE=y on the make command line.
2. Build the kernel as usual (i.e. using GCC). Trying to build it with Clang is unlikely to succeed.
Build SPL and ZFS if necessary.
Using the scripts in the ll-clang dowload, build Lustre software:
```
# cd lustre-release
# git clean -fxd
# sh autogen.sh && ./configure
# ~/ll-clang/usage-make.sh GCOV_PROFILE=y
# ~/ll-clang/usage-post.sh
```
For each .c to .o compilation performed by GCC, usage-make.sh will also run Clang with the LL.so plugin. When compiling file.c the output file.o is produced by GCC and should be identical to a normal Lustre build. Simultaneously usage-make.sh instructs Clang to read file.c and produce corresponding file.o._Ref and file.o._Err. Note: Clang does not compile file.c to an object file. Clang parses file.c and produces an AST in memory. The file.o._Err file records the errors Clang discovered during processing file.c.
In practice, file.o is first build as .tmp_file.o then renamed to file.o. This happens in the Makefile so GCC and Clang are unaware of the file.o.

Products from Clang static analysis

For the purposes of static analysis provided by the Clang plugin, the file.o._Ref file is of primary interest. For example:

# find lustre/llite -name '*._Ref'
./lustre/llite/.tmp_rw.o._Ref
./lustre/llite/.tmp_dir.o._Ref
./lustre/llite/.tmp_vvp_lock.o._Ref
./lustre/llite/.tmp_remote_perm.o._Ref
...
#
# ls -lh ./lustre/llite/.tmp_file.o._Ref
-rw-r--r-- 1 root root 7.9M May 16 10:47 ./lustre/llite/.tmp_file.o._Ref
# grep ll_file_read ./lustre/llite/.tmp_file.o._Ref
Decl  Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:1062:16
Def   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:1062:16
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3026:27
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3046:27
Use   Function  lustre/llite/file.c:1062:16:ll_file_read  lustre/llite/file.c:3069:27
#
# grep lu_fid ./lustre/llite/.tmp_file.o._Ref
Decl  Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:118:8
Def   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:118:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq  lustre/include/lustre/lustre_user.h:125:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_oid  lustre/include/lustre/lustre_user.h:127:8
Decl  Field     lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_ver  lustre/include/lustre/lustre_user.h:133:8
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:137:16
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:142:16
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:170:17
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:189:17
Use   Record    lustre/include/lustre/lustre_user.h:118:8:lu_fid        lustre/include/lustre/lustre_user.h:379:16
#
# grep lu_fid:f_seq ./lustre/llite/.tmp_file.o._Ref
Decl  Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_user.h:125:8
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:395:21
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:548:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:650:14
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:747:8
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:771:6
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:772:3
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:820:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:835:7
Use   Field  lustre/include/lustre/lustre_user.h:118:8:lu_fid:f_seq     lustre/include/lustre/lustre_idl.h:846:7
#

In each ._Ref files, the columns from left to right are:

Column 1

Column 2

Column 3

Column 4

The type of operation declared on the symbol. Either:

Decl = declared

Def = defined

Use = used

The type of the symbol. Either:

Record

Field

Function

Canonical declaration of the symbol referenced

Location of the reference

Automatic processing of analysis output

Executing the script useage-post.sh will create a directory called Unused-2.4.50 (or more generally: Unused-$(git describe))

# ls -lh Unused-2.4.50/# ls -lh Unused-2.4.50/
total 97M
-rw-r--r-- 1 root root 2.7M May 16 15:48 Decl
-rw-r--r-- 1 root root 1.7M May 16 15:48 Def
-rw-r--r-- 1 root root 295K May 16 15:48 Err
-rw-r--r-- 1 root root 423K May 16 15:48 MacroDef
-rw-r--r-- 1 root root  76K May 16 15:48 MacroUnused
-rw-r--r-- 1 root root 349K May 16 15:48 MacroUse
-rw-r--r-- 1 root root  88M May 16 15:48 Ref
-rw-r--r-- 1 root root 1.1M May 16 15:48 Undefined
-rw-r--r-- 1 root root 152K May 16 15:48 Unused
-rw-r--r-- 1 root root 2.6M May 16 15:48 Use
# wc -l Unused-2.4.50/*
    33789 Unused-2.4.50/Decl
    20778 Unused-2.4.50/Def
     2322 Unused-2.4.50/Err
     5878 Unused-2.4.50/MacroDef
     1072 Unused-2.4.50/MacroUnused
     4830 Unused-2.4.50/MacroUse
   639812 Unused-2.4.50/Ref
    13011 Unused-2.4.50/Undefined
     1859 Unused-2.4.50/Unused
    32070 Unused-2.4.50/Use
   755421 total

The file Ref is the sorted and uniqed concatenation of the ._Ref files produced by usage-make.sh. The remaining files are produced by filtering and comparing various parts of Ref. For example, there are the unused declarations in lustre/llite:

# grep /llite/ Unused-2.4.50/Unused
EnumConstant  lustre/llite/llite_internal.h:121:9:LLIF_CONTENDED
EnumConstant  lustre/llite/llite_internal.h:123:9:LLIF_SRVLOCK
EnumConstant  lustre/llite/llite_internal.h:379:9:STATS_TRACK_LAST
Field         lustre/llite/llite_internal.h:1020:8:vvp_thread_info:vti_lvb
Field         lustre/llite/llite_internal.h:1020:8:vvp_thread_info:vti_queue
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_fd
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_locked_list
Field         lustre/llite/llite_internal.h:1078:8:ll_lock_tree:lt_root
Field         lustre/llite/llite_internal.h:1611:8:if_quotactl_18:obd_type
Field         lustre/llite/llite_internal.h:638:8:it_cb_data:hash
Field         lustre/llite/llite_internal.h:71:8:ll_dentry_data:lld_cwd_och
Field         lustre/llite/llite_internal.h:71:8:ll_dentry_data:lld_mnt_och
Field         lustre/llite/llite_internal.h:920:8:vvp_io:cui_partpage
Function      lustre/llite/llite_internal.h:1086:28:ll_node_from_inode
Function      lustre/llite/llite_internal.h:1154:19:ll_mds_max_easize
Function      lustre/llite/llite_internal.h:1216:5:ll_is_file_contended
Function      lustre/llite/llite_internal.h:663:20:ll_ra_read_get
Function      lustre/llite/llite_internal.h:713:6:ll_removepage
Function      lustre/llite/llite_internal.h:716:5:ll_file_punch
Function      lustre/llite/llite_internal.h:717:9:ll_file_lockless_io
Function      lustre/llite/llite_internal.h:718:6:ll_clear_file_contended
Function      lustre/llite/llite_internal.h:719:5:ll_sync_page_range
Function      lustre/llite/llite_internal.h:830:7:ll_read_opt
Function      lustre/llite/llite_internal.h:836:15:ll_inode_from_lock
Function      lustre/llite/llite_internal.h:864:6:lustre_dump_inode
Function      lustre/llite/llite_lib.c:1085:6:lu_context_keys_dump
Function      lustre/llite/llite_mmap.c:577:29:file_to_user
Function      lustre/llite/llite_mmap.c:58:14:ll_nopage
Function      lustre/llite/lloop.c:901:1:__inittest
Function      lustre/llite/lloop.c:902:1:__exittest
Function      lustre/llite/lloop.c:904:1:__check_max_loop
Function      lustre/llite/namei.c:78:5:ll_unlock
Function      lustre/llite/super25.c:232:1:__inittest
Function      lustre/llite/super25.c:233:1:__exittest
Function      lustre/llite/vvp_dev.c:172:5:vvp_global_init
Function      lustre/llite/vvp_dev.c:185:6:vvp_global_fini
Record        lustre/llite/llite_internal.h:1078:8:ll_lock_tree
Typedef       lustre/llite/llite_internal.h:1075:25:rb_node_t
Var           lustre/llite/llite_internal.h:658:25:ll_async_page_slab
Var           lustre/llite/llite_internal.h:659:15:ll_async_page_slab_size
Var           lustre/llite/llite_internal.h:84:31:ll_pgcache_seq_fops
Var           lustre/llite/llite_internal.h:884:31:ll_special_chr_inode_fops
Var           lustre/llite/llite_internal.h:885:31:ll_special_chr_file_fops
Var           lustre/llite/llite_internal.h:886:31:ll_special_blk_inode_fops
Var           lustre/llite/llite_internal.h:887:31:ll_special_fifo_inode_fops
Var           lustre/llite/llite_internal.h:888:31:ll_special_fifo_file_fops
Var           lustre/llite/llite_internal.h:889:31:ll_special_sock_inode_fops
Var           lustre/llite/rw26.c:559:33:ll_aops
Var           lustre/llite/vvp_dev.c:544:24:vvp_dump_pgcache_file_ops

Usage != declaration

Some of the variables listed above maybe recognised as in use. It is important to note that this is a list of variable declaration usage. The ._Err file (described later) is useful for correcting this situation.

Profiling

In order to perform profiling, Lustre must be up and running. Once Lustre is mounted, a suitable workload must be performed to drive the profiling tools. For example, a general work-load might-be:

# TESTS="sanity sanityn sanity-benchmark metadata-updates racer lnet-selftest replay-single conf-sanity recovery-small replay-ost-single replay-dual replay-vbr insanity sanity-quota sanity-sec sanity-gss lustre-rsync-test ost-pools mmp obdfilter-survey sgpdd-survey sanity-scrub lfsck sanity-hsm"
# export FSTYPE=ldiskfs
# export MDSCOUNT=2
# for test in $TESTS; do sh ./lustre/tests/$test.sh; done
...
# export FSTYPE=zfs
# export MDSCOUNT=2
# for test in $TESTS; do sh ./lustre/tests/$test.sh; done
...
# export SAHRED_DIRECTORY=/tmp/will-this-work
# export MDSCOUNT=1
# sh ./lustre/tests/lfsck.sh
#
...

To collect the profiling data first mount debugfs then run gcov-collect. GCOV helper scripts to be linked when available. Example output is:

# mount -t debugfs none /sys/kernel/debug
# ls -l /sys/kernel/debug/gcov/root/lustre-release/lustre/llite
-rw------- 1 root root 0 May 17 13:54 dcache.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 dcache.gcno -> /root/lustre-release/lustre/llite/.tmp_dcache.gcno
-rw------- 1 root root 0 May 17 13:54 dir.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 dir.gcno -> /root/lustre-release/lustre/llite/.tmp_dir.gcno
-rw------- 1 root root 0 May 17 13:54 file.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 file.gcno -> /root/lustre-release/lustre/llite/.tmp_file.gcno
-rw------- 1 root root 0 May 17 13:54 llite_capa.gcda
lrwxrwxrwx 1 root root 0 May 17 13:54 llite_capa.gcno -> /root/lustre-release/lustre/llite/.tmp_llite_capa.gcno
-rw------- 1 root root 0 May 17 13:54 llite_close.gcda
...
# mkdir /tmp/gcov-2.4.50
# gcov-collect /tmp/gcov-2.4.50
# ls -lh /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov 
-rw------- 1 root root 164K May 16 21:15 /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov
# cat /tmp/gcov-2.4.50/root/lustre-release/lustre/llite/file.c.gcov
        -:    0:Source:/root/lustre-release/lustre/llite/file.c
        -:    0:Graph:/sys/kernel/debug/gcov/root/lustre-release/lustre/llite/file.gcno
        -:    0:Data:/sys/kernel/debug/gcov/root/lustre-release/lustre/llite/file.gcda
        -:    0:Runs:0
        -:    0:Programs:0
...
   588675: 1062:static ssize_t ll_file_read(struct file *file, char *buf, size_t count,
        -: 1063:                            loff_t *ppos)
        -: 1064:{
        -: 1065:        struct lu_env *env;
        -: 1066:        struct iovec  *local_iov;
        -: 1067:        struct kiocb  *kiocb;
        -: 1068:        ssize_t        result;
        -: 1069:        int            refcheck;
   588675: 1070:        ENTRY;
        -: 1071:
   588675: 1072:        env = cl_env_get(&refcheck);
   588734: 1073:        if (IS_ERR(env))
    #####: 1074:                RETURN(PTR_ERR(env));
        -: 1075:
   588734: 1076:        local_iov = &vvp_env_info(env)->vti_local_iov;
   588740: 1077:        kiocb = &vvp_env_info(env)->vti_kiocb;
   588740: 1078:        local_iov->iov_base = (void __user *)buf;
   588740: 1079:        local_iov->iov_len = count;
  1177480: 1080:        init_sync_kiocb(kiocb, file);
   588740: 1081:        kiocb->ki_pos = *ppos;
   588740: 1082:        kiocb->ki_left = count;
        -: 1083:
   588740: 1084:        result = ll_file_aio_read(kiocb, local_iov, 1, kiocb->ki_pos);
   588716: 1085:        *ppos = kiocb->ki_pos;
        -: 1086:
   588716: 1087:        cl_env_put(env, &refcheck);
   588741: 1088:        RETURN(result);
        -: 1089:}
...

The output from gcov reads as follows

Column 1	Column 2	Column3
A count of times the line was executed. '-' means no code was generated. '#####' means count=0.	The number of the line of interest in the source file.	The line of interest.

The output from this profiling phase can be used to prune the ._Ref data to provide an enhanced usage picture. To do this, we eliminate references form dead lines ('#####').

# export LC_ALL=C# cd Unused-2.4.50# gcov-filter.py /tmp/gcov-2.4.50 Ref > LiveRef# awk '$1 == "Use" && $2 != "Macro" { print $2, $3; }' LiveRef | sort | uniq > LiveUse# comm -23 Decl LiveUse > Dead# wc -l Dead3277 Dead## comm -23 Dead Unused | grep /llite/Field     lustre/llite/llite_internal.h:992:9:-:sendfileField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_access_permField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_fsgidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_fsuidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_gidField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_listField     lustre/llite/llite_internal.h:99:8:ll_remote_perm:lrp_uidField     lustre/llite/llite_nfs.c:118:8:lustre_nfs_fid:lnf_childField     lustre/llite/llite_nfs.c:118:8:lustre_nfs_fid:lnf_parentFunction  lustre/llite/dir.c:919:12:ll_ioc_copy_startFunction  lustre/llite/dir.c:992:12:ll_ioc_copy_endFunction  lustre/llite/file.c:1261:12:ll_lov_recreateFunction  lustre/llite/file.c:1307:12:ll_lov_recreate_objFunction  lustre/llite/file.c:1325:12:ll_lov_recreate_fidFunction  lustre/llite/file.c:1523:12:ll_lov_getstripeFunction  lustre/llite/file.c:1760:12:ll_ioctl_fiemap...Record    lustre/llite/llite_nfs.c:118:8:lustre_nfs_fidVar       lustre/llite/llite_capa.c:67:27:ll_capa_renewedVar       lustre/llite/llite_capa.c:68:27:ll_capa_renewal_noentVar       lustre/llite/llite_capa.c:69:27:ll_capa_renewal_failedVar       lustre/llite/llite_capa.c:70:27:ll_capa_renewal_retriesVar       lustre/llite/llite_internal.h:725:31:ll_file_operations

Gotchas

The static usage tracker is confused by disconnected declarations.

TODO:

Explain _Err file.

Page tree