Need the correct vmlinux and vmcore (don't need the systemmap if you have the correct vmlinux)
-> system.map depends on how the kernel is compiled.
-> debug kernel info
Debugging Kernel Crash dump
crash /boot/System.map-2.6.32.lustremaster vmlinux vmcore
--> vmlinux is located in ./BUILD/kernel-2.6.32.lustremaster/vmlinux
--> /var/crash/*/vmcore
It helps to have the debug info for the modules. This will allow crash to display source code line numbers as well as enable it to know the Lustre/LNet structures, which can then be printed.
This is accomplished by installing the debug info rpm: lustre-debuginfo-*.rpm
Then you load the kernel modules (including Lustre/LNet modules):
mod -S /usr/lib/debug/usr/lib/modules/ |
Once modules are loaded you cand perform the following commands
# Display stack trace for crashed task
bt
# gives you stack trace for all the CPUs
bt -a
# gives you task list in condensed form
ps
# give you more info on each call, including stack addresses.
bt -f
# print stack traces for all tasks
foreach bt | less
# print the stack trace for wanted task
bt [<PID> | <task pointer>]
# to examine type definitions
whatis <type name>
# EXAMPLE:
crash> whatis the_lnet
lnet_t the_lnet
crash> whatis lnet_t
typedef struct {
....
} lnet_t;
# examining global variables
crash> the_lnet
the_lnet = $1 = {
ln_cpt_table = 0xffff883cecde6940,
ln_cpt_number = 16,
ln_cpt_bits = 4,
ln_res_lock = 0xffff883ce8a8a1a0,
...
}
# examining local variables
<struct name> <address>
# EXAMPLE (more details below)
lnet_peer_ni <address> |
It is often necessary to print certain structures and their values for testing. In order to do that we need to find the pointer to the structure memory. To accomplish that we need some understanding of AMD64 assembly and registry usage:
reference http://www.x86-64.org/documentation/abi.pdf
First, disassemble function
dis <function name> |
Second, trace down the pointer. Best way to demonstrate is through an example.
We have the following assert triggered in lnet_destroy_peer_ni_locked().
lnet_destroy_peer_ni_locked(struct lnet_peer_ni *lpni) |
The stack trace
PID: 107343 TASK: ffff883cee985c00 CPU: 50 COMMAND: "socknal_sd05_00" #0 [ffff883ce36dbb38] machine_kexec at ffffffff81051beb #1 [ffff883ce36dbb98] crash_kexec at ffffffff810f2602 #2 [ffff883ce36dbc68] panic at ffffffff8162eb21 #3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs] #4 [ffff883ce36dbd08] lnet_destroy_peer_ni_locked at ffffffffa09a2f96 [lnet] #5 [ffff883ce36dbd28] lnet_return_tx_credits_locked at ffffffffa0993cec [lnet] #6 [ffff883ce36dbd68] lnet_msg_decommit at ffffffffa0987630 [lnet] #7 [ffff883ce36dbd98] lnet_finalize at ffffffffa0987e19 [lnet] #8 [ffff883ce36dbe00] ksocknal_tx_done at ffffffffa087aed4 [ksocklnd] #9 [ffff883ce36dbe30] ksocknal_scheduler at ffffffffa087fc92 [ksocklnd] #10 [ffff883ce36dbec8] kthread at ffffffff810a5acf #11 [ffff883ce36dbf50] ret_from_fork at ffffffff81645998 |
We would like to print out the passed in parameter: lpni
According to the reference above (Figure 3.4: Register Usage):
%rbx: callee-saved register; optionally used as base pointer %rdi: used to pass 1st argument to functions |
Our task becomes to track down through the disassembled code the usage of rdi and rbx
First disassemble the code for lnet_destroy_peer_ni_locked()
crash> dis lnet_destroy_peer_ni_locked 0xffffffffa09a2cb0 <lnet_destroy_peer_ni_locked>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa09a2cb5 <lnet_destroy_peer_ni_locked+5>: push %rbp 0xffffffffa09a2cb6 <lnet_destroy_peer_ni_locked+6>: mov %rsp,%rbp 0xffffffffa09a2cb9 <lnet_destroy_peer_ni_locked+9>: push %r12 0xffffffffa09a2cbb <lnet_destroy_peer_ni_locked+11>: push %rbx 0xffffffffa09a2cbc <lnet_destroy_peer_ni_locked+12>: mov 0xb8(%rdi),%edx 0xffffffffa09a2cc2 <lnet_destroy_peer_ni_locked+18>: mov %rdi,%rbx 0xffffffffa09a2cc5 <lnet_destroy_peer_ni_locked+21>: test %edx,%edx |
We can see the instruction
mov %rdi, %rbx |
This stores the content of %rdi into %rbx. But %rbx probably gets reused down the call stack. But if so, then its contents will need to be stored by the callee on the stack.
Therefore, lbug_with_lock will definitely save the rbx on the stack, so we go there to find the address. disassemble lbug_with_lock
crash> dis lbug_with_loc 0xffffffffa0912d30 <lbug_with_loc>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp 0xffffffffa0912d36 <lbug_with_loc+6>: xor %eax,%eax 0xffffffffa0912d38 <lbug_with_loc+8>: mov $0xffffffffa092fe94,%rsi 0xffffffffa0912d3f <lbug_with_loc+15>: mov %rsp,%rbp 0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx <<<<<<<<< pushes it on the stack 0xffffffffa0912d43 <lbug_with_loc+19>: mov %rdi,%rbx 0xffffffffa0912d46 <lbug_with_loc+22>: sub $0x8,%rsp 0xffffffffa0912d4a <lbug_with_loc+26>: movl $0x1,0x4ca54(%rip) # 0xffffffffa095f7a8 <libcfs_catastrophe> |
View the stack for lbug_with_loc()
bt -f
#3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs]
ffff883ce36dbcf0: ffff8fbcec316010 ffff8abccf727e00
ffff883ce36dbd00: ffff883ce36dbd20 ffffffffa09a2f96 |
To interpret the stack: Bottom of the stack (bottom right corner) is the first entry pushed. So order of pushed items on the stack would be
ffffffffa09a2f96
ffff883ce36dbd20
ffff8abccf727e00
ffff8fbcec316010
The first entry pushed on the stack is done by the call instruction which will push the return address on the stack. In the above example
ffffffffa09a2f96 (sym <return address> : designated by fffff -> shows the location in the function to which the caller would return after it's done) 0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp ---> ffff883ce36dbd20 0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx ---> ffff8abccf727e00 |
then, knowing the type of the structure we can print it out by providing the address
#> struct lnet_peer_ni ffff8abccf727e00 |
To print a field in the structure you can:
#> struct lnet_peer_ni.<fieldname> <address> |
To print all numerical untyped values in hex:
#> set radix 16 |
crash 'help' command provides further information.