Need the correct vmlinux and vmcore (don't need the systemmap if you have the correct vmlinux)
-> system.map depends on how the kernel is compiled.
-> debug kernel info
helps to have the debug info for the modules
bt -a # gives you stack trace for all the CPUs
ps - task list in condensed form
bt -f
mod -S /usr/lib/debug/usr/lib/modules/
to disassemble function
dis <function name>
example: to find the first argument on lnet_destroy_peer_ni_locked()
PID: 107343 TASK: ffff883cee985c00 CPU: 50 COMMAND: "socknal_sd05_00" #0 [ffff883ce36dbb38] machine_kexec at ffffffff81051beb #1 [ffff883ce36dbb98] crash_kexec at ffffffff810f2602 #2 [ffff883ce36dbc68] panic at ffffffff8162eb21 #3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs] #4 [ffff883ce36dbd08] lnet_destroy_peer_ni_locked at ffffffffa09a2f96 [lnet] #5 [ffff883ce36dbd28] lnet_return_tx_credits_locked at ffffffffa0993cec [lnet] #6 [ffff883ce36dbd68] lnet_msg_decommit at ffffffffa0987630 [lnet] #7 [ffff883ce36dbd98] lnet_finalize at ffffffffa0987e19 [lnet] #8 [ffff883ce36dbe00] ksocknal_tx_done at ffffffffa087aed4 [ksocklnd] #9 [ffff883ce36dbe30] ksocknal_scheduler at ffffffffa087fc92 [ksocklnd] #10 [ffff883ce36dbec8] kthread at ffffffff810a5acf #11 [ffff883ce36dbf50] ret_from_fork at ffffffff81645998
reference http://www.x86-64.org/documentation/abi.pdf
first disassemble the code
rbx: the passed in parameter, but it could be overwritten later on.
So the next
crash> dis lnet_destroy_peer_ni_locked 0xffffffffa09a2cb0 <lnet_destroy_peer_ni_locked>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa09a2cb5 <lnet_destroy_peer_ni_locked+5>: push %rbp 0xffffffffa09a2cb6 <lnet_destroy_peer_ni_locked+6>: mov %rsp,%rbp 0xffffffffa09a2cb9 <lnet_destroy_peer_ni_locked+9>: push %r12 0xffffffffa09a2cbb <lnet_destroy_peer_ni_locked+11>: push %rbx 0xffffffffa09a2cbc <lnet_destroy_peer_ni_locked+12>: mov 0xb8(%rdi),%edx 0xffffffffa09a2cc2 <lnet_destroy_peer_ni_locked+18>: mov %rdi,%rbx 0xffffffffa09a2cc5 <lnet_destroy_peer_ni_locked+21>: test %edx,%edx
lbug_with_lock will definitely save the rbx on the stack, so we go there to find the address. disassemble lbug_with_lock
crash> dis lbug_with_loc 0xffffffffa0912d30 <lbug_with_loc>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp 0xffffffffa0912d36 <lbug_with_loc+6>: xor %eax,%eax 0xffffffffa0912d38 <lbug_with_loc+8>: mov $0xffffffffa092fe94,%rsi 0xffffffffa0912d3f <lbug_with_loc+15>: mov %rsp,%rbp 0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx <<<<<<<<< pushes it on the stack 0xffffffffa0912d43 <lbug_with_loc+19>: mov %rdi,%rbx 0xffffffffa0912d46 <lbug_with_loc+22>: sub $0x8,%rsp 0xffffffffa0912d4a <lbug_with_loc+26>: movl $0x1,0x4ca54(%rip) # 0xffffffffa095f7a8 <libcfs_catastrophe>
View the stack for lbug_with_loc()
bt -f
#3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs]
ffff883ce36dbcf0: ffff8fbcec316010 ffff8abccf727e00
ffff883ce36dbd00: ffff883ce36dbd20 ffffffffa09a2f96
To interpret the stack. Bottom of the stack is the first entry pushed.
The call instruction will push the return address on the stack. In the above example
ffffffffa09a2f96 (sym <return address> : designated by fffff -> shows the location in the function to which the caller would return after it's done)
0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp ---> ffff883ce36dbd20
0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx ---> ffff8abccf727e00
#> struct lnet_peer_ni ffff8abccf727e00
To print a field in the structure you can:
#> struct lnet_peer_ni.<fieldname> <address>
To print all untyped values in hex:
#> set radix 16
help command should be helpful for further information.