Need the correct vmlinux and vmcore (don't need the systemmap if you have the correct vmlinux)
-> system.map depends on how the kernel is compiled.
-> debug kernel info
helps to have the debug info for the modules
bt -a # gives you stack trace for all the CPUs
ps - task list in condensed form
bt -f
mod -S /usr/lib/debug/usr/lib/modules/
to disassemble function
dis <function name>
example: to find the first argument on lnet_destroy_peer_ni_locked()
PID: 107343 TASK: ffff883cee985c00 CPU: 50 COMMAND: "socknal_sd05_00" #0 [ffff883ce36dbb38] machine_kexec at ffffffff81051beb #1 [ffff883ce36dbb98] crash_kexec at ffffffff810f2602 #2 [ffff883ce36dbc68] panic at ffffffff8162eb21 #3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs] #4 [ffff883ce36dbd08] lnet_destroy_peer_ni_locked at ffffffffa09a2f96 [lnet] #5 [ffff883ce36dbd28] lnet_return_tx_credits_locked at ffffffffa0993cec [lnet] #6 [ffff883ce36dbd68] lnet_msg_decommit at ffffffffa0987630 [lnet] #7 [ffff883ce36dbd98] lnet_finalize at ffffffffa0987e19 [lnet] #8 [ffff883ce36dbe00] ksocknal_tx_done at ffffffffa087aed4 [ksocklnd] #9 [ffff883ce36dbe30] ksocknal_scheduler at ffffffffa087fc92 [ksocklnd] #10 [ffff883ce36dbec8] kthread at ffffffff810a5acf #11 [ffff883ce36dbf50] ret_from_fork at ffffffff81645998 |
reference http://www.x86-64.org/documentation/abi.pdf
first disassemble the code
rbx: the passed in parameter, but it could be overwritten later on.
So the next
crash> dis lnet_destroy_peer_ni_locked 0xffffffffa09a2cb0 <lnet_destroy_peer_ni_locked>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa09a2cb5 <lnet_destroy_peer_ni_locked+5>: push %rbp 0xffffffffa09a2cb6 <lnet_destroy_peer_ni_locked+6>: mov %rsp,%rbp 0xffffffffa09a2cb9 <lnet_destroy_peer_ni_locked+9>: push %r12 0xffffffffa09a2cbb <lnet_destroy_peer_ni_locked+11>: push %rbx 0xffffffffa09a2cbc <lnet_destroy_peer_ni_locked+12>: mov 0xb8(%rdi),%edx 0xffffffffa09a2cc2 <lnet_destroy_peer_ni_locked+18>: mov %rdi,%rbx 0xffffffffa09a2cc5 <lnet_destroy_peer_ni_locked+21>: test %edx,%edx |
lbug_with_lock will definitely save the rbx on the stack, so we go there to find the address. disassemble lbug_with_lock
crash> dis lbug_with_loc 0xffffffffa0912d30 <lbug_with_loc>: nopl 0x0(%rax,%rax,1) [FTRACE NOP] 0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp 0xffffffffa0912d36 <lbug_with_loc+6>: xor %eax,%eax 0xffffffffa0912d38 <lbug_with_loc+8>: mov $0xffffffffa092fe94,%rsi 0xffffffffa0912d3f <lbug_with_loc+15>: mov %rsp,%rbp 0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx <<<<<<<<< pushes it on the stack 0xffffffffa0912d43 <lbug_with_loc+19>: mov %rdi,%rbx 0xffffffffa0912d46 <lbug_with_loc+22>: sub $0x8,%rsp 0xffffffffa0912d4a <lbug_with_loc+26>: movl $0x1,0x4ca54(%rip) # 0xffffffffa095f7a8 <libcfs_catastrophe> |
View the stack for lbug_with_loc()
bt -f
#3 [ffff883ce36dbce8] lbug_with_loc at ffffffffa0912ddb [libcfs]
ffff883ce36dbcf0: ffff8fbcec316010 ffff8abccf727e00
ffff883ce36dbd00: ffff883ce36dbd20 ffffffffa09a2f96 |
To interpret the stack. Bottom of the stack is the first entry pushed.
The call instruction will push the return address on the stack. In the above example
ffffffffa09a2f96 (sym <return address> : designated by fffff -> shows the location in the function to which the caller would return after it's done)
0xffffffffa0912d35 <lbug_with_loc+5>: push %rbp ---> ffff883ce36dbd20
0xffffffffa0912d42 <lbug_with_loc+18>: push %rbx ---> ffff8abccf727e00
#> struct lnet_peer_ni ffff8abccf727e00 |
To print a field in the structure you can:
#> struct lnet_peer_ni.<fieldname> <address> |
To print all untyped values in hex:
#> set radix 16 |
help command should be helpful for further information.