Introduction

It is extremely useful to be able to do kernel remote live debugging. After picking Oleg Drokin's mind, I documented the below instructions to give a very basic overview to get started:

Instructions

  1. Create a KVM image
  2. Update the the KVM xml file:
    1. sudo virsh edit <uuid>
 
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>


###and at the end add:


  <qemu:commandline>
    <qemu:arg value='-gdb'/>
    <qemu:arg value='tcp::1200'/> ## 1200 is the TCP port that gdb will connect on
  </qemu:commandline>
</domain>
    2. On Ubuntu host the "sudo virsh edit" approach didn't seem to update the VM's definition. The following approach work:
      1. virsh dumpxml test03 > test03.xml
vi test03.xml # Update it as above
virsh define test03.xml
  3. Grab the vmlinux and unzip it 
    1. /boot/vmlinux-2.6.32.431.23.3.el6_lustre.bz2
    2. place the vmlinux on y our guest OS

  4. On Centos 7.5 guest and above, disable Kernel Address Space Randomization (KASLR ): add "nokaslr" in GRUB_CMDLINE_LINUX line of /etc/default/grub, then make sure changes take effect:  

    grub2-mkconfig -o /boot/grub2/grub.cfg
  5. Download and place the attached .gdbinit in your home directory
  6. Place the modules with debugging symbols on your host machine
  7. Startup the guest OS
  8. On the guest, load the modules you wish to debug: lnet.ko for example.
  9. On the host, startup gdb with the vmlinux 


    gdb vmlinux-2.6.32.431.23.3.el6_lustre

    In GDB on the guest: 

    (gdb) target remote localhost:1200
## This is defined int he .gdbinit and it will list all the addresses of the loaded modules
(gdb) mod-list-syms
add-symbol-file lnet.ko 0xffffffffa03c8000
add-symbol-file sha512_generic.ko 0xffffffffa00a8000
add-symbol-file sha256_generic.ko 0xffffffffa0065000
add-symbol-file crc32c_intel.ko 0xffffffffa0023000
add-symbol-file libcfs.ko 0xffffffffa0356000
add-symbol-file fuse.ko 0xffffffffa0231000
add-symbol-file autofs4.ko 0xffffffffa034e000
add-symbol-file sunrpc.ko 0xffffffffa02f5000
add-symbol-file bnx2fc.ko 0xffffffffa02d7000
....
## Add the symbols to GDB using the module with debugging symbols you copied to your host machine
(gdb) add-symbol-file /path/to/lnet.ko 0xffffffffa03c8000
## Add breakpoints as you wish
## continue execution 
(gdb) continue

    By this point the symbols for the module you wish to debug is loaded in GDB and GDB is attached to your guest OS.

  10. Add strategic breakpoints to enhance your debugging
  11. Invoke the scenario you wish to debug
  12. GDB will break at the desired break points, allowing you to debug as you normally would a user space program from gdb.