Introduction

It is extremely useful to be able to do kernel remote live debugging. After picking Oleg Drokin's mind, I documented the below instructions to give a very basic overview to get started:

Instructions

  1. Create a KVM image
  2. Update the the KVM xml file:
    1. sudo virsh edit <uuid>
 
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>


###and at the end add:


  <qemu:commandline>
    <qemu:arg value='-gdb'/>
    <qemu:arg value='tcp::1200'/> ## 1200 is the TCP port that gdb will connect on
  </qemu:commandline>
</domain>
    2. On Ubuntu host the "sudo virsh edit" approach didn't seem to update the VM's definition. The following approach work:
      1. virsh dumpxml test03 > test03.xml
vi test03.xml # Update it as above
virsh define test03.xml
  3. Grab the vmlinux and unzip it 
    1. /boot/vmlinux-2.6.32.431.23.3.el6_lustre.bz2
    2. place the vmlinux on y our guest OS

  4. On Centos 7.5 guest and above, disable Kernel Address Space Randomization (KASLR ): add "nokaslr" in GRUB_CMDLINE_LINUX line of /etc/default/grub, then make sure changes take effect: 

    1. GRUB_CMDLINE_LINUX="rootdelay=3 nokaslr"
      grub2-mkconfig -o /boot/grub2/grub.cfg
  5. Download and place the attached .gdbinit in your home directory
  6. Place the modules with debugging symbols on your host machine
  7. Startup the guest OS
  8. On the guest, load the modules you wish to debug: lnet.ko for example.
  9. On the host, startup gdb with the vmlinux 


    gdb vmlinux-2.6.32.431.23.3.el6_lustre

    In GDB on the guest: 

    (gdb) target remote localhost:1200
## This is defined int he .gdbinit and it will list all the addresses of the loaded modules
(gdb) mod-list-syms
add-symbol-file lnet.ko 0xffffffffa03c8000
add-symbol-file sha512_generic.ko 0xffffffffa00a8000
add-symbol-file sha256_generic.ko 0xffffffffa0065000
add-symbol-file crc32c_intel.ko 0xffffffffa0023000
add-symbol-file libcfs.ko 0xffffffffa0356000
add-symbol-file fuse.ko 0xffffffffa0231000
add-symbol-file autofs4.ko 0xffffffffa034e000
add-symbol-file sunrpc.ko 0xffffffffa02f5000
add-symbol-file bnx2fc.ko 0xffffffffa02d7000
....
## Add the symbols to GDB using the module with debugging symbols you copied to your host machine
(gdb) add-symbol-file /path/to/lnet.ko 0xffffffffa03c8000
## Add breakpoints as you wish
## continue execution 
(gdb) continue

    By this point the symbols for the module you wish to debug is loaded in GDB and GDB is attached to your guest OS.

  10. Add strategic breakpoints to enhance your debugging
  11. Invoke the scenario you wish to debug
  12. GDB will break at the desired break points, allowing you to debug as you normally would a user space program from gdb.

Later Kernels

Later kernels come packed with a script vmlinux-gdb.py 

This can be used instead of the script above.

ashehata@sn300:/OS/5.4.0$ gdb vmlinux
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vmlinux...
warning: File "/OS/5.4.0/vmlinux-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/ashehata/OS/5.4.0/vmlinux-gdb.py
line to your configuration file "/home/ashehata/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/ashehata/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) set auto-load safe-path /
(gdb) target remote localhost:1202
(gdb) source OS/5.4.0/vmlinux-gdb.py
# if you have the kernel modules in the local directory (which is what I do), you can do:
(gdb) lx-symbols 
scanning for modules in /home/ashehata/OS/5.4.0
loading @0xffffffffc04f6000: /OS/5.4.0/kvm-amd.ko
loading @0xffffffffc05cb000: /OS/5.4.0/ccp.ko
loading @0xffffffffc0421000: /OS/5.4.0/kvm.ko
loading @0xffffffffc0416000: /OS/5.4.0/crct10dif-pclmul.ko
loading @0xffffffffc0386000: /OS/5.4.0/ghash-clmulni-intel.ko
...

Once the symbols are loaded you can commence debugging.

https://www.kernel.org/doc/html/v4.14/dev-tools/gdb-kernel-debugging.html