...

1. Create a KVM image
2. Update the the KVM xml file:

   1. Code Block
      sudo virsh edit <uuid>   <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> ###and at the end add: <qemu:commandline> <qemu:arg value='-gdb'/> <qemu:arg value='tcp::1200'/> ## 1200 is the TCP port that gdb will connect on </qemu:commandline> </domain>

   2. On Ubuntu host the "sudo virsh edit" approach didn't seem to update the VM's definition. The following approach work:

      1. Code Block
         virsh dumpxml test03 > test03.xml vi test03.xml # Update it as above virsh define test03.xml

3. Grab the vmlinux and unzip it 

   1. Code Block
      /boot/vmlinux-2.6.32.431.23.3.el6_lustre.bz2

   2. place the vmlinux on y our guest OS

4. On Centos 7.5 guest and above, disable Kernel Address Space Randomization (KASLR ): add "nokaslr" in GRUB_CMDLINE_LINUX line of /etc/default/grub, then make sure changes take effect: 

   1. Code Block
      GRUB_CMDLINE_LINUX="rootdelay=3 nokaslr"

      Code Block
      grub2-mkconfig -o /boot/grub2/grub.cfg

5. Download and place the attached .gdbinit in your home directory
6. Place the modules with debugging symbols on your host machine
7. Startup the guest OS
8. On the guest, load the modules you wish to debug: lnet.ko for example.
9. On the host, startup gdb with the vmlinux 

   
   

   Code Block
   gdb vmlinux-2.6.32.431.23.3.el6_lustre

   In GDB on the guest: 

   Code Block

   (gdb) target remote localhost:1200 ## This is defined int he .gdbinit and it will list all the addresses of the loaded modules (gdb) mod-list-syms add-symbol-file lnet.ko 0xffffffffa03c8000 add-symbol-file sha512_generic.ko 0xffffffffa00a8000 add-symbol-file sha256_generic.ko 0xffffffffa0065000 add-symbol-file crc32c_intel.ko 0xffffffffa0023000 add-symbol-file libcfs.ko 0xffffffffa0356000 add-symbol-file fuse.ko 0xffffffffa0231000 add-symbol-file autofs4.ko 0xffffffffa034e000 add-symbol-file sunrpc.ko 0xffffffffa02f5000 add-symbol-file bnx2fc.ko 0xffffffffa02d7000 .... ## Add the symbols to GDB using the module with debugging symbols you copied to your host machine (gdb) add-symbol-file /path/to/lnet.ko 0xffffffffa03c8000 ## Add breakpoints as you wish ## continue execution (gdb) continue

   By this point the symbols for the module you wish to debug is loaded in GDB and GDB is attached to your guest OS.

10. Add strategic breakpoints to enhance your debugging
11. Invoke the scenario you wish to debug
12. GDB will break at the desired break points, allowing you to debug as you normally would a user space program from gdb.

Later Kernels

Later kernels come packed with a script vmlinux-gdb.py 

This can be used instead of the script above.

ashehata@sn300:/OS/5.4.0$ gdb vmlinux
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from vmlinux...
warning: File "/OS/5.4.0/vmlinux-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/ashehata/OS/5.4.0/vmlinux-gdb.py
line to your configuration file "/home/ashehata/.gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/ashehata/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) set auto-load safe-path /
(gdb) target remote localhost:1202
(gdb) source OS/5.4.0/vmlinux-gdb.py
# if you have the kernel modules in the local directory (which is what I do), you can do:
(gdb) lx-symbols 
scanning for modules in /home/ashehata/OS/5.4.0
loading @0xffffffffc04f6000: /OS/5.4.0/kvm-amd.ko
loading @0xffffffffc05cb000: /OS/5.4.0/ccp.ko
loading @0xffffffffc0421000: /OS/5.4.0/kvm.ko
loading @0xffffffffc0416000: /OS/5.4.0/crct10dif-pclmul.ko
loading @0xffffffffc0386000: /OS/5.4.0/ghash-clmulni-intel.ko
...

Once the symbols are loaded you can commence debugging.