Debugging Linux Kernel In Cloud Hypervisor With GDB

This short article records some tips for debugging Linux kernel in Cloud Hypervisor with gdb.

Cloud Hypervisor is an open source virtual machine monitor (VMM) software written in Rust language. It supports x86_64 and aarch64 architectures. For hypervisor, it supports kvm and Hyper-V.

For virtualization developers, it would be very practical if the VMM can work with gdb to debug the kernel. Now I will show how to do it with Cloud Hypervisor.

Kernel Configuration

This item need to be enabled in kernel configuration, it will add symbols used by the debugger in kernel binary:

CONFIG_DEBUG_INFO=y

Then build the kernel.

Build & Run

Make sure your Cloud Hypervisor source code is the release v23.0 or later.

Build the source code with gdb feature:

> cd cloud-hypervisor
> cargo build --features gdb

Run Cloud Hypervisor. Make sure you specified the --kernel parameter to the new built kernel binary vmlinux with the debug symbols. And use --gdb option to specify the socket path that the debugger can connect.

# start Cloud Hypervisor with `--gdb` parameter
> sudo ./target/debug/cloud-hypervisor --cpus boot=1 --memory size=512M --kernel <kernel folder>/vmlinux --cmdline "root=/dev/vda1 console=hvc0 rw systemd.journald.forward_to_console=1" --disk path=osdisk.img path=cloudinit --net tap=,mac=12:34:56:78:90:01,ip=192.168.1.1,mask=255.255.255.0 --serial off --console tty -vvv --log-file log.log --gdb path=/tmp/ch-gdb-sock

After issuing the command, Cloud Hypervisor will hold and wait for gdb to connect.

Start GDB

Run gdb from another terminal, load the vmlinux binary.

After entering gdb, connect the remote side via the socket that was opened by Cloud Hypervisor.

# run gdb
> sudo gdb <kernel folder>/vmlinux
(gdb) target remote /tmp/ch-gdb-sock
(gdb) c

Type c to continue to execute. You can find the virtual machine of the other terminal begin to boot.

If you type Ctrl + c, the guest kernel will be paused and you can use bt to check the kernel back trace.

AArch64 Gap

What I have shown was done on a x86_64 desktop. Now gdb is supported on x86_64 architecture only in Cloud Hypervisor. On Arm64 this has not been done.