Debugging Linux Guest VM With Cloud Hypervisor On Arm64
GDB support for AArch64 was enabled in Cloud Hypervisor upstream. In this article I will show how to debug a Linux VM with Cloud Hypervisor on a Arm64 server. At the end, there will be a summary of all the source code changes.
This article concludes my series “Cloud Hypervisor + GDB + Arm64”. To review other articles of the topic, please visit:
- Cloud Hypervisor + GDB + Arm64 Part 1: GDB Remote Serial Protocol
- Cloud Hypervisor + GDB + Arm64 Part 2: Using `gdbstub`
- Cloud Hypervisor + GDB + Arm64 Part 3: Gap analysis
- Cloud Hypervisor + GDB + Arm64 Part 4:AT Instruction
- Cloud Hypervisor + GDB + Arm64 Part 5: AArch64 Address Translation Sketch
- Cloud Hypervisor + GDB + Arm64 Part 6: Accessing System Registers
- Address Translation Related Arm-v8a Features
GDB
I don’t have any knowledge about how GDB versions evolve. To be safe, I updated my GDB to a very recent version - v12.1. All my tests were done on it.
Building Linux Kernel
The kernel configuration item CONFIG_RANDOMIZE_BASE should be disabled. This item is used for randomizing the virtual and physical base address of the kernel [2].
If this option keeps enabled, the VMM will have trouble in translating the guest virtual address (GVA) to the guest physical address (GPA).
Building Cloud Hypervisor
Now the GDB support in Cloud Hypervisor is not totally ready yet. The feature is not included in any formal release. You need to build Cloud Hypervisor from source code to use GDB, and make sure you are using commit c798b9 or later.
GDB feature is not covered by the default feature set. To enable GDB support, you need to build like:
cargo build --features gdbTesting
Start Cloud Hypervisor:
sudo cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock --kernel /path/to/kernel/arch/arm64/boot/Image <irrelevant parameters ignored> --gdb path=/tmp/ch-gdb-sockThe parameter --gdb path=/tmp/ch-gdb-sock tells Cloud Hypervisor that GDB is required and it should talk to the client via the socket specified by path .
Start GDB:
sudo gdb /path/to/kernel/vmlinux -ex "target remote /tmp/ch-gdb-sock"The command loads the binary to debug and connects to the socket which has been opened by Cloud Hypervisor.
Note about the binary that was loaded by GDB, it is vmlinx rather than the file loaded by Cloud Hypervisor arch/arm64/boot/Image. That is something special on Arm64: The kernel file to boot must be in the format of PE, but GDB wants ELF.
Then you should enter the GDB command line. It’s time to practice your debug skills.
Here comes a screenshot after I interrupted the kernel execution and printed the back-trace.
Things To Improve
Now there are still things to improve. Some functionalities of GDB are missing. And there are bugs to fix.
Here is a list of know issues:
- Hardware watchpoint is not supported
- Software breakpoint and watchpoint is not supported (but this may not be planed)
- If you set multiple breakpoints, after the first one is hit, others will all miss
- Only the general purposed registers of Arm64 are supported (
info registerscommand)
Records Of Progress
There was a ticket in the community requiring the GDB support on AArch64: 3980. To implement it, I created a couple of pull requests to Cloud Hypervisor and rust-vmm repositories:
- https://github.com/cloud-hypervisor/cloud-hypervisor/pull/4298
- https://github.com/cloud-hypervisor/cloud-hypervisor/pull/4384
- https://github.com/cloud-hypervisor/cloud-hypervisor/pull/4519
- https://github.com/cloud-hypervisor/cloud-hypervisor/pull/4534
- https://github.com/rust-vmm/kvm-ioctls/pull/202
At the same time, a Crosvm developer enabled AArch64 support in gdbstub_arch crate. There was a wonderful discussion about how the registers of Arm64 should be supported in the PR:
