Learning Virtualization with kvmtool

Introduction

kvmtool is a lightweight virtual machine monitor (VMM) written in C language. As the name suggests, it supports only kvm hypervisor. For now it has been enabled on 3 architectures: X86_64, AArch64 and Risc-V.

kvmtool is very simple. It hasn’t a rich set of features like QEMU does. It has a quite small code base. So it is a good learning tool for virtualization beginners. Researching kvmtool makes it easy for you to understand every steps of making a virtual machine (VM).

Here is some words quoted from the original announcement email of kvmtool:

The goal of this tool is to provide a clean, from-scratch, lightweight KVM host tool implementation that can boot Linux guest images (just a hobby, won't be big and professional like QEMU) with no BIOS dependencies and with only the minimal amount of legacy device emulation.

Believe me, by looking into the source code of kvmtool, you can obtain the knowledge in following areas of virtualization:

• How the VM memory is organized
• How the virtual CPU’s are created
• How interrupts are generated in VMM
• How legacy devices (like the serial port device) are simulated
• How virtio devices are implemented
• How iommu & vfio works
• …

But you also need to know the limitation of kvmtool even from the view of learning: kvm itself is a type-2 hypervisor, all the important things of the hypervisor itself is in Linux kernel. kvmtool is only the tool that invokes the APIs. If you want to look deeply into the implementation of kvm, you need to research the kernel code.

Usage

Clone source code and build:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git
cd kvmtool 
make

kvmtool does not support UEFI & ACPI. On AArch64 architecture, specifying a direct kernel binary is the only way to boot. For the ease of debugging, I suggest you build a kernel image from source code.

There is one thing to notice: you cannot use any module in your kernel, because the modules cannot be installed into the file system. If you want to enable an option (CONFIG_...)in your kernel configuration (.config), you need to set it y instead of m. Such a configuration file was prepared in Cloud Hypervisor repo, which is used in the project integration test. You can use it to build with kernel version 5.15.12.

Regarding the root filesystem, there are many choices:

1. http://wiki.qemu.org/download/linux-0.2.img.bz2: This is the one recommended in kvmtool document.
2. https://s3.amazonaws.com/spec.ccfc.min/img/aarch64/ubuntu_with_ssh/fsfiles/xenial.rootfs.ext4: This is the rootfs used in Firecracker project. The image is very small, because it’s created for their micro VMs. I recommended this one.
3. Ubuntu cloud image is also a good choice. But there is some work to do before you can login the VM.
4. Or build your own rootfs.

Now it’s time to start the VM:

sudo ./lkvm run --disk ./rootfs.img --kernel <KERNEL_SRC_FOLDER>/arch/arm64/boot/Image

If everything goes well, you should be able to see the login message from the VM. If you are using the Firecracker rootfs, try root/root as username/passwd to access.

Some Options

In this section I would like to share some tips for virtualization developers, especially those working on arm64 architecture:

• The default virtio transport option is mmio. If you want PCI, specify the option --force-pci.
• On arm64, there are different types of Generic Interrupt Controller (GIC) to choose. If you have a particular requirement, apply this option: --irqchip <[gicv2|gicv2m|gicv3|gicv3-its]>.
• If you need to look into the FDT, use --dump-dtb to dump it and decode.
• Use -p option to specify command line arguments to kernel.

See the help message for all the options.

Tracing

When a VM is running, usually the command line is occupied by the VM’s console. If the VMM’s journal is also printed to the command line, it is very confusing for the user. The log of VMM must go into another way.

I made a small modification to the printing system of kvmtool:

1. I wrote a simple program, named reporter , to listen to a Unix socket. When there is any data come from the socket, reporter prints it.
2. In the main function, kvmtool opens the same socket. All the debugging messages will be sent to the socket.

The source code is on a branch of my github kvmtool repo. You can use it this way after re-building the source code:

# Open a terminal for monitoring the trace
./reportor
# Open another terminal for running VM
# Make sure to append the `--debug` option
sudo ./lkvm run --disk ./rootfs.img --kernel Image --debug

The output of reporter would be like:

The line of +++ marks the starting of a VM, --- the end of the VM.

Hopefully this tool bring some convenience to your debugging.