Using vfio-user in Cloud Hypervisor

This article introduces how to test vfio-user NVMe device in Cloud Hypervisor (a virtual machine monitor written in Rust language).

vfio-user

vfio-user is a framework for emulating PCI devices in user space.

vfio-user protocol is experimental, you can find the document here: https://github.com/nutanix/libvfio-user/blob/master/docs/vfio-user.rst

vfio vs vfio-user

vfio is a framework that allows a physical device to be securely passed through to a user space process. In the use case of virtualization, the device is passed through to the virtual machine monitor (VMM) process, and the device is further assigned to a VM. The VMM call the API of vfio in the form of ioctl.

vfio-user implements a similar device assignment mechanism. It reuses the core VFIO concepts defined in its API. But the difference is:

• The device is emulated by another user space process
• API is not carried via ioctl, but via UNIX socket

SPDK

From release v21.01, SPDK began to support vfio-user. A new custom transport SPDK_NVME_TRANSPORT_VFIOUSER was added to enable NVMe driver running with NVMe over vfio-user target.

Testing vfio-user With Cloud Hypervisor

In this test, I will build SPDK from source code, use it to emulate a NVMe device and access the device from Cloud Hypervisor. Additional software are required for building SPDK and Cloud Hypervisor. To keep my host environment clean, I chose to start a container and do all the test in it.

Build Container

This step is optional. You can also use the public container ghcr.io/cloud-hypervisor/cloud-hypervisor:20230316–0, that’s the same thing.

Cloud Hypervisor provide a Dockerfile for building a container to run the integration test. The Dockerfile contains most essential software components for testing Cloud Hypervisor. You can use it as the starting point if you want to build your own container:

> cd /path/to/cloud-hypervisor
> docker build -t cloud-hypervisor:debug -f resources/Dockerfile \
        --build-arg arm64 resources

Start Container

Use following instruction to enter the container:

> docker run -it --workdir / --rm --privileged \
      --security-opt seccomp=unconfined \
      --ipc=host --net=bridge --mount type=tmpfs,destination=/tmp \
      --volume /dev:/dev \
      --volume /path/to/cloud-hypervisor:/cloud-hypervisor \
      --env USER=root --env CH_LIBC=gnu cloud-hypervisor:debug bash

Build & Start SPDK

In the container bash, use following instructions to build SPDK and start the vfio-user server:

# start_spdk.sh: Build SPDK and start vfio-user server
echo "Configuring hugepage"
HUGEPAGESIZE=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
PAGE_NUM=`echo $((12288 * 1024 / $HUGEPAGESIZE))`
echo $PAGE_NUM | tee /proc/sys/vm/nr_hugepages
chmod a+rwX /dev/hugepages || exit 1
echo "/dev/hugepages: "
echo `ls /dev/hugepages`

echo "Building SPDK"
cd /
git clone https://github.com/spdk/spdk.git
cd spdk/
# The following commit was verified in Cloud Hypervisor CI.
# AT the time of writing, latest main branch also works.
git reset --hard 6301f8915de32baed10dba1eebed556a6749211a
git submodule update --init
apt update
./scripts/pkgdep.sh
./configure --with-vfio-user
chmod +x /usr/local/lib/python3.8/dist-packages/ninja/data/bin/ninja
make -j `nproc` || exit 1

echo "Creating user space disk file"
cd /
mkdir images
truncate /images/test-disk.raw -s 128M
mkfs.ext4 /images/test-disk.raw

echo "Starting SPDK"
./spdk//build/bin/nvmf_tgt -i 0 -m 0x1 &
./spdk/scripts/rpc.py nvmf_create_transport -t VFIOUSER
./spdk/scripts/rpc.py bdev_aio_create /images/test-disk.raw test 512
./spdk/scripts/rpc.py nvmf_create_subsystem nqn.2019-07.io.spdk:cnode -a -s test
./spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2019-07.io.spdk:cnode test
mkdir /tmp/nvme-vfio-user
./spdk/scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode -t VFIOUSER -a /tmp/nvme-vfio-user -s 0

Build & Start Cloud Hypervisor

In the same container bash, build and start Cloud Hypervisor, the vfio-user client:

> cd /cloud-hypervisor
> cargo build
> ./target/debug/cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock \
    --cpus boot=1 --memory size=512M,shared=on \
    --kernel /root/workloads/CLOUDHV_EFI.fd --serial tty --console off \
    --user-device socket=/tmp/nvme-vfio-user/cntrl,id=vfio_user0 \
    --disk path=osdisk.img --disk path=ubuntu-cloudinit.img \
    --net tap=,mac=12:34:56:78:90:01,ip=192.168.1.1,mask=255.255.255.0 -v

Some files were used in the command, but have not been mentioned so far:

• CLOUDHV_EFI.fd: The EDK2 firmware binary file. See this document for how to build it.
• osdisk.img: The Ubuntu disk image. See this document for creating.
• ubuntu-cloudinit.img: The cloud-init disk. Use this tool to generate.

Once the VM boots successfully, you can find the NVMe device (nvme0n1) and use it in the VM:

# Check PCI devices
> cloud@cloud:~$ lspci
00:00.0 Host bridge: Intel Corporation Device 0d57
00:01.0 Mass storage controller: Red Hat, Inc. Virtio block device (rev 01)
00:02.0 Mass storage controller: Red Hat, Inc. Virtio block device (rev 01)
00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
00:04.0 Unassigned class [ffff]: Red Hat, Inc. Virtio RNG (rev 01)
00:05.0 Non-Volatile memory controller: Device 4e58:0001

# Check block devices
> cloud@cloud:~$ lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda     252:0    0  2.2G  0 disk
├─vda1  252:1    0  2.1G  0 part /
└─vda15 252:15   0   99M  0 part /boot/efi
vdb     252:16   0    8M  0 disk
nvme0n1 259:1    0  128M  0 disk

Reference

• https://spdk.io/news/2021/05/04/vfio-user/
• https://github.com/tmakatos/qemu/blob/master/docs/devel/vfio-user.rst
• https://github.com/nutanix/libvfio-user
• https://github.com/spdk/spdk/releases/tag/v21.01
• https://www.youtube.com/watch?v=NBT8rImx3VE
• https://www.youtube.com/watch?v=vz2kbCrUZf0