PCI workaround

Overview

In OpenStack Ocata, VMs of QEMU machine type q35 only have a single spare PCI port. This allows to hot-plug a single PCI device only. Typical devices added to a VM are an additional volume or an additional network interface. Adding more than one such device after creating the VM, results in an error message. This page describes two work arounds for the problem.

Solution 1: Change machine type

The image property hw_machine_type=pc defines the QEMU machine type of the VMs created from this image. If no machine type is set, then the q35 machine type is used. This is the prefered machine type because it can take advantage of modern virtualization technologies. It does, however, present the problem described at the top of this page.

A possible work around is to instruct OpenStack to create VMs of the machine type pc-i440fx. This is a legacy type which lacks some features and performance when compare to the more modern q35. However, it allows to hot-plug up to 31 PCI devices after VM creation.

If this solution is chose, we recommend creating a separate VM image and only building select VMs from this type while defaulting to an image which uses q35.

The property can be set during the initial upload of the image. Here is an example:

openstack image create \
    --disk-format raw \
    --min-ram 1 \
    --file $(realpath ./scratch/CentOS-7-x86_64-GenericCloud-1901.raw) \
    --protected \
    --public \
    --project openstack  \
    --project-domain default \
    --property hw_qemu_guest_agent=yes \
    --property os_require_quiesce=yes \
    --property os_type=linux \
    --property os_distro=centos \
    --property hw_machine_type=pc \
    "CentOS 7.6 (1901) i440fx"

After creating the image, we can query the properties:

openstack image show \
    --column properties \
    --format value \
    'a416afda-3024-4404-aec3-193c210eb954'

hw_machine_type='pc', hw_qemu_guest_agent='yes', os_distro='centos', os_require_quiesce='yes', os_type='linux'

Note: It is possible to rebuild an existing VM on such an image to convert it's machine type. However, depending on the Operating System in question, this might well prevent the VM from booting because of the changes PCI layout.

Solution 2: Rebuild the VM

On initial creation, a VM of the default machine type q35 has one spare PCI slot. When adding a device to the VM after it's initial creation, this slot gets used up. Howerver, if we rebuild the VM after adding a device, a new spare PCI slot is added. This allows us to add yet another device. This process can be repeated many times.

To rebuild a VM, run a command similar to the below example. Note: rebuilding a VM will also restart the VM.

image_id=b72d596f-12af-4ee3-9902-a2afbdfbf4c1
vm_id=e3eeb7c2-0bf1-492d-a6d5-8be3b3d30ea1

openstack server rebuild \
    --image ${image_id} \
    ${vm_id}

After this, the VM will have exactly one spare PCI slot.