PCI passthrough via OVMF/Troubleshooting - ArchWiki

Jump to content

# PCI passthrough via OVMF/Troubleshooting

From ArchWiki
< PCI passthrough via OVMF

If your issue is not mentioned below, you may want to browse QEMU#Troubleshooting.

## QEMU 4.0: Unable to load graphics drivers/BSOD/Graphics stutter after driver install using Q35

Starting with QEMU 4.0, the Q35 machine type changes the default `kernel_irqchip` from `off` to `split`, which breaks some guest devices, such as NVIDIA graphics (the driver fails to load / black screen / code 43 / graphics stutters, usually when mouse moving).

Switch to full KVM mode instead by adding `<ioapic driver='kvm'/>` under libvirt's `<features>` tag in your virtual machine configuration or by adding `kernel_irqchip=on` in the `-machine` QEMU arg.

## QEMU 5.0: host-passthrough with kernel version 5.5 to 5.8.1 when using Zen 2 processors: Windows 10 BSOD loop 'KERNEL SECURITY CHECK FAILURE'

Note As of kernel version 5.8.2, disabling STIBP is not required anymore.

Starting with QEMU 5.0 virtual machines running on Zen 2 and kernels newer than 5.4 will cause a BSOD loop of: 'KERNEL SECURITY CHECK FAILURE'. This can be fixed by either updating to kernel version 5.8.2 or higher, or disabling STIBP:

     <cpu mode='host-passthrough' ...>
       ...
       <feature policy='disable' name='amd-stibp'/>
       ...
     </cpu>

This requires libvirt 6.5 or higher. On older versions, several workarounds exist:

- Switch CPU mode from `host-passthrough` to `host-model`. This only works on libvirt 6.4 or lower.
- Manually patch qemu-desktop in order to revert this commit.
- On qemu commandline, add `amd-stibp=off` to the cpu flags string. This can also be invoked through libvirt via a `<qemu:commandline>` entry.

## "Error 43: Driver failed to load" with mobile (Optimus/max-q) NVIDIA GPUs

This error occurs because the NVIDIA driver wants to check the status of the power supply. If no battery is present, the driver does not work. Whether Libvirt or QEMU, by default none of them provide the possibility to simulate a battery. This might also result in a reduced screen resolution and the NVIDIA Desktop Manager refusing to load when right-clicking the desktop, saying it requires Windows 10, a compatible GPU and the NVIDIA graphics driver.

You can however create and add a custom acpi table file to the virtual machine which will do the work.

First you have to create the custom acpi table file by saving the following file as SSDT1.dat (base64 encoded here):

    echo 'U1NEVKEAAAAB9EJPQ0hTAEJYUENTU0RUAQAAAElOVEwYEBkgoA8AFVwuX1NCX1BDSTAGABBMBi5f
    U0JfUENJMFuCTwVCQVQwCF9ISUQMQdAMCghfVUlEABQJX1NUQQCkCh8UK19CSUYApBIjDQELcBcL
    cBcBC9A5C1gCCywBCjwKPA0ADQANTElPTgANABQSX0JTVACkEgoEAAALcBcL0Dk=' | base64 -d > SSDT1.dat

Next you must add the processed file to the main domain of the virtual machine:

    <domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
      ...
      <qemu:commandline>
        <qemu:arg value="-acpitable"/>
        <qemu:arg value="file=/path/to/your/SSDT1.dat"/>
      </qemu:commandline>
    </domain>

Make sure your XML file has the correct namespace in the `<domain>` tag as visible above, otherwise the XML verification will fail [1].

## "BAR 3: cannot reserve [mem]" error in dmesg after starting virtual machine

This article or section needs expansion.

Reason: This error is actually related to the boot_vgs issue and should be merged together with everything else concerning GPU ROMs. (Discuss in Talk:PCI passthrough via OVMF/Troubleshooting#UEFI (OVMF) Compatibility in VBIOS)

With respect to this article:

If you still have code 43 check dmesg(1) for memory reservation errors after starting up your virtual machine, if you have similar it could be the case:

    vfio-pci 0000:09:00.0: BAR 3: cannot reserve [mem 0xf0000000-0xf1ffffff 64bit pref]

Find out a PCI Bridge your graphics card is connected to. This will give actual hierarchy of devices:

    $ lspci -t

Before starting the virtual machine run the following lines, replacing IDs with actual values from previous output.

    # echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove
    # echo 1 > /sys/bus/pci/rescan

Note Probably setting kernel parameter `video=efifb:off` is required as well. [2]

In addition try adding kernel parameter `pci=realloc` which also helps with hotplugging issues.

## UEFI (OVMF) compatibility in VBIOS

This article or section is being considered for removal.

Reason: Flashing your guest GPU for the purpose of a GPU passthrough is never good advice. A full section should be dedicated to VBIOS compatibility. (Discuss in Talk:PCI passthrough via OVMF/Troubleshooting#UEFI (OVMF) Compatibility in VBIOS)

With respect to this article:

Error 43 can be caused by the GPU's VBIOS without UEFI support. To check whenever your VBIOS supports it, you will have to use rom-parser:

    $ git clone https://github.com/awilliam/rom-parser
    $ cd rom-parser && make

Dump the GPU VBIOS:

    # echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
    # cat /sys/bus/pci/devices/0000:01:00.0/rom > /tmp/image.rom
    # echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

And test it for compatibility:

    $ ./rom-parser /tmp/image.rom

    Valid ROM signature found @600h, PCIR offset 190h
    	PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1184, class: 030000
    	PCIR: revision 0, vendor revision: 1
    Valid ROM signature found @fa00h, PCIR offset 1ch
    	PCIR: type 3 (EFI), vendor: 10de, device: 1184, class: 030000
    	PCIR: revision 3, vendor revision: 0
    		EFI: Signature Valid, Subsystem: Boot, Machine: X64
    	Last image

To be UEFI compatible, you need a "type 3 (EFI)" in the result. If it is not there, try updating your GPU VBIOS. GPU manufacturers often share VBIOS upgrades on their support pages. A large database of known compatible and working VBIOSes (along with their UEFI compatibility status!) is available on TechPowerUp.

Updated VBIOS can be used in the virtual machine without flashing. To load it in QEMU:

    -device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \

And in libvirt:

    <hostdev>
         ...
         <rom file='/path/to/your/gpu/bios.bin'/>
         ...
       </hostdev>

One should compare VBIOS versions between host and guest systems using nvflash (Linux versions under Show more versions) or GPU-Z (in Windows guest). To check the currently loaded VBIOS:

    $ ./nvflash --version

    ...
    Version               : 80.04.XX.00.97
    ...
    UEFI Support          : No
    UEFI Version          : N/A
    UEFI Variant Id       : N/A ( Unknown )
    UEFI Signer(s)        : Unsigned
    ...

And to check a given VBIOS file:

    $ ./nvflash --version NV299MH.rom

    ...
    Version               : 80.04.XX.00.95
    ...
    UEFI Support          : Yes
    UEFI Version          : 0x10022 (Jul  2 2013 @ 16377903 )
    UEFI Variant Id       : 0x0000000000000004 ( GK1xx )
    UEFI Signer(s)        : Microsoft Corporation UEFI CA 2011
    ...

If the external ROM did not work as it should in the guest, you will have to flash the newer VBIOS image to the GPU. In some cases it is possible to create your own VBIOS image with UEFI support using GOPUpd tool, however this is risky and may result in GPU brick.

Warning Failure during flashing may "brick" your GPU - recovery may be possible, but rarely easy and often requires additional hardware. DO NOT flash VBIOS images for other GPU models (different boards may use different VBIOSes, clocks, fan configuration). If it breaks, you get to keep all the pieces.

In order to avoid the irreparable damage to your graphics adapter it is necessary to unload the NVIDIA kernel driver first:

    # modprobe -r nvidia_modeset nvidia

Flashing the VBIOS can be done with:

    # ./nvflash romfile.bin

Warning DO NOT interrupt the flashing process, even if it looks like it is stuck. Flashing should take about a minute on most GPUs, but may take longer.

## Slowed down audio pumped through HDMI on the video card

For some users, the virtual machine's audio slows down/starts stuttering/becomes demonic after a while when it is pumped through HDMI on the video card. This usually also slows down graphics. A possible solution consists of enabling MSI (Message Signal-Based Interrupts) instead of the default (Line-Based Interrupts).

In order to check whether MSI is supported or enabled, run the following command as root:

    # lspci -vs device | grep 'MSI:'

where `device` is the GPU's PCI address (e.g. `01:00.0`).

The output should be similar to:

    Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+

A `-` after `Enable` means MSI is supported, but not used by the virtual machine, while a `+` says that the virtual machine is using it.

The procedure to enable it is quite complex, instructions and an overview of the setting can be found here.

On a Linux guest you can use modinfo(8) to see if there is option to enable MSI, for example:

    $ modinfo snd_hda_intel | grep msi

If there is, one can enable it by adding the relevant option to a custom modprobe.d file:

    /etc/modprobe.d/snd-hda-intel.conf

    options snd-hda-intel enable_msi=1

Other hints can be found on the lime-technology's wiki[dead link 2023-05-06—HTTP 404], or in this article.

A UI tool called MSI Utility (FOSS Version 2) works with Windows 10 64-bit and simplifies the process.

In order to fix the issues, enabling MSI on the `.0` function of a NVIDIA card

    01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1) (prog-if 00 [VGA controller])

is not enough; it will also be required to enable it on the other function

    01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)

to fix the issue.

## No HDMI audio output on host when intel_iommu is enabled

If after enabling `intel_iommu` the HDMI output device of Intel GPU becomes unusable on the host, setting the kernel parameters `intel_iommu=on,igfx_off` might bring the audio back. Please read iommu.html for details about setting `igfx_off`.

## X does not start after enabling vfio_pci

This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from `lspci -n` or the Xorg log [3]. Note that the value from the `lspci` output is hexadecimal and should be converted to decimal in the `.conf` file.

    /etc/X11/xorg.conf.d/10-intel.conf

    Section "Device"
            Identifier "Intel GPU"
            Driver "modesetting"
            BusID  "PCI:0:2:0"
    EndSection

## Chromium ignores integrated graphics for acceleration

Chromium and friends will try to detect as many GPUs as they can in the system and pick which one is preferred (usually discrete NVIDIA/AMD graphics).

It tries to pick a GPU by looking at PCI devices, not OpenGL renderers available in the system. The result is that Chromium may ignore the integrated GPU available for rendering and try to use the dedicated GPU bound to the `vfio-pci` driver (and unusable on the host system), regardless of whenever a guest virtual machine is running or not.

This results in software rendering being used (leading to higher CPU load, which may also result in choppy video playback, scrolling and general lack of smoothness).

This can be fixed by explicitly telling Chromium which GPU you want to use.

## Virtual machine only uses one core

For some users, even if IOMMU is enabled and the core count is set to more than `1`, the virtual machine still only uses one CPU core and thread. To solve this enable "Manually set CPU topology" in virt-manager and set it to the desirable amount of CPU sockets, cores and threads. Keep in mind that "Threads" refers to the thread count per CPU core, not the total count.

## Passthrough seems to work but no output is displayed

If you are using virt-manager, ensure that UEFI firmware is selected for your virtual machine. Also, make sure you have passed the correct device to the virtual machine.

## Host lockup after virtual machine shutdown

This issue seems to primarily affect users running a Windows 10 guest and usually after the virtual machine has been run for a prolonged period of time: the host will experience multiple CPU core lockups (see [4]).

To fix this try enabling Message Signal-Based Interrupts (MSI) on the GPU passed through to the guest. A good guide for how to do this can be found in [5]. You can also download this application for Windows here [6] that should make the process easier.

## Host lockup if guest is left running during sleep

VFIO-enabled virtual machines tend to become unstable if left running through a sleep/wakeup cycle and have been known to cause the host machine to lockup when an attempt is then made to shut them down.

In order to avoid this, one can simply prevent the host from going into sleep while the guest is running using the following libvirt hook script and systemd unit. The hook file needs executable permissions to work.

    /etc/libvirt/hooks/qemu

    #!/bin/sh

    OBJECT="$1"
    OPERATION="$2"

    case "$OPERATION" in
        prepare)
            systemctl start libvirt-nosleep@"$OBJECT"
            ;;
        release)
            systemctl stop libvirt-nosleep@"$OBJECT"
            ;;
        *)
            exit 0
            ;;
    esac

    /etc/systemd/system/libvirt-nosleep@.service

    [Unit]
    Description=Preventing sleep while libvirt domain "%i" is running

    [Service]
    Type=simple
    ExecStart=/usr/bin/systemd-inhibit --what=sleep --why="Libvirt domain \"%i\" is running" --who=%U --mode=block sleep infinity

## Bluescreen at boot since Windows 10 1803

Since Windows 10 1803, there is a problem when you are using "host-passthrough" as cpu model. The machine may be unable to boot and it either starts boot looping or you may get a BSOD. You can workaround this by using the following kernel parameter:

    # echo 1 > /sys/module/kvm/parameters/ignore_msrs

To make it permanently you can create a modprobe.d file `kvm.conf`:

    /etc/modprobe.d/kvm.conf

    options kvm ignore_msrs=1

To prevent clogging up dmesg with "ignored rdmsr" messages you can additionally add:

    options kvm report_ignored_msrs=0

## AMD Ryzen / BIOS updates (AGESA) yields "Error: internal error: Unknown PCI header type ‘127’"

AMD users have been experiencing breakage of their KVM setups after updating the BIOS on their motherboard. There is a kernel patch[dead link 2024-07-30—HTTP 404], (see Kernel/Arch build system for instruction on compiling kernels with custom patches) that can resolve the issue as of now (28th July 2019), but this is not the first time AMD has made an error of this very nature, so take this into account if you are considering updating your BIOS in the future as a VFIO user.

## AMD GPU not resetting properly yielding "Error: internal error: Unknown PCI header type ‘127’" (Separate issue from the one above)

Passing through an AMD GPU may result into a problem known as the "AMD reset bug". Upon power cycling the guest, the GPU does not properly reset its state, which causes the device to malfunction until the host is also rebooted. This is usually paired with a `code 43` driver error in a Windows guest, and the message `Error: internal error: Unknown PCI header type '127'` in the libvirt log on the host.

In the past, this meant having to use workarounds to manually reset the GPU, or resorting to the use of kernel patches that were unlikely to land in upstream. Currently, the recommended solution that does not require patching of the kernel is to install vendor-reset-gitAUR or vendor-reset-dkms-gitAUR[broken link: package not found] and making sure the `vendor-reset` kernel module is loaded before booting the guest.

For convenience, you can load the module automatically, either at boot time or with a libvirt hook:

    /etc/libvirt/hooks/qemu

    #!/bin/sh

    OPERATION="$2"

    case "$OPERATION" in
        prepare)
            modprobe -s vendor-reset
            ;;
        release)
            modprobe -sr vendor-reset
            ;;
        *)
            exit 0
            ;;
    esac

Make the file executable, if needed.

Note Make sure you do not have any of the AMD reset bug kernel patches installed if you are using vendor-reset-gitAUR or vendor-reset-dkms-gitAUR[broken link: package not found].

## Host crashes when hotplugging NVIDIA card with USB

If attempting to hotplug an NVIDIA card with a USB port, you may have to blacklist the `i2c_nvidia_gpu` driver. You can achieve this by creating a modprobe.d file:

    /etc/modprobe.d/blacklist.conf

    blacklist i2c_nvidia_gpu

## Host unable to boot and stuck in black screen after enabling vfio

If debug kernel messages during boot are enabled by adding with the `debug ignore_loglevel` kernel parameters, you may see boot stuck with the last message similar to:

    vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

This can be mitigated by disconnecting the passed-through GPU from your monitor. You may reconnect the passed-through GPU to a monitor after the host has booted.

If you do not want to plug the cable in each time you boot the host, you can disable the framebuffer in your boot loader to bypass this message. For UEFI systems you can add `video=efifb:off` as a kernel parameter. For legacy support, use `video=vesafb:off` instead or in conjunction. Note that doing this may cause issues with Xorg.

If you encounter problems with Xorg, the following solution may help (remember to substitute with your own values):

    /etc/X11/xorg.conf.d/10-amd.conf

    Section "Device"
            Identifier "AMD GPU"
            Driver "amdgpu"
            BusID  "PCI:0:2:0"
    EndSection

## AER errors when passing through PCIe USB hub

In some cases, passing through a PCIe USB hub, such as one connected to the guest GPU, might fail with AER errors similar to the following:

    kernel: pcieport 0000:00:01.1: AER: Uncorrected (Non-Fatal) error received: 0000:00:01.1
    kernel: pcieport 0000:00:01.1: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
    kernel: pcieport 0000:00:01.1: AER:   device [8086:1905] error status/mask=00100000/00000000
    kernel: pcieport 0000:00:01.1: AER:    [20] UnsupReq               (First)
    kernel: pcieport 0000:00:01.1: AER:   TLP Header: 00000000 00000000 00000000 00000000
    kernel: pcieport 0000:00:01.1: AER: device recovery successful

## Reserved Memory Region Reporting (RMRR) Conflict

If you run into an issue passing through a device because of the BIOS's usage of RMRR, like the error below:

    vfio-pci 0000:01:00.1: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

You can try the patches here: [7]

## Too-low frequency limit for AMD GPU passed-through to virtual machine

On some machines with AMD GPUs, binding the devices to `vfio-pci` may be insufficient to prevent interference from the host, since the amdgpu driver on the host may query global ATIF methods which can alter the behavior of the GPU.

For example, a user with a Dell Precision 7540 laptop containing a Radon Pro WX 3200 AMD GPU reported that, with the AMD GPU bound to `vfio-pci`, the passed-through AMD GPU was limited to `501 MHz` instead of the correct `1295 MHz` limit. Blacklisting the amdgpu kernel module using the kernel command line was a workaround.

See this kernel mailing list discussion for further details.

## Host clocksource is HPET rather than TSC

If the clocksource of your host machine is HPET, you may see severely crippled performance in games. While this issue also occurs when running games outside of a virtual machine, unaware users will likely begin troubleshooting their virtual machine when the issue is actually with their host system.

To see your current clocksource, run this command on your host machine:

    $ cat /sys/devices/system/clocksource/clocksource*/current_clocksource

If your clocksource is HPET, you can try to force it to TSC by setting the kernel parameters `clocksource=tsc` and `tsc=reliable`. See this Reddit thread for more information.

## Code 43 while Resizable Bar is turned on in the bios

If you own a GPU that supports Resizable BAR / SAM and have the corresponding BIOS option enabled, you might get code 43 because this feature is disabled in QEMU [8]. Linux Kernel release 6.1 added option to manipulate PCIe Resizable BARs through sysfs, so you can try permanently resizing it using a udev rule:

    /etc/udev/rules.d/01-amd.rules

    ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1002", ATTR{device}=="0x73bf", ATTR{resource0_resize}="14"
    ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x1002", ATTR{device}=="0x73bf", ATTR{resource2_resize}="8"

Alternatively, to temporarily resize the PCIe BAR, you can write the new size to `/sys/bus/pci/devices/device_address/resourcenumber_resize`, where device_address is the address of your GPU, such as `0000:03:00.0` (note that for changing this parameter, no driver can be loaded for the device).

You can confirm values using `lspci -vvvxxxx`.

AMD graphics processors may require a specific value to be set for BAR 2, which is 8 MB, even though the maximum supported value is 256 MB. However, this requirement may depend on several factors, such as the QEMU version on your system. If you have already exhausted other options to prevent error code 43, set the value to `3` in the `resource2_resize` file, which will force BAR 2 to be set to 8 MB.

## Operating system hangs

If your virtualized OS (particularly Windows) hangs when the GPU is attached, you can also try opening your motherboard's settings and:

- disabling `Re-Size BAR Support`,
- disabling `Above 4G memory / Crypto Currency mining`,
- enabling `SR-IOV`,
- changing `Max TOLUD` from `dynamic` to a specific value,
- switching your `Initiate Graphic Adapter` from `PEG` to `IGD`.

At the very least, VEGA cards seem to be picky about those settings and won't actually work unless you change them.

## GPU timeouts when running games in the virtual machine

Starting with recent kernels (6.12+ for AMD, 5.x for Intel), split lock detection is enabled by default. This causes a forced 10ms pause for the process creating the split lock, which can result in GPU timeouts and similar virtual machine hanging. This behavior can be disabled by running:

    # sysctl -w kernel.split_lock_mitigate=0

This can be applied automatically with a sysctl configuration file:

    /etc/sysctl.d/split-lock.conf

    kernel.split_lock_mitigate = 0

## vfio: DMA mapping failed, unable to continue

When starting the virtual machine, an error like `qemu: hardware error: vfio: DMA mapping failed, unable to continue` may occur and prevent the VM from booting. To fix this, add the following to the XML file: [9]

    <cpu>
      <maxphysaddr mode="passthrough" limit="40"/>
    </cpu>

If directly using QEMU, the example above would be `-cpu host,host-phys-bits-limit=0x28`.

If the VM still doesn't work, try decreasing the value (like 39 and 0x27, respectively).

You can use lscpu(1) to see the actual address size of your CPU and use it rather than the values shown above. Unless you need a large address space, it's safer to choose the physical address size intead of the virtual size.

## vIOMMU: Failed to set vIOMMU: aw-bits X > host aw-bits Y

When starting the virtual machine, an error like the following may occur:

    libvirt.libvirtError: internal error: QEMU unexpectedly closed the monitor (vm='win11'): 2025-08-30T21:38:26.333149Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:03:00.0: Failed to set vIOMMU: aw-bits 48 > host aw-bits 39

Add `caching_mode="on" aw_bits="number_of_bits"` to the following section in your XML. Usually on x86 the default is either 39 or 48 bits [10], but you can use your own CPU's address size instead.

     <devices>
      ... other entries ...
      <iommu model="intel">
        <driver intremap="on" caching_mode="on" eim="on" aw_bits="39"/>
      </iommu>
     </devices>

For more information:

- Proxmox forum thread discussing this
- libvirt options

## The VM stucks during boot when using Looking Glass

Sometimes, when using Looking Glass, the VM may get stuck indefinitely at the boot process [11]. This might be caused by divergences between P-Core and E-Core specs on new processors with hybrid architectures.

To solve this, follow these instructions. It's better to pick the physical address size instead of the virtual address size [12].

Hidden categories:

- Pages or sections flagged with Template:Expansion
- Sections flagged with Template:Remove
- Pages with dead links
- Pages with broken package links

PCI passthrough via OVMF/Troubleshooting
Add topic