From patchwork Mon May 13 20:55:34 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Knut Omang X-Patchwork-Id: 243537 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 0210C2C009E for ; Tue, 14 May 2013 06:56:21 +1000 (EST) Received: from localhost ([::1]:37865 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ubzn9-0000ty-99 for incoming@patchwork.ozlabs.org; Mon, 13 May 2013 16:56:19 -0400 Received: from eggs.gnu.org ([208.118.235.92]:50260) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ubzmj-0000kX-QA for qemu-devel@nongnu.org; Mon, 13 May 2013 16:55:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Ubzmh-000076-UZ for qemu-devel@nongnu.org; Mon, 13 May 2013 16:55:53 -0400 Received: from mail-out5.uio.no ([129.240.10.17]:48920) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Ubzmh-0008VR-Aj for qemu-devel@nongnu.org; Mon, 13 May 2013 16:55:51 -0400 Received: from mail-mx4.uio.no ([129.240.10.45]) by mail-out5.uio.no with esmtp (Exim 4.80.1) (envelope-from ) id 1UbzmS-0004Rk-0a; Mon, 13 May 2013 22:55:36 +0200 Received: from 146.80-202-81.nextgentel.com ([80.202.81.146] helo=[172.25.2.6]) by mail-mx4.uio.no with esmtpsa (SSLv3:DHE-RSA-CAMELLIA256-SHA:256) user knuto (Exim 4.80) (envelope-from ) id 1UbzmR-0001VW-81; Mon, 13 May 2013 22:55:35 +0200 Message-ID: <1368478534.14279.201.camel@ori.omang.mine.nu> From: Knut Omang To: Alex Williamson Date: Mon, 13 May 2013 22:55:34 +0200 In-Reply-To: <1367621792.22436.49.camel@ul30vt.home> References: <1367621792.22436.49.camel@ul30vt.home> X-Mailer: Evolution 3.6.4 (3.6.4-3.fc18) Mime-Version: 1.0 X-UiO-SPF-Received: X-UiO-Ratelimit-Test: rcpts/h 3 msgs/h 1 sum rcpts/h 3 sum msgs/h 1 total rcpts 323 max rcpts/h 13 ratelimit 0 X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, TVD_RCVD_IP=0.001, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO) X-UiO-Scanned: 11756602DA772F31DAD74CCF46A32667984AF647 X-UiO-SPAM-Test: remote_host: 80.202.81.146 spam_score: -49 maxlevel 80 minaction 2 bait 0 mail/h: 1 total 3 max/h 1 blacklist 0 greylist 0 ratelimit 0 X-UiOonly: 0DE359661752AD8287887D687A66E405C9E11594 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 129.240.10.17 Cc: qemu-devel , kvm Subject: Re: [Qemu-devel] VFIO VGA test branches X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Hi all, Perfect timing from my perspective, thanks Alex! I spent the better part of the weekend testing your branches on a new system I just put together for this purpose, results below.. On Fri, 2013-05-03 at 16:56 -0600, Alex Williamson wrote: ... > git://github.com/awilliam/linux-vfio.git vfio-vga-reset > git://github.com/awilliam/qemu-vfio.git vfio-vga-reset System setup: - Fedora 18 on - Gigabyte Z77X-UD5H motherboard - Intel Core i7 3770 (Ivy bridge w/integrated graphics) - 2 discrete graphics cards: lspci | egrep 'VGA|Audio' 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Caicos [Radeon HD 6450] 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Caicos HDMI Audio [Radeon HD 6400 Series] 02:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde PRO [Radeon HD 7700 Series] 02:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] Short summary: - Once I got past a few time consuming obstacles explained below - the graphics part of the graphics/hdmi audio passthrough seems to work perfect on both discrete graphics cards (though so far only one at at time and with some minor issues, see below) - no success with the hdmi audio yet (ideas for further investigation appreciated!) - Contrary to deniv@lavabit.com I had no success with using pci-assign for VGA with a standard fedora 18 kernel and fairly recent qemu, nor with your branches, Details: - I started off with the required kernel parameter 'intel_iommu=on' + necessary parameters for disabling radeon (radeon.modeset=0 rd.driver.blacklist=radeon) using the integrated graphics as primary display - this caused the system to freeze (with color artifacts on the console) - In my naivity and because of the "i" in ifgx I tried both with 'intel_iommu=ifgx_off' and then 'intel_iommu=on,igfx_off' and a full set of combinations of vfio, cards, kernels and pci-assign before I suspected that iommu support was turned off for **all** graphics cards with igfx_off - The solution was to have integrated graphics turned off in the BIOS, and 'intel_iommu=on': - iommu groups: ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices total 0 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0 lrwxrwxrwx 1 root root 0 May 11 08:55 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1 - eg. both the VGA/HDMI Audio pairs + the two root ports they are plugged into are in the same group: # lspci -n ... 01:00.0 0300: 1002:683f 01:00.1 0403: 1002:aab0 02:00.0 0300: 1002:6779 02:00.1 0403: 1002:aa98 ... modprobe vfio_pci echo 0000:01:00.1 > /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind echo 0000:02:00.1 > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind echo 1002 683f > /sys/bus/pci/drivers/vfio-pci/new_id echo 1002 aab0 > /sys/bus/pci/drivers/vfio-pci/new_id echo 1002 6779 > /sys/bus/pci/drivers/vfio-pci/new_id echo 1002 aa98 > /sys/bus/pci/drivers/vfio-pci/new_id # lsusb ... Bus 001 Device 008: ID 046d:c315 Logitech, Inc. Classic New Touch Keyboard Bus 001 Device 004: ID 046d:c05b Logitech, Inc. M-U0004 810-001317 [B110 Optical USB Mouse] ... - I also applied your suggested patch to the quirk function in VFIO (see below) - Here is a (trimmed for readability) command line I successfully used to boot from the Windows 7 install DVD, notice the cd and disk device descriptions and the bus parameter - I struggled a while with that until I came across a comment by Gerd Hoffmann here: https://bugzilla.redhat.com/show_bug.cgi?id=922670 (Thanks, Gerd!) qemu-kvm -M q35 \ -nodefconfig -readconfig $SRC/qemu/docs/q35-chipset.cfg \ -device vfio-pci,host=2:00.0,x-vga=on,multifunction=on,bus=ich9-pcie-port-1,addr=0.0 \ -device vfio-pci,host=2:00.1,bus=ich9-pcie-port-1,addr=0.1 \ -L $SRC/seabios/out/ -L $SRC/qemu/pc-bios \ -vga none -nographic -cpu host -rtc base=localtime -k no -m 8192 -smp 2 \ -drive file=/dev/sr0,index=2,media=cdrom,id=cd \ -drive file=ivm03.img,index=0,media=disk,id=ivm03 \ -device ide-drive,drive=ivm03,bus=ide.0 \ -device ide-cd,drive=cd,bus=ide.1 \ -net nic,vlan=0,model=virtio -net tap,vlan=0 \ -enable-kvm \ -device usb-host,hostbus=1,hostaddr=8 \ -device usb-host,hostbus=1,hostaddr=4 - Both the graphics card seems to have a rom but only the HD6450 let itself to "scraping". Anyway, supplying it to vfio did not seem to make any difference. find /sys -name rom /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/rom /sys/devices/pci0000:00/0000:00:01.1/0000:02:00.0/rom ... Some observations and remaining unresolved issues: - VFIO patch: Initially (while still running with igfx_off) I observed exactly the same behaviour as deniv@lavabit.com reported a while ago: With vfio_pci debug enabled, vfio_pci ended up spinning with repeated calls to vfio_ati_3c3_quirk_read and repeated logs: vfio: vfio_vga_read(0x3c3, 1) = 0x0 I patched up accordingly with This of course did not help much until I actually got the iommu enabled for the radeons (similar "repeated patters" as deniv reported) but what I have observed after I got it working is that if I disable the patch above, things are not that well: the Fedora VM comes up with VGA and the Fedora boot screen, then goes blank when switching to X. - The fact that the iommu group now extends across all my available graphics devices now makes it difficult to get the radeon (or catalyst) driver use to the other card since the vfio_pci driver needs to hold it. Not a complete showstopper since the vesa driver comes up with 1024x768.. Might it be a good idea to have an override option (exception list or similar?) to allow the vfio_pci to be less restrictive about owning the whole group - allow functionality over security in such case? This of course is further complicated by the need for graphics drivers to be disabled/enabled already at the kernel prompt.. - There seems to be a bug in the (version F8) UEFI BIOS on the motherboard, The BIOS offers (undocumented) a full range of selections of which PCIe (or PCIe 1x) graphics card to use as primary, but any other selection than the first PCIe 16x slot has no effect and the motherboard reverts to the first slot, so to be able to test both cards, I had to put the card under test into the second (8x) PCIe slot. I am waiting for feedback from Gigabyte on possible fixes for this in newer BIOSes. - The ultimate goal is to try to consolidate some older Windows desktops as "seats" on the new system, using the discrete graphics with HDMI/Displayport audio. With the HD7700 moved to the second PCIe slot I tested both Windows and Linux guests to try to get some sound through the HDMI audio device. Windows complains that no usable device is available. On Linux (Fedora 18, KDE desktop), the system settings -> multimedia dialogue never opens up which seems to indicate that PulseAudio has problems communicating with the passed through device (?), any hints/pointers here appreciated. From the vfio log it seems at least config space is accessed ok. - There also seems to be issues with radeon and intel_iommu=on - if I try to enable modesetting and normal X support for the radeon cards, X fails to start. - It would be nice if the integrated graphics could be used as the host primary display - I would be happy if someone has any hints as to if/how the ifgx_off option could be extended/modified to only affect iommu operation on selected device(s), if at all possible.. Thanks, Knut Omang diff --git a/hw/misc/vfio.c b/hw/misc/vfio.c index da0e5f9..a361d06 100644 --- a/hw/misc/vfio.c +++ b/hw/misc/vfio.c @@ -1291,7 +1291,7 @@ static uint64_t vfio_ati_3c3_quirk_read(void *opaque, uint64_t data = vfio_vga_read(&vdev->vga.region[QEMU_PCI_VGA_IO_HI], addr + quirk->data.base_offset, size); - if (data == quirk->data.address_match) { + if (1 || data == quirk->data.address_match) { data = vfio_pci_read_config(&vdev->pdev, quirk->data.address_val, size); DPRINTF("%s(0x3c3, 1) = 0x%"PRIx64"\n", __func__, data); }