From patchwork Tue Nov 29 02:01:34 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 128197 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [140.186.70.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 290261007D2 for ; Tue, 29 Nov 2011 13:02:16 +1100 (EST) Received: from localhost ([::1]:49071 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVD1P-0003mW-JM for incoming@patchwork.ozlabs.org; Mon, 28 Nov 2011 21:02:11 -0500 Received: from eggs.gnu.org ([140.186.70.92]:40122) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVD1H-0003mI-Sa for qemu-devel@nongnu.org; Mon, 28 Nov 2011 21:02:05 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RVD1F-0008Ck-8q for qemu-devel@nongnu.org; Mon, 28 Nov 2011 21:02:03 -0500 Received: from e23smtp08.au.ibm.com ([202.81.31.141]:57158) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVD1E-0008B1-LW for qemu-devel@nongnu.org; Mon, 28 Nov 2011 21:02:01 -0500 Received: from /spool/local by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 29 Nov 2011 02:00:08 +1000 Received: from d23relay03.au.ibm.com ([202.81.31.245]) by e23smtp08.au.ibm.com ([202.81.31.205]) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 29 Nov 2011 02:00:05 +1000 Received: from d23av01.au.ibm.com (d23av01.au.ibm.com [9.190.234.96]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id pAT21aFf5001422 for ; Tue, 29 Nov 2011 13:01:39 +1100 Received: from d23av01.au.ibm.com (loopback [127.0.0.1]) by d23av01.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id pAT21ZEC026774 for ; Tue, 29 Nov 2011 13:01:36 +1100 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.190.163.12]) by d23av01.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id pAT21ZFa026733; Tue, 29 Nov 2011 13:01:35 +1100 Received: from [10.61.2.175] (haven.au.ibm.com [9.190.164.82]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 400307347E; Tue, 29 Nov 2011 13:01:35 +1100 (EST) Message-ID: <4ED43CFE.8040009@au1.ibm.com> Date: Tue, 29 Nov 2011 13:01:34 +1100 From: Alexey Kardashevskiy User-Agent: Mozilla/5.0 (X11; Linux i686; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Alex Williamson References: <20111103195452.21259.93021.stgit@bling.home> <4ED43AD9.5090509@au1.ibm.com> In-Reply-To: <4ED43AD9.5090509@au1.ibm.com> x-cbid: 11112816-5140-0000-0000-00000054F0BB X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 202.81.31.141 Cc: aafabbri@cisco.com, kvm@vger.kernel.org, pmac@au1.ibm.com, qemu-devel@nongnu.org, joerg.roedel@amd.com, konrad.wilk@oracle.com, agraf@suse.de, dwg@au1.ibm.com, chrisw@sous-sol.org, B08248@freescale.com, iommu@lists.linux-foundation.org, avi@redhat.com, linux-pci@vger.kernel.org, B07421@freescale.com, benve@cisco.com Subject: Re: [Qemu-devel] [RFC PATCH] vfio: VFIO Driver core framework X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Hi all, Another problem I hit on POWER - MSI interrupts allocation. The existing VFIO does not expect a PBH to support less interrupts that a device might request. In my case, PHB's limit is 8 interrupts while my test card (10Gb ethernet CXGB3) wants 9. Below are the patches to demonstrate the idea. KERNEL patch: === end === On 29/11/11 12:52, Alexey Kardashevskiy wrote: > Hi! > > I tried (successfully) to run it on POWER and while doing that I found some issues. I'll try to > explain them in separate mails. > > > > On 04/11/11 07:12, Alex Williamson wrote: >> VFIO provides a secure, IOMMU based interface for user space >> drivers, including device assignment to virtual machines. >> This provides the base management of IOMMU groups, devices, >> and IOMMU objects. See Documentation/vfio.txt included in >> this patch for user and kernel API description. >> >> Note, this implements the new API discussed at KVM Forum >> 2011, as represented by the drvier version 0.2. It's hoped >> that this provides a modular enough interface to support PCI >> and non-PCI userspace drivers across various architectures >> and IOMMU implementations. >> >> Signed-off-by: Alex Williamson >> --- >> >> Fingers crossed, this is the last RFC for VFIO, but we need >> the iommu group support before this can go upstream >> (http://lkml.indiana.edu/hypermail/linux/kernel/1110.2/02303.html), >> hoping this helps push that along. >> >> Since the last posting, this version completely modularizes >> the device backends and better defines the APIs between the >> core VFIO code and the device backends. I expect that we >> might also adopt a modular IOMMU interface as iommu_ops learns >> about different types of hardware. Also many, many cleanups. >> Check the complete git history for details: >> >> git://github.com/awilliam/linux-vfio.git vfio-ng >> >> (matching qemu tree: git://github.com/awilliam/qemu-vfio.git) >> >> This version, along with the supporting VFIO PCI backend can >> be found here: >> >> git://github.com/awilliam/linux-vfio.git vfio-next-20111103 >> >> I've held off on implementing a kernel->user signaling >> mechanism for now since the previous netlink version produced >> too many gag reflexes. It's easy enough to set a bit in the >> group flags too indicate such support in the future, so I >> think we can move ahead without it. >> >> Appreciate any feedback or suggestions. Thanks, >> >> Alex >> > > diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c index 7d45c6b..d44b9bf 100644 --- a/drivers/vfio/pci/vfio_pci_intrs.c +++ b/drivers/vfio/pci/vfio_pci_intrs.c @@ -458,17 +458,32 @@ int vfio_pci_setup_msix(struct vfio_pci_device *vdev, int nvec, int __user *inta vdev->msix[i].entry = i; vdev->ev_msix[i] = ctx; } - if (!ret) + if (!ret) { ret = pci_enable_msix(pdev, vdev->msix, nvec); + /* + The kernel is unable to allocate requested number of IRQs + and returned the available number. + */ + if (0 < ret) { + ret = pci_enable_msix(pdev, vdev->msix, ret); + } + } vdev->msix_nvec = 0; - for (i = 0; i < nvec && !ret; i++) { - ret = request_irq(vdev->msix[i].vector, msihandler, 0, - "vfio", vdev->ev_msix[i]); - if (ret) - break; - vdev->msix_nvec = i+1; + if (0 == ret) { + vdev->msix_nvec = 0; + ret = 0; + for (i = 0; i < nvec && !ret; i++) { + ret = request_irq(vdev->msix[i].vector, msihandler, 0, + "vfio", vdev->ev_msix[i]); + if (ret) + break; + vdev->msix_nvec = i+1; + } + if ((0 == vdev->msix_nvec) && (0 != ret)) + vfio_pci_drop_msix(vdev); + else + ret = vdev->msix_nvec; } - if (ret) - vfio_pci_drop_msix(vdev); + return ret; } === end === QEMU patch: diff --git a/hw/vfio_pci.c b/hw/vfio_pci.c index 020961a..980eec7 100644 --- a/hw/vfio_pci.c +++ b/hw/vfio_pci.c @@ -341,7 +341,8 @@ static void vfio_enable_msi(VFIODevice *vdev, bool msix) } } - if (ioctl(vdev->fd, VFIO_DEVICE_SET_IRQ_EVENTFDS, fds)) { + ret = ioctl(vdev->fd, VFIO_DEVICE_SET_IRQ_EVENTFDS, fds); + if (0 > ret) { fprintf(stderr, "vfio: Error: Failed to setup MSI/X fds %s\n", strerror(errno)); for (i = 0; i < vdev->nr_vectors; i++) { @@ -355,6 +356,8 @@ static void vfio_enable_msi(VFIODevice *vdev, bool msix) qemu_free(vdev->msi_vectors); vdev->nr_vectors = 0; return; + } else if (0 < ret) { + vdev->nr_vectors = ret; } vdev->interrupt = msix ? INT_MSIX : INT_MSI;