From patchwork Tue Nov 29 01:52:25 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 128196 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [140.186.70.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id EFA3A1007D3 for ; Tue, 29 Nov 2011 12:53:07 +1100 (EST) Received: from localhost ([::1]:60104 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVCsS-0002Z9-5d for incoming@patchwork.ozlabs.org; Mon, 28 Nov 2011 20:52:56 -0500 Received: from eggs.gnu.org ([140.186.70.92]:44843) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVCsL-0002Z1-H4 for qemu-devel@nongnu.org; Mon, 28 Nov 2011 20:52:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RVCsI-0005DE-FS for qemu-devel@nongnu.org; Mon, 28 Nov 2011 20:52:49 -0500 Received: from e23smtp03.au.ibm.com ([202.81.31.145]:39337) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RVCsH-0005CD-Jq for qemu-devel@nongnu.org; Mon, 28 Nov 2011 20:52:46 -0500 Received: from /spool/local by e23smtp03.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 29 Nov 2011 01:47:16 +1000 Received: from d23relay04.au.ibm.com ([202.81.31.246]) by e23smtp03.au.ibm.com ([202.81.31.209]) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 29 Nov 2011 01:47:14 +1000 Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay04.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id pAT1n3XX3465390 for ; Tue, 29 Nov 2011 12:49:06 +1100 Received: from d23av04.au.ibm.com (loopback [127.0.0.1]) by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id pAT1qRtL004381 for ; Tue, 29 Nov 2011 12:52:28 +1100 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.190.163.12]) by d23av04.au.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id pAT1qQna004351; Tue, 29 Nov 2011 12:52:26 +1100 Received: from [10.61.2.175] (haven.au.ibm.com [9.190.164.82]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 7E7377347E; Tue, 29 Nov 2011 12:52:26 +1100 (EST) Message-ID: <4ED43AD9.5090509@au1.ibm.com> Date: Tue, 29 Nov 2011 12:52:25 +1100 From: Alexey Kardashevskiy User-Agent: Mozilla/5.0 (X11; Linux i686; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Alex Williamson References: <20111103195452.21259.93021.stgit@bling.home> In-Reply-To: <20111103195452.21259.93021.stgit@bling.home> x-cbid: 11112815-6102-0000-0000-00000047B851 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 202.81.31.145 Cc: aafabbri@cisco.com, kvm@vger.kernel.org, pmac@au1.ibm.com, qemu-devel@nongnu.org, joerg.roedel@amd.com, konrad.wilk@oracle.com, agraf@suse.de, dwg@au1.ibm.com, chrisw@sous-sol.org, B08248@freescale.com, iommu@lists.linux-foundation.org, avi@redhat.com, linux-pci@vger.kernel.org, B07421@freescale.com, benve@cisco.com Subject: Re: [Qemu-devel] [RFC PATCH] vfio: VFIO Driver core framework X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Hi! I tried (successfully) to run it on POWER and while doing that I found some issues. I'll try to explain them in separate mails. IOMMU domain setup. On POWER, the linux drivers capable of DMA transfer want to know a DMA window, i.e. its start and length in the PHB address space. This comes from hardware. On X86 (correct if I am wrong), every device driver in the guest allocates memory from the same pool. On POWER, device drivers get DMA window and allocate pages for DMA within this window. In the case of VFIO, that means that QEMU has to preallocate this DMA window before running a quest, pass it to a guest (via device tree) and then a guest tells the host what pages are taken/released by calling map/unmap callbacks of iommu_ops. Deallocation is made in a device detach callback as I did not want to add more ioctls. So, there are 2 patches: - new VFIO_IOMMU_SETUP ioctl introduced which allocates a DMA window via IOMMU API on POWER. btw do we need an additional capability bit for it? KERNEL PATCH: On 04/11/11 07:12, Alex Williamson wrote: > VFIO provides a secure, IOMMU based interface for user space > drivers, including device assignment to virtual machines. > This provides the base management of IOMMU groups, devices, > and IOMMU objects. See Documentation/vfio.txt included in > this patch for user and kernel API description. > > Note, this implements the new API discussed at KVM Forum > 2011, as represented by the drvier version 0.2. It's hoped > that this provides a modular enough interface to support PCI > and non-PCI userspace drivers across various architectures > and IOMMU implementations. > > Signed-off-by: Alex Williamson > --- > > Fingers crossed, this is the last RFC for VFIO, but we need > the iommu group support before this can go upstream > (http://lkml.indiana.edu/hypermail/linux/kernel/1110.2/02303.html), > hoping this helps push that along. > > Since the last posting, this version completely modularizes > the device backends and better defines the APIs between the > core VFIO code and the device backends. I expect that we > might also adopt a modular IOMMU interface as iommu_ops learns > about different types of hardware. Also many, many cleanups. > Check the complete git history for details: > > git://github.com/awilliam/linux-vfio.git vfio-ng > > (matching qemu tree: git://github.com/awilliam/qemu-vfio.git) > > This version, along with the supporting VFIO PCI backend can > be found here: > > git://github.com/awilliam/linux-vfio.git vfio-next-20111103 > > I've held off on implementing a kernel->user signaling > mechanism for now since the previous netlink version produced > too many gag reflexes. It's easy enough to set a bit in the > group flags too indicate such support in the future, so I > think we can move ahead without it. > > Appreciate any feedback or suggestions. Thanks, > > Alex > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 10615ad..a882e08 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -247,3 +247,12 @@ int iommu_device_group(struct device *dev, unsigned int *groupid) return -ENODEV; } EXPORT_SYMBOL_GPL(iommu_device_group); + +int iommu_setup(struct iommu_domain *domain, + size_t requested_size, size_t *allocated_size, + phys_addr_t *start_address) +{ + return domain->ops->setup(domain, requested_size, allocated_size, + start_address); +} +EXPORT_SYMBOL_GPL(iommu_setup); diff --git a/drivers/vfio/vfio_iommu.c b/drivers/vfio/vfio_iommu.c index 029dae3..57fb70d 100644 --- a/drivers/vfio/vfio_iommu.c +++ b/drivers/vfio/vfio_iommu.c @@ -507,6 +507,23 @@ static long vfio_iommu_unl_ioctl(struct file *filep, if (!ret && copy_to_user((void __user *)arg, &dm, sizeof dm)) ret = -EFAULT; + + } else if (cmd == VFIO_IOMMU_SETUP) { + struct vfio_setup setup; + size_t allocated_size = 0; + phys_addr_t start_address = 0; + + if (copy_from_user(&setup, (void __user *)arg, sizeof setup)) + return -EFAULT; + + printk("udomain %p, priv=%p\n", iommu->domain, iommu->domain->priv); + ret = iommu_setup(iommu->domain, setup.requested_size, + &allocated_size, &start_address); + setup.allocated_size = allocated_size; + setup.start_address = start_address; + + if (!ret && copy_to_user((void __user *)arg, &setup, sizeof setup)) + ret = -EFAULT; } return ret; } diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 93617e7..355cf8b 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -45,6 +45,7 @@ struct iommu_domain { #define IOMMU_CAP_CACHE_COHERENCY 0x1 #define IOMMU_CAP_INTR_REMAP 0x2 /* isolates device intrs */ +#define IOMMU_CAP_SETUP_REQUIRED 0x3 /* requires setup to be called */ #ifdef CONFIG_IOMMU_API @@ -62,6 +63,9 @@ struct iommu_ops { int (*domain_has_cap)(struct iommu_domain *domain, unsigned long cap); int (*device_group)(struct device *dev, unsigned int *groupid); + int (*setup)(struct iommu_domain *domain, + size_t requested_size, size_t *allocated_size, + phys_addr_t *start_address); }; extern int bus_set_iommu(struct bus_type *bus, struct iommu_ops *ops); @@ -80,6 +84,9 @@ extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, unsigned long iova); extern int iommu_domain_has_cap(struct iommu_domain *domain, unsigned long cap); +extern int iommu_setup(struct iommu_domain *domain, + size_t requested_size, size_t *allocated_size, + phys_addr_t *start_address); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler); extern int iommu_device_group(struct device *dev, unsigned int *groupid); diff --git a/include/linux/vfio.h b/include/linux/vfio.h index 971e3b1..5e0ee75 100644 --- a/include/linux/vfio.h +++ b/include/linux/vfio.h @@ -26,6 +26,7 @@ * Author: Michael S. Tsirkin */ #include +#include #ifndef VFIO_H #define VFIO_H @@ -172,4 +173,13 @@ enum { VFIO_PCI_NUM_IRQS }; +/* Setup domain */ +#define VFIO_IOMMU_SETUP _IOWR(';', 150, struct vfio_setup) + +struct vfio_setup { + __u64 requested_size; + __u64 allocated_size; + __u64 start_address; +}; + #endif /* VFIO_H */ === end === QEMU PATCH: diff --git a/hw/linux-vfio.h b/hw/linux-vfio.h index ac48d85..a2c719f 100644 --- a/hw/linux-vfio.h +++ b/hw/linux-vfio.h @@ -172,4 +172,13 @@ enum { VFIO_PCI_NUM_IRQS }; +/* Setup domain */ +#define VFIO_IOMMU_SETUP _IOWR(';', 150, struct vfio_setup) + +struct vfio_setup { + __u64 requested_size; + __u64 allocated_size; + __u64 start_address; +}; + #endif /* VFIO_H */ diff --git a/hw/vfio_pci.c b/hw/vfio_pci.c index 1c97c35..b438bbe 100644 --- a/hw/vfio_pci.c +++ b/hw/vfio_pci.c @@ -1501,6 +1503,17 @@ static int vfio_initfn(struct PCIDevice *pdev) if (vfio_map_resources(vdev)) goto out_disable_msi; + struct vfio_setup setup = { 1 << 26, 0, 0 }; + if ((ret = ioctl(vdev->group->iommu->fd, VFIO_IOMMU_SETUP, &setup))) { + return ret; + } + printf("SETUP: requested %lluMB, allocated %lluMB at %llx\n", + (unsigned long long)setup.requested_size, + (unsigned long long)setup.allocated_size, + (unsigned long long)setup.start_address); + vdev->start_address = setup.start_address; + vdev->window_size = setup.allocated_size; + if (vfio_enable_intx(vdev)) goto out_unmap_resources; diff --git a/hw/vfio_pci.h b/hw/vfio_pci.h index 96b09bb..6b7ab6f 100644 --- a/hw/vfio_pci.h +++ b/hw/vfio_pci.h @@ -79,6 +79,10 @@ typedef struct VFIODevice { bool msix; uint8_t msix_bar; uint16_t msix_entries; +#ifdef TARGET_PPC + uint64_t start_address; + uint32_t window_size; +#endif } VFIODevice; typedef struct VFIOGroup { === end === - changed __vfio_close_iommu function to do unmapall first and detach devices then as actual deallocation happens on device detach callback of IOMMU ops. diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c index 6169356..f78f411 100644 --- a/drivers/vfio/vfio_main.c +++ b/drivers/vfio/vfio_main.c @@ -28,6 +28,7 @@ #include #include #include +#include #include "vfio_private.h" @@ -242,6 +243,13 @@ static void __vfio_close_iommu(struct vfio_iommu *iommu) if (!iommu->domain) return; + /* + * On POWER, device detaching (which is done by __vfio_iommu_detach_group) + * should happen after all pages unmapped because + * the only way to do actual iommu_unmap_page a device detach callback + */ + vfio_iommu_unmapall(iommu); + list_for_each(pos, &iommu->group_list) { struct vfio_group *group; group = list_entry(pos, struct vfio_group, iommu_next); @@ -249,7 +257,7 @@ static void __vfio_close_iommu(struct vfio_iommu *iommu) __vfio_iommu_detach_group(iommu, group); } - vfio_iommu_unmapall(iommu); + /* vfio_iommu_unmapall(iommu); */ iommu_domain_free(iommu->domain); iommu->domain = NULL;