From patchwork Sat Oct 12 20:32:33 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Varun Sethi X-Patchwork-Id: 283017 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from ozlabs.org (localhost [IPv6:::1]) by ozlabs.org (Postfix) with ESMTP id 1B0342C03C8 for ; Sun, 13 Oct 2013 07:47:38 +1100 (EST) Received: from co9outboundpool.messaging.microsoft.com (co9ehsobe002.messaging.microsoft.com [207.46.163.25]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client CN "mail.global.frontbridge.com", Issuer "MSIT Machine Auth CA 2" (not verified)) by ozlabs.org (Postfix) with ESMTPS id 938AD2C0370 for ; Sun, 13 Oct 2013 07:46:00 +1100 (EST) Received: from mail125-co9-R.bigfish.com (10.236.132.244) by CO9EHSOBE042.bigfish.com (10.236.130.105) with Microsoft SMTP Server id 14.1.225.22; Sat, 12 Oct 2013 20:45:56 +0000 Received: from mail125-co9 (localhost [127.0.0.1]) by mail125-co9-R.bigfish.com (Postfix) with ESMTP id 364533A007F; Sat, 12 Oct 2013 20:45:56 +0000 (UTC) X-Forefront-Antispam-Report: CIP:70.37.183.190; KIP:(null); UIP:(null); IPV:NLI; H:mail.freescale.net; RD:none; EFVD:NLI X-SpamScore: 1 X-BigFish: VS1(zzzz1f42h208ch1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzd2iz1de098h1de097h8275bhz2dh2a8h839he5bhf0ah1288h12a5h12a9h12bdh12e5h137ah139eh13b6h1441h1504h1537h162dh1631h1758h1898h18e1h1946h19b5h1ad9h1b0ah1b2fh1fb3h1d0ch1d2eh1d3fh1dfeh1dffh1e23h1fe8h1ff5h1155h) Received: from mail125-co9 (localhost.localdomain [127.0.0.1]) by mail125-co9 (MessageSwitch) id 1381610753255226_11780; Sat, 12 Oct 2013 20:45:53 +0000 (UTC) Received: from CO9EHSMHS031.bigfish.com (unknown [10.236.132.243]) by mail125-co9.bigfish.com (Postfix) with ESMTP id 391A84001A3; Sat, 12 Oct 2013 20:45:53 +0000 (UTC) Received: from mail.freescale.net (70.37.183.190) by CO9EHSMHS031.bigfish.com (10.236.130.41) with Microsoft SMTP Server (TLS) id 14.16.227.3; Sat, 12 Oct 2013 20:45:53 +0000 Received: from tx30smr01.am.freescale.net (10.81.153.31) by 039-SN1MMR1-004.039d.mgd.msft.net (10.84.1.14) with Microsoft SMTP Server (TLS) id 14.3.158.2; Sat, 12 Oct 2013 20:45:51 +0000 Received: from nmglablinux27.ap.freescale.net ([10.232.15.69]) by tx30smr01.am.freescale.net (8.14.3/8.14.0) with ESMTP id r9CKjTeR019958; Sat, 12 Oct 2013 13:45:48 -0700 From: Varun Sethi To: , , , , , , , Subject: [PATCH 2/3] iommu/fsl: Enable default DMA window for PCIe devices once detached Date: Sun, 13 Oct 2013 02:02:33 +0530 Message-ID: <1381609954-15283-3-git-send-email-Varun.Sethi@freescale.com> X-Mailer: git-send-email 1.7.9.5 In-Reply-To: <1381609954-15283-1-git-send-email-Varun.Sethi@freescale.com> References: <1381609954-15283-1-git-send-email-Varun.Sethi@freescale.com> MIME-Version: 1.0 X-OriginatorOrg: freescale.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% Cc: Varun Sethi X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.16rc2 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" from domain. Once the PCIe device assigned to a guest VM (via VFIO) gets detached from the iommu domain (when guest terminates), its PAMU table entry is disabled. So, this would prevent the device from being used once it's assigned back to the host. This patch allows for creation of a default DMA window corresponding to the device and subsequently enabling the PAMU table entry. Before we enable the entry, we ensure that the device's bus master capability is disabled (device quiesced). Signed-off-by: Varun Sethi --- drivers/iommu/fsl_pamu.c | 43 +++++++++++++++++++++++++++++++-------- drivers/iommu/fsl_pamu.h | 1 + drivers/iommu/fsl_pamu_domain.c | 35 +++++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/fsl_pamu.c b/drivers/iommu/fsl_pamu.c index cba0498..fb4a031 100644 --- a/drivers/iommu/fsl_pamu.c +++ b/drivers/iommu/fsl_pamu.c @@ -225,6 +225,21 @@ static struct paace *pamu_get_spaace(struct paace *paace, u32 wnum) return spaace; } +/* + * Defaul PPAACE settings for an LIODN. + */ +static void setup_default_ppaace(struct paace *ppaace) +{ + pamu_init_ppaace(ppaace); + /* window size is 2^(WSE+1) bytes */ + set_bf(ppaace->addr_bitfields, PPAACE_AF_WSE, 35); + ppaace->wbah = 0; + set_bf(ppaace->addr_bitfields, PPAACE_AF_WBAL, 0); + set_bf(ppaace->impl_attr, PAACE_IA_ATM, + PAACE_ATM_NO_XLATE); + set_bf(ppaace->addr_bitfields, PAACE_AF_AP, + PAACE_AP_PERMS_ALL); +} /** * pamu_get_fspi_and_allocate() - Allocates fspi index and reserves subwindows * required for primary PAACE in the secondary @@ -253,6 +268,24 @@ static unsigned long pamu_get_fspi_and_allocate(u32 subwin_cnt) return (spaace_addr - (unsigned long)spaact) / (sizeof(struct paace)); } +/* Reset the PAACE entry to the default state */ +void enable_default_dma_window(int liodn) +{ + struct paace *ppaace; + + ppaace = pamu_get_ppaace(liodn); + if (!ppaace) { + pr_debug("Invalid liodn entry\n"); + return; + } + + memset(ppaace, 0, sizeof(struct paace)); + + setup_default_ppaace(ppaace); + mb(); + pamu_enable_liodn(liodn); +} + /* Release the subwindows reserved for a particular LIODN */ void pamu_free_subwins(int liodn) { @@ -752,15 +785,7 @@ static void __init setup_liodns(void) continue; } ppaace = pamu_get_ppaace(liodn); - pamu_init_ppaace(ppaace); - /* window size is 2^(WSE+1) bytes */ - set_bf(ppaace->addr_bitfields, PPAACE_AF_WSE, 35); - ppaace->wbah = 0; - set_bf(ppaace->addr_bitfields, PPAACE_AF_WBAL, 0); - set_bf(ppaace->impl_attr, PAACE_IA_ATM, - PAACE_ATM_NO_XLATE); - set_bf(ppaace->addr_bitfields, PAACE_AF_AP, - PAACE_AP_PERMS_ALL); + setup_default_ppaace(ppaace); if (of_device_is_compatible(node, "fsl,qman-portal")) setup_qbman_paace(ppaace, QMAN_PORTAL_PAACE); if (of_device_is_compatible(node, "fsl,qman")) diff --git a/drivers/iommu/fsl_pamu.h b/drivers/iommu/fsl_pamu.h index 8fc1a12..0edcbbbb 100644 --- a/drivers/iommu/fsl_pamu.h +++ b/drivers/iommu/fsl_pamu.h @@ -406,5 +406,6 @@ void get_ome_index(u32 *omi_index, struct device *dev); int pamu_update_paace_stash(int liodn, u32 subwin, u32 value); int pamu_disable_spaace(int liodn, u32 subwin); u32 pamu_get_max_subwin_cnt(void); +void enable_default_dma_window(int liodn); #endif /* __FSL_PAMU_H */ diff --git a/drivers/iommu/fsl_pamu_domain.c b/drivers/iommu/fsl_pamu_domain.c index e02e1de..553ef3c 100644 --- a/drivers/iommu/fsl_pamu_domain.c +++ b/drivers/iommu/fsl_pamu_domain.c @@ -340,6 +340,40 @@ static inline struct device_domain_info *find_domain(struct device *dev) return dev->archdata.iommu_domain; } +/* Disable device DMA capability and enable default DMA window */ +static void disable_device_dma(struct device_domain_info *info) +{ + struct device_domain_info *tmp; + int enable_dma_window = 1; + +#ifdef CONFIG_PCI + if (info->dev->bus == &pci_bus_type) { + struct pci_dev *pdev = NULL; + pdev = to_pci_dev(info->dev); + if (pci_is_enabled(pdev)) + pci_disable_device(pdev); + } +#endif + /* + * Sanity check, to ensure that this is not a + * shared LIODN. In case of a PCIe controller + * it's possible that all PCIe devices share + * the same LIODN. We can't enable the default + * DMA window till all the devices have been + * quiesced (for PCIe devices, we explicitly + * disable the bus master capability). + */ + list_for_each_entry(tmp, &info->domain->devices, link) { + if (info->dev->iommu_group == tmp->dev->iommu_group) { + enable_dma_window = 0; + break; + } + } + + if (enable_dma_window) + enable_default_dma_window(info->liodn); +} + static void remove_device_ref(struct device_domain_info *info, u32 win_cnt) { unsigned long flags; @@ -351,6 +385,7 @@ static void remove_device_ref(struct device_domain_info *info, u32 win_cnt) pamu_disable_liodn(info->liodn); spin_unlock_irqrestore(&iommu_lock, flags); spin_lock_irqsave(&device_domain_lock, flags); + disable_device_dma(info); info->dev->archdata.iommu_domain = NULL; kmem_cache_free(iommu_devinfo_cache, info); spin_unlock_irqrestore(&device_domain_lock, flags);