From patchwork Thu May 17 12:00:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Doug Meyer X-Patchwork-Id: 915375 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=linux-pci-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=gigaio.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gigaio-com.20150623.gappssmtp.com header.i=@gigaio-com.20150623.gappssmtp.com header.b="EteM55T7"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 40mqdk45mNz9s0q for ; Thu, 17 May 2018 22:01:14 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750938AbeEQMA7 (ORCPT ); Thu, 17 May 2018 08:00:59 -0400 Received: from mail-pf0-f193.google.com ([209.85.192.193]:46714 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751804AbeEQMAx (ORCPT ); Thu, 17 May 2018 08:00:53 -0400 Received: by mail-pf0-f193.google.com with SMTP id p12-v6so1986657pff.13 for ; Thu, 17 May 2018 05:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gigaio-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=0MprUk9sDP02lwWFy+oADPSagNTKzJ8N3wUzLSI5LJM=; b=EteM55T7ehkPpGL1FsyVbX7hyI0rP+nvqRLmeN0iBwC8srXEXFLnXyradliSv270wF i2fFpMY2G1/OmGB7L8PEgsCNmzrkXUiyxriw2MJViRdLd1HvIrLdZ2Kcvs7nhBvs2KbM awLLBYcWgW+0KQcOvdcZefjtolmA7hPmnzBE6qlabRFMQMXhBERiuhWwkx3hThL02O58 h3aqBTg92z9yq9p+DxIHiWRM2aKtHY7X0dX7iYTz1WOTQP/sCemV/2l/UjHXuAel8y10 o/VxKIdJciWalZyCL66ztdNFvtBbxclo2BEjj2BC+YYJ/yL3/ZFkxZWhtKdJezm7wXt4 +6WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=0MprUk9sDP02lwWFy+oADPSagNTKzJ8N3wUzLSI5LJM=; b=DbhonNPRAvDtC8uWPM7sYQ0fW7TrriKEtjtauBeGx3+oyvHf4tNulfQVc1VLo29aJ8 w7ELms74aBiZRh2Fy0tovWzhQwByZopoCQ3FCqnyjm1msYanyzsPngKX8l2KPch7udvA 0atWDXQwmwZU2Cq50RanWcWfKBmyeXDqtaeELP8MyS3+JnZ4H6Oh8N7ft/2BgoZiKQ5e RqlAJiWfrPDRzqyJaG/PHLoSUTrS8BbB1rPO838HRMillYgvrE7jVkZzsTu7cngMFqj3 1mbBa2/WEaDy4BaLeW5XHu9ZdtUx9o8WdGR2QEi3ACcl+Q9BlzvQeW5HncK+tZtlhs0U P68A== X-Gm-Message-State: ALKqPwcq+i38rCtbq8FCt04HapYMLuPR+2hhZY0jRBTiZmoZh3CxeRkV hWZqF6r1BnoQOKL+MxCWGXdnAQ== X-Google-Smtp-Source: AB8JxZrQTTNOta6Yl8a71TMfjE01V2kYvin6iH5x7DSFUfAzIzQIuCRiHxl7+O2tLV7hF4eYvJrWQQ== X-Received: by 2002:a63:741b:: with SMTP id p27-v6mr3845275pgc.97.1526558452576; Thu, 17 May 2018 05:00:52 -0700 (PDT) Received: from stryx3.evonexus.local ([69.43.204.50]) by smtp.gmail.com with ESMTPSA id r90-v6sm13052643pfg.122.2018.05.17.05.00.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 17 May 2018 05:00:51 -0700 (PDT) From: dmeyer@gigaio.com To: logang@deltatee.com, kurt.schwemmer@microsemi.com, linux-pci@vger.kernel.org, linux-ntb@googlegroups.com Cc: bhelgaas@google.com, jdmason@kudzu.us, dave.jiang@intel.com, allenbh@gmail.com, linux-kernel@vger.kernel.org, Doug Meyer Subject: [PATCH 2/2] NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On Date: Thu, 17 May 2018 05:00:13 -0700 Message-Id: <1526558413-23113-3-git-send-email-dmeyer@gigaio.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1526558413-23113-1-git-send-email-dmeyer@gigaio.com> References: <1526558413-23113-1-git-send-email-dmeyer@gigaio.com> Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Doug Meyer Here we add the PCI quirk for the Microsemi Switchtec parts to allow non-transparent bridging to work when the IOMMU is turned on. When a requestor on one NT endpoint accesses memory on another NT endpoint, it does this via a devfn proxy ID. Proxy IDs are uniquely assigned on a per-requestor basis within the NT hardware, and are not visible via PCI enumeration. As a result, a memory access from a peer NT endpoint will have an unrecognized requestor ID devfn which the IOMMU will reject. The quirk introduced here interrogates each configured remote NT endpoint at runtime to obtain the proxy IDs that have been assigned, and aliases every proxy ID to the local (enumerated) NT endpoint's device. The IOMMU then accepts the remote proxy IDs as if they were requests coming directly from the enumerated endpoint, giving remote requestors access to memory resources which a given host has made available. Operation with the IOMMU off is unchanged by this quirk. Signed-off-by: Doug Meyer --- drivers/pci/quirks.c | 196 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 2990ad1..c7e27b3 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -27,6 +27,7 @@ #include #include #include +#include #include /* isa_dma_bridge_buggy */ #include "pci.h" @@ -4741,3 +4742,198 @@ static void quirk_gpu_hda(struct pci_dev *hda) PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda); DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID, PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda); + +/* + * Microsemi Switchtec NTB uses devfn proxy IDs to move TLPs between + * NT endpoints via the internal switch fabric. These IDs replace the + * originating requestor ID TLPs which access host memory on peer NTB + * ports. Therefore, all proxy IDs must be aliased to the NTB device + * to permit access when the IOMMU is turned on. + */ +static void quirk_switchtec_ntb_dma_alias(struct pci_dev *pdev) +{ + void __iomem *mmio = NULL; + struct ntb_info_regs __iomem *mmio_ntb = NULL; + struct ntb_ctrl_regs __iomem *mmio_ctrl = NULL; + struct sys_info_regs __iomem *mmio_sys_info = NULL; + + u64 partition_map; + u8 partition; + int pp; + + if (pci_enable_device(pdev)) { + dev_err(&pdev->dev, "Cannot enable Switchtec device\n"); + return; + } + + mmio = pci_iomap(pdev, 0, 0); + if (mmio == NULL) { + dev_err(&pdev->dev, "Cannot iomap Switchtec device\n"); + return; + } + + dev_info(&pdev->dev, "Setting Switchtec proxy ID aliases\n"); + + mmio_ntb = mmio + SWITCHTEC_GAS_NTB_OFFSET; + mmio_ctrl = (void * __iomem) mmio_ntb + SWITCHTEC_NTB_REG_CTRL_OFFSET; + mmio_sys_info = mmio + SWITCHTEC_GAS_SYS_INFO_OFFSET; + + partition = ioread8(&mmio_ntb->partition_id); + + partition_map = (u64) ioread32((void * __iomem) &mmio_ntb->ep_map); + partition_map |= + ((u64) ioread32((void * __iomem) &mmio_ntb->ep_map + 4)) << 32; + partition_map &= ~(1ULL << partition); + + for (pp = 0; pp < (sizeof(partition_map) * 8); pp++) { + struct ntb_ctrl_regs __iomem *mmio_peer_ctrl; + u32 table_sz = 0; + int te; + + if (!(partition_map & (1ULL << pp))) + continue; + + dev_dbg(&pdev->dev, "Processing partition %d\n", pp); + + mmio_peer_ctrl = &mmio_ctrl[pp]; + + table_sz = ioread16(&mmio_peer_ctrl->req_id_table_size); + if (!table_sz) { + dev_warn(&pdev->dev, "Partition %d table_sz 0\n", pp); + continue; + } + + if (table_sz > 512) { + dev_warn(&pdev->dev, + "Invalid Switchtec partition %d table_sz %d\n", + pp, table_sz); + continue; + } + + for (te = 0; te < table_sz; te++) { + u32 rid_entry; + u8 devfn; + + rid_entry = ioread32(&mmio_peer_ctrl->req_id_table[te]); + devfn = (rid_entry >> 1) & 0xFF; + dev_dbg(&pdev->dev, + "Aliasing Partition %d Proxy ID %02d.%d\n", + pp, PCI_SLOT(devfn), PCI_FUNC(devfn)); + pci_add_dma_alias(pdev, devfn); + } + } + + pci_iounmap(pdev, mmio); +} +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX24XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX32XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX48XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX64XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX80XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFX96XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PSX48XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PSX64XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PSX80XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PSX96XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX24XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX32XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX48XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX64XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX80XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PAX96XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL24XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL32XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL48XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL64XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL80XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXL96XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI24XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI32XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI48XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI64XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI80XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias); +DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_MICROSEMI, + PCI_DEVICE_ID_MICROSEMI_PFXI96XG3, + PCI_CLASS_BRIDGE_OTHER, 8, + quirk_switchtec_ntb_dma_alias);