From patchwork Tue Feb 13 05:51:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 872616 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zgWtW2n20z9t39 for ; Tue, 13 Feb 2018 16:53:39 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zgWtW1ms4zF120 for ; Tue, 13 Feb 2018 16:53:39 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=ozlabs.ru (client-ip=107.173.13.209; helo=ozlabs.ru; envelope-from=aik@ozlabs.ru; receiver=) Received: from ozlabs.ru (ozlabs.ru [107.173.13.209]) by lists.ozlabs.org (Postfix) with ESMTP id 3zgWry1SlLzDqpc for ; Tue, 13 Feb 2018 16:52:15 +1100 (AEDT) Received: from vpl1.ozlabs.ibm.com (localhost [IPv6:::1]) by ozlabs.ru (Postfix) with ESMTP id 79A253A60051; Tue, 13 Feb 2018 00:51:46 -0500 (EST) From: Alexey Kardashevskiy To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH kernel] powerpc/npu: Do not try invalidating 32bit table when 64bit table is enabled Date: Tue, 13 Feb 2018 16:51:35 +1100 Message-Id: <20180213055135.25639-1-aik@ozlabs.ru> X-Mailer: git-send-email 2.11.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexey Kardashevskiy , Alistair Popple , David Gibson Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" GPUs and the corresponding NVLink bridges get different PEs as they have separate translation validation entries (TVEs). We put these PEs to the same IOMMU group so they cannot be passed through separately. So the iommu_table_group_ops::set_window/unset_window for GPUs do set tables to the NPU PEs as well which means that iommu_table's list of attached PEs (iommu_table_group_link) has both GPU and NPU PEs linked. This list is used for TCE cache invalidation. The problem is that NPU PE has just a single TVE and can be programmed to point to 32bit or 64bit windows while GPU PE has two (as any other PCI device). So we end up having an 32bit iommu_table struct linked to both PEs even though only the 64bit TCE table cache can be invalidated on NPU. And a relatively recent skiboot detects this and prints errors. This changes GPU's iommu_table_group_ops::set_window/unset_window to make sure that NPU PE is only linked to the table actually used by the hardware. If there are two tables used by an IOMMU group, the NPU PE will use the last programmed one which with the current use scenarios is expected to be a 64bit one. Signed-off-by: Alexey Kardashevskiy --- Do we need BUG_ON(IOMMU_TABLE_GROUP_MAX_TABLES != 2)? This is an example for: 0004:04:00.0 3D: NVIDIA Corporation Device 1db1 (rev a1) 0006:00:00.0 Bridge: IBM Device 04ea (rev 01) 0006:00:00.1 Bridge: IBM Device 04ea (rev 01) Before the patch (npu2_tce_kill messages are from skiboot): pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000 pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000 pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000 pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000 NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000) NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000) NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000) NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000) NPU6: npu2_tce_kill: Unexpected TCE size (got 0x1000 expected 0x10000) ... pci 0004:04 : [PE# 00] Removing DMA window #0 pci 0006:00:00.0: [PE# 0d] Removing DMA window pci 0004:04 : [PE# 00] Removing DMA window #1 pci 0006:00:00.0: [PE# 0d] Removing DMA window pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000 pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000 pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000 pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000 After the patch (no errors here): pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000 pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000 pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000 pci 0006:00:00.0: [PE# 0d] Removing DMA window pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000 pci 0004:04 : [PE# 00] Removing DMA window #0 pci 0004:04 : [PE# 00] Removing DMA window #1 pci 0006:00:00.0: [PE# 0d] Removing DMA window pci 0004:04 : [PE# 00] Setting up window#0 0..3fffffff pg=1000 pci 0006:00:00.0: [PE# 0d] Setting up window 0..3fffffff pg=1000 pci 0004:04 : [PE# 00] Setting up window#1 800000000000000..8000000ffffffff pg=10000 pci 0006:00:00.0: [PE# 0d] Removing DMA window pci 0006:00:00.0: [PE# 0d] Setting up window 800000000000000..8000000ffffffff pg=10000 --- arch/powerpc/platforms/powernv/pci-ioda.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 496e476..2f91815 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -2681,14 +2681,23 @@ static struct pnv_ioda_pe *gpe_table_group_to_npe( static long pnv_pci_ioda2_npu_set_window(struct iommu_table_group *table_group, int num, struct iommu_table *tbl) { + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group); + int num2 = (num == 0) ? 1 : 0; long ret = pnv_pci_ioda2_set_window(table_group, num, tbl); if (ret) return ret; - ret = pnv_npu_set_window(gpe_table_group_to_npe(table_group), num, tbl); - if (ret) + if (table_group->tables[num2]) + pnv_npu_unset_window(npe, num2); + + ret = pnv_npu_set_window(npe, num, tbl); + if (ret) { pnv_pci_ioda2_unset_window(table_group, num); + if (table_group->tables[num2]) + pnv_npu_set_window(npe, num2, + table_group->tables[num2]); + } return ret; } @@ -2697,12 +2706,24 @@ static long pnv_pci_ioda2_npu_unset_window( struct iommu_table_group *table_group, int num) { + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group); + int num2 = (num == 0) ? 1 : 0; long ret = pnv_pci_ioda2_unset_window(table_group, num); if (ret) return ret; - return pnv_npu_unset_window(gpe_table_group_to_npe(table_group), num); + if (!npe->table_group.tables[num]) + return 0; + + ret = pnv_npu_unset_window(npe, num); + if (ret) + return ret; + + if (table_group->tables[num2]) + ret = pnv_npu_set_window(npe, num2, table_group->tables[num2]); + + return ret; } static void pnv_ioda2_npu_take_ownership(struct iommu_table_group *table_group)