From patchwork Fri Jan 11 01:09:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023317 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPvc2MNpz9sCh for ; Fri, 11 Jan 2019 12:11:24 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPvc1DXXzDqyG for ; Fri, 11 Jan 2019 12:11:24 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPst5RD2zDqwW for ; Fri, 11 Jan 2019 12:09:54 +1100 (AEDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14hr6096098 for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pxd7083pm-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:47 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19kdP5439942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:46 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4E012A405D; Fri, 11 Jan 2019 01:09:46 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A6667A404D; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 8DEAAA0231; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:21 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-4275-0000-0000-000002FD5BC1 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-4276-0000-0000-0000380B6F14 Message-Id: <2cea91e0c5a43217684f87a1c09de5ad09b94d24.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 01/15] hw/npu2: Simplify BAR structures X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" At the moment, we have struct npu2_bar to represent an NPU BAR, and struct npu2_pcie_bar which wraps an npu2_bar and does nothing other than adding an additional flags field, which is exposed through the PCI virtual device. In npu2_bar, we have another flags field which is used to store state about the BAR, but it's only used internally, which means we use a lot of bitwise manipulations with no benefit other than saving a couple of bytes. Get rid of npu2_pcie_bar, convert the flags in npu2_bar::flags into individual bools, and fix references accordingly. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - rebase on top of Reza's cleanups --- hw/npu2-opencapi.c | 12 ++--- hw/npu2.c | 104 +++++++++++++++++++++------------------------- include/npu2.h | 27 ++++-------- 3 files changed, 66 insertions(+), 77 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 8075d4e536d4..c10cf9863dda 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -807,8 +807,8 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, size); - dev->bars[0].npu2_bar.base = addr; - dev->bars[0].npu2_bar.size = size; + dev->bars[0].base = addr; + dev->bars[0].size = size; reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, addr >> 16); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(size >> 16)); @@ -832,8 +832,8 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", addr); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR), addr, size); - dev->bars[1].npu2_bar.base = addr; - dev->bars[1].npu2_bar.size = size; + dev->bars[1].base = addr; + dev->bars[1].size = size; } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, @@ -1293,7 +1293,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].npu2_bar.base + + genid_base = dev->bars[1].base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; @@ -1351,7 +1351,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].npu2_bar.base + + genid_base = dev->bars[1].base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; diff --git a/hw/npu2.c b/hw/npu2.c index d6b634d75f0c..79b87f196b3a 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -114,7 +114,6 @@ static inline void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) { uint64_t reg, val; - int enabled; reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); val = npu2_read(p, reg); @@ -122,7 +121,7 @@ static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) switch (NPU2_REG(bar->reg)) { case NPU2_PHY_BAR: bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; - enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_2) /* This is the global MMIO BAR */ @@ -133,22 +132,20 @@ static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) case NPU2_NTL0_BAR: case NPU2_NTL1_BAR: bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; - enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); break; case NPU2_GENID_BAR: bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; - enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); bar->size = 0x20000; break; default: bar->base = 0ul; - enabled = 0; + bar->enabled = false; bar->size = 0; break; } - - bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, bar->flags, enabled); } static void npu2_write_bar(struct npu2 *p, @@ -156,23 +153,23 @@ static void npu2_write_bar(struct npu2 *p, uint32_t gcid, uint32_t scom) { - uint64_t reg, val, enable = !!(bar->flags & NPU2_BAR_FLAG_ENABLED); + uint64_t reg, val; int block; switch (NPU2_REG(bar->reg)) { case NPU2_PHY_BAR: val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); break; case NPU2_NTL0_BAR: case NPU2_NTL1_BAR: val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); val = SETFIELD(NPU2_NTL_BAR_SIZE, val, 1); break; case NPU2_GENID_BAR: val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); break; default: val = 0ul; @@ -211,10 +208,10 @@ static int64_t npu2_cfg_write_cmd(void *dev, * one GENID BAR, which is exposed via the first brick. */ enabled = !!(*data & PCI_CFG_CMD_MEM_EN); - ntl_npu_bar = &ndev->bars[0].npu2_bar; - genid_npu_bar = &ndev->bars[1].npu2_bar; + ntl_npu_bar = &ndev->bars[0]; + genid_npu_bar = &ndev->bars[1]; - ntl_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, ntl_npu_bar->flags, enabled); + ntl_npu_bar->enabled = enabled; npu2_write_bar(ndev->npu, ntl_npu_bar, 0, 0); /* @@ -223,16 +220,12 @@ static int64_t npu2_cfg_write_cmd(void *dev, * track the enables separately. */ if (NPU2DEV_BRICK(ndev)) - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED1, genid_npu_bar->flags, - enabled); + genid_npu_bar->enabled1 = enabled; else - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED0, genid_npu_bar->flags, - enabled); + genid_npu_bar->enabled0 = enabled; /* Enable the BAR if either device requests it enabled, otherwise disable it */ - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, genid_npu_bar->flags, - !!(genid_npu_bar->flags & (NPU2_BAR_FLAG_ENABLED0 | - NPU2_BAR_FLAG_ENABLED1))); + genid_npu_bar->enabled = genid_npu_bar->enabled0 || genid_npu_bar->enabled1; npu2_write_bar(ndev->npu, genid_npu_bar, 0, 0); return OPAL_PARTIAL; @@ -243,20 +236,21 @@ static int64_t npu2_cfg_read_bar(struct npu2_dev *dev __unused, uint32_t offset, uint32_t size, uint32_t *data) { - struct npu2_pcie_bar *bar = (struct npu2_pcie_bar *) pcrf->data; + struct npu2_bar *bar = (struct npu2_bar *) pcrf->data; - if (!(bar->flags & NPU2_PCIE_BAR_FLAG_TRAPPED)) + if (!(bar->trapped)) return OPAL_PARTIAL; if ((size != 4) || (offset != pcrf->start && offset != pcrf->start + 4)) return OPAL_PARAMETER; - if (bar->flags & NPU2_PCIE_BAR_FLAG_SIZE_HI) - *data = bar->npu2_bar.size >> 32; + if (bar->size_hi) + *data = bar->size >> 32; else - *data = bar->npu2_bar.size; - bar->flags &= ~(NPU2_PCIE_BAR_FLAG_TRAPPED | NPU2_PCIE_BAR_FLAG_SIZE_HI); + *data = bar->size; + bar->trapped = false; + bar->size_hi = false; return OPAL_SUCCESS; } @@ -266,8 +260,8 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, uint32_t offset, uint32_t size, uint32_t data) { - struct npu2_pcie_bar *bar = (struct npu2_pcie_bar *) pcrf->data; - struct npu2_bar old_bar, *npu2_bar = &bar->npu2_bar; + struct npu2_bar *bar = (struct npu2_bar *) pcrf->data; + struct npu2_bar old_bar; if ((size != 4) || (offset != pcrf->start && offset != pcrf->start + 4)) @@ -275,34 +269,34 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, /* Return BAR size on next read */ if (data == 0xffffffff) { - bar->flags |= NPU2_PCIE_BAR_FLAG_TRAPPED; + bar->trapped = true; if (offset == pcrf->start + 4) - bar->flags |= NPU2_PCIE_BAR_FLAG_SIZE_HI; + bar->size_hi = true; return OPAL_SUCCESS; } if (offset == pcrf->start) { - npu2_bar->base &= 0xffffffff00000000; - npu2_bar->base |= (data & 0xfffffff0); + bar->base &= 0xffffffff00000000; + bar->base |= (data & 0xfffffff0); } else { - npu2_bar->base &= 0x00000000ffffffff; - npu2_bar->base |= ((uint64_t)data << 32); + bar->base &= 0x00000000ffffffff; + bar->base |= ((uint64_t)data << 32); - if (NPU2_REG(npu2_bar->reg) == NPU2_GENID_BAR && NPU2DEV_BRICK(dev)) - npu2_bar->base -= 0x10000; + if (NPU2_REG(bar->reg) == NPU2_GENID_BAR && NPU2DEV_BRICK(dev)) + bar->base -= 0x10000; - old_bar.reg = npu2_bar->reg; + old_bar.reg = bar->reg; npu2_read_bar(dev->npu, &old_bar); /* Only allow changing the base address if the BAR is not enabled */ - if ((npu2_bar->flags & NPU2_BAR_FLAG_ENABLED) && - (npu2_bar->base != old_bar.base)) { - npu2_bar->base = old_bar.base; + if (bar->enabled && + (bar->base != old_bar.base)) { + bar->base = old_bar.base; return OPAL_HARDWARE; } - npu2_write_bar(dev->npu, &bar->npu2_bar, 0, 0); + npu2_write_bar(dev->npu, bar, 0, 0); } /* To update the config cache */ @@ -1408,13 +1402,13 @@ static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint /* NPU_REGS must be first in this list */ { .type = NPU_REGS, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_PHY, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_PHY, .index = 1, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_NTL, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL0_BAR) }, { .type = NPU_NTL, .index = 1, @@ -1670,7 +1664,7 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, static void npu2_populate_cfg(struct npu2_dev *dev) { struct pci_virt_device *pvd = dev->nvlink.pvd; - struct npu2_pcie_bar *bar; + struct npu2_bar *bar; uint32_t pos; /* 0x00 - Vendor/Device ID */ @@ -1692,9 +1686,9 @@ static void npu2_populate_cfg(struct npu2_dev *dev) /* 0x10/14 - BAR#0, NTL BAR */ bar = &dev->bars[0]; PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR0, 4, - (bar->npu2_bar.base & 0xfffffff0) | (bar->flags & 0xF), + (bar->base & 0xfffffff0) | bar->flags, 0x0000000f, 0x00000000); - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR1, 4, (bar->npu2_bar.base >> 32), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR1, 4, (bar->base >> 32), 0x00000000, 0x00000000); pci_virt_add_filter(pvd, PCI_CFG_BAR0, 8, PCI_REG_FLAG_READ | PCI_REG_FLAG_WRITE, @@ -1703,16 +1697,16 @@ static void npu2_populate_cfg(struct npu2_dev *dev) /* 0x18/1c - BAR#1, GENID BAR */ bar = &dev->bars[1]; if (NPU2DEV_BRICK(dev) == 0) - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->npu2_bar.base & 0xfffffff0) | - (bar->flags & 0xF), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->base & 0xfffffff0) | + bar->flags, 0x0000000f, 0x00000000); else /* Brick 1 gets the upper portion of the generation id register */ - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, ((bar->npu2_bar.base + 0x10000) & 0xfffffff0) | - (bar->flags & 0xF), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, ((bar->base + 0x10000) & 0xfffffff0) | + bar->flags, 0x0000000f, 0x00000000); - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR3, 4, (bar->npu2_bar.base >> 32), 0x00000000, + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR3, 4, (bar->base >> 32), 0x00000000, 0x00000000); pci_virt_add_filter(pvd, PCI_CFG_BAR2, 8, PCI_REG_FLAG_READ | PCI_REG_FLAG_WRITE, @@ -1802,7 +1796,7 @@ static void npu2_populate_devices(struct npu2 *p, /* Populate BARs. BAR0/1 is the NTL bar. */ stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); - npu2_bar = &dev->bars[0].npu2_bar; + npu2_bar = &dev->bars[0]; npu2_bar->type = NPU_NTL; npu2_bar->index = dev->brick_index; npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? @@ -1812,7 +1806,7 @@ static void npu2_populate_devices(struct npu2 *p, dev->bars[0].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; /* BAR2/3 is the GENID bar. */ - npu2_bar = &dev->bars[1].npu2_bar; + npu2_bar = &dev->bars[1]; npu2_bar->type = NPU_GENID; npu2_bar->index = NPU2DEV_STACK(dev); npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); diff --git a/include/npu2.h b/include/npu2.h index f51a6a2a8ffb..fcaeb55f67dc 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -67,26 +67,21 @@ struct npu2_bar { enum phys_map_type type; int index; -#define NPU2_BAR_FLAG_ENABLED 0x0010 + uint32_t flags; /* NVLink: exported to PCI config space */ + uint64_t base; + uint64_t size; + uint64_t reg; + + bool enabled; /* Generation ID's are a single space in the hardware but we split * them in two for the emulated PCIe devices so we need to keep track * of which one has been enabled/disabled. */ -#define NPU2_BAR_FLAG_ENABLED0 0x0080 -#define NPU2_BAR_FLAG_ENABLED1 0x0100 - uint32_t flags; - uint64_t base; - uint64_t size; - uint64_t reg; -}; + bool enabled0; + bool enabled1; -/* Rpresents a BAR that is exposed via the PCIe emulated - * devices */ -struct npu2_pcie_bar { -#define NPU2_PCIE_BAR_FLAG_SIZE_HI 0x0020 -#define NPU2_PCIE_BAR_FLAG_TRAPPED 0x0040 - uint32_t flags; - struct npu2_bar npu2_bar; + bool size_hi; + bool trapped; }; enum npu2_dev_type { @@ -121,7 +116,7 @@ struct npu2_dev { uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; - struct npu2_pcie_bar bars[2]; + struct npu2_bar bars[2]; struct npu2 *npu; uint32_t bdfn; From patchwork Fri Jan 11 01:09:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023320 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPwR1vgWz9sCh for ; Fri, 11 Jan 2019 12:12:07 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPwQ6dXJzDqwh for ; Fri, 11 Jan 2019 12:12:06 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsv5ybZzDqwL for ; Fri, 11 Jan 2019 12:09:55 +1100 (AEDT) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14F6Y124516 for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pxe1he7bq-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:49 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:47 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19kR87668082 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:46 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4B55A42047; Fri, 11 Jan 2019 01:09:46 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A5A1542041; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id A2EA8A0252; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:22 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0028-0000-0000-000003375C7D X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0029-0000-0000-000023F4734A Message-Id: <905b7b786e7e50ef3845493c8a467224f5cb5200.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 02/15] hw/npu2: Merge implementations of BAR accessor functions X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Move npu2_{get,read,write}_bar() to common code. Get rid of the OpenCAPI write_bar() function. Fix the OpenCAPI code to use the NVLink-style BAR accessor functions and struct npu2_bar. While we're there, fix a trivial bug in npu2_read_bar() when reading PHY BARs - the global MMIO BAR is stack 0, not stack 2. I don't think we ever read that BAR, so this has never been a problem. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - remove dead phys_map_get() (Fred) --- hw/npu2-common.c | 77 ++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 112 ++++++++++++++++------------------------------ hw/npu2.c | 78 +-------------------------------- include/npu2.h | 4 ++- 4 files changed, 121 insertions(+), 150 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 6e6b12f0d1ae..3446acb45bea 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -22,6 +22,7 @@ #include #include #include +#include /* * We use the indirect method because it uses the same addresses as @@ -97,6 +98,82 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas (uint64_t)new_val << 32); } +void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) +{ + phys_map_get(gcid, bar->type, bar->index, &bar->base, &bar->size); +} + +void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) +{ + uint64_t reg, val; + + reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); + val = npu2_read(p, reg); + + switch (NPU2_REG(bar->reg)) { + case NPU2_PHY_BAR: + bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; + bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); + + if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_0) + /* This is the global MMIO BAR */ + bar->size = 0x1000000; + else + bar->size = 0x200000; + break; + case NPU2_NTL0_BAR: + case NPU2_NTL1_BAR: + bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; + bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); + bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); + break; + case NPU2_GENID_BAR: + bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; + bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); + bar->size = 0x20000; + break; + default: + bar->base = 0ul; + bar->enabled = false; + bar->size = 0; + break; + } +} + +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, + uint32_t scom) +{ + uint64_t reg, val; + int block; + + switch (NPU2_REG(bar->reg)) { + case NPU2_PHY_BAR: + val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); + val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); + break; + case NPU2_NTL0_BAR: + case NPU2_NTL1_BAR: + val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); + val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); + val = SETFIELD(NPU2_NTL_BAR_SIZE, val, ilog2(bar->size >> 16)); + break; + case NPU2_GENID_BAR: + val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); + val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); + break; + default: + val = 0ul; + } + + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + reg = NPU2_REG_OFFSET(0, block, bar->reg); + if (p) + npu2_write(p, reg, val); + else + npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); + } +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index c10cf9863dda..ffa4d706bbad 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -730,63 +730,29 @@ static void address_translation_config(uint32_t gcid, uint32_t scom_base, } } -/* TODO: Merge this with NVLink implementation - we don't use the npu2_bar - * wrapper for the PHY BARs yet */ -static void write_bar(uint32_t gcid, uint32_t scom_base, uint64_t reg, - uint64_t addr, uint64_t size) -{ - uint64_t val; - int block; - switch (NPU2_REG(reg)) { - case NPU2_PHY_BAR: - val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, addr >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, 1); - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, addr >> 16); - val = SETFIELD(NPU2_NTL_BAR_SIZE, val, ilog2(size >> 16)); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, 1); - break; - case NPU2_GENID_BAR: - val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, addr >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, 1); - break; - default: - val = 0ul; - } - - for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { - npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(0, block, reg), - NPU2_MISC_DA_LEN_8B, val); - prlog(PR_DEBUG, "OCAPI: Setting BAR %llx to %llx\n", - NPU2_REG_OFFSET(0, block, reg), val); - } -} - static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, uint64_t reg[]) { - uint64_t addr, size; - - prlog(PR_DEBUG, "OCAPI: patching up PHY0 bar, %s\n", __func__); - phys_map_get(gcid, NPU_PHY, 0, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - addr, size); - prlog(PR_DEBUG, "OCAPI: patching up PHY1 bar, %s\n", __func__); - phys_map_get(gcid, NPU_PHY, 1, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - addr, size); - - prlog(PR_DEBUG, "OCAPI: setup global mmio, %s\n", __func__); - phys_map_get(gcid, NPU_REGS, 0, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - addr, size); - reg[0] = addr; - reg[1] = size; + struct npu2_bar *bar; + struct npu2_bar phy_bars[] = { + { .type = NPU_PHY, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 1, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_REGS, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), + .enabled = true }, + }; + + for (int i = 0; i < ARRAY_SIZE(phy_bars); i++) { + bar = &phy_bars[i]; + npu2_get_bar(gcid, bar); + npu2_write_bar(NULL, bar, gcid, scom_base); + } + reg[0] = phy_bars[2].base; + reg[1] = phy_bars[2].size; } /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ @@ -799,19 +765,21 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, uint64_t pa_offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; - uint64_t addr, size, reg; + uint64_t reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - phys_map_get(gcid, NPU_OCAPI_MMIO, dev->brick_index, &addr, &size); - - prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); - write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, - size); - dev->bars[0].base = addr; - dev->bars[0].size = size; - - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, addr >> 16); - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(size >> 16)); + dev->bars[0].type = NPU_OCAPI_MMIO; + dev->bars[0].index = dev->brick_index; + dev->bars[0].reg = NPU2_REG_OFFSET(stack, 0, offset); + dev->bars[0].enabled = true; + npu2_get_bar(gcid, &dev->bars[0]); + + prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", + dev->bars[0].base, dev->bars[0].size); + npu2_write_bar(NULL, &dev->bars[0], gcid, scom_base); + + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->bars[0].base >> 16); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->bars[0].size >> 16)); prlog(PR_DEBUG, "OCAPI: PA translation %llx\n", reg); npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, @@ -825,15 +793,15 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, { uint64_t stack = index_to_stack(dev->brick_index); int stack_num = stack - NPU2_STACK_STCK_0; - uint64_t addr, size; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); - phys_map_get(gcid, NPU_GENID, stack_num, &addr, &size); - prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", addr); - write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR), - addr, size); - dev->bars[1].base = addr; - dev->bars[1].size = size; + dev->bars[1].type = NPU_GENID; + dev->bars[1].index = stack_num; + dev->bars[1].reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); + dev->bars[1].enabled = true; + npu2_get_bar(gcid, &dev->bars[1]); + prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->bars[1].base); + npu2_write_bar(NULL, &dev->bars[1], gcid, scom_base); } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, diff --git a/hw/npu2.c b/hw/npu2.c index 79b87f196b3a..8176a81262f0 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -106,84 +106,6 @@ static struct npu2_dev *npu2_bdf_to_dev(struct npu2 *p, return NULL; } -static inline void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) -{ - phys_map_get(gcid, bar->type, bar->index, &bar->base, &bar->size); -} - -static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) -{ - uint64_t reg, val; - - reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); - val = npu2_read(p, reg); - - switch (NPU2_REG(bar->reg)) { - case NPU2_PHY_BAR: - bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; - bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); - - if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_2) - /* This is the global MMIO BAR */ - bar->size = 0x1000000; - else - bar->size = 0x200000; - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; - bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); - bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); - break; - case NPU2_GENID_BAR: - bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; - bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); - bar->size = 0x20000; - break; - default: - bar->base = 0ul; - bar->enabled = false; - bar->size = 0; - break; - } -} - -static void npu2_write_bar(struct npu2 *p, - struct npu2_bar *bar, - uint32_t gcid, - uint32_t scom) -{ - uint64_t reg, val; - int block; - - switch (NPU2_REG(bar->reg)) { - case NPU2_PHY_BAR: - val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); - val = SETFIELD(NPU2_NTL_BAR_SIZE, val, 1); - break; - case NPU2_GENID_BAR: - val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); - break; - default: - val = 0ul; - } - - for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { - reg = NPU2_REG_OFFSET(0, block, bar->reg); - if (p) - npu2_write(p, reg, val); - else - npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); - } -} - /* Trap for PCI command (0x4) to enable or disable device's BARs */ static int64_t npu2_cfg_write_cmd(void *dev, struct pci_cfg_reg_filter *pcrf __unused, diff --git a/include/npu2.h b/include/npu2.h index fcaeb55f67dc..b2a3ead858d2 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -204,6 +204,10 @@ void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val); uint64_t npu2_read(struct npu2 *p, uint64_t reg); void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask); +void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar); +void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar); +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, + uint32_t scom); int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); From patchwork Fri Jan 11 01:09:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023315 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPv50Zjsz9sCh for ; Fri, 11 Jan 2019 12:10:57 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPv44xQmzDqwN for ; Fri, 11 Jan 2019 12:10:56 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPst5tBKzDqwb for ; Fri, 11 Jan 2019 12:09:54 +1100 (AEDT) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14GQ2135701 for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxa9x7vp8-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:47 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19kPF6488436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:46 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4A889AE053; Fri, 11 Jan 2019 01:09:46 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A4C75AE051; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B377CA0332; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:23 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0020-0000-0000-00000304A96B X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0021-0000-0000-00002155AFAF Message-Id: <30cda5baa5bafbb1070fcd2b1bab1150a7b42b89.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=839 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 03/15] hw/npu2: Replace npu2_dev::bars array with individual BARs X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" We currently maintain the state of the NTL and GENID BARs for each device in npu2_dev::bars[2], with the first BAR being the NTL BAR and the second being the GENID BAR. Given we only have two BARs, it's a lot clearer to get rid of the array and just have two struct members, ntl_bar and genid_bar. No functional change. Signed-off-by: Andrew Donnellan Reviewed-by: Alexey Kardashevskiy --- v1->v2: - split this patch out --- hw/npu2-opencapi.c | 36 ++++++++++++++++----------------- hw/npu2.c | 52 ++++++++++++++++++++++------------------------- include/npu2.h | 3 ++- 3 files changed, 45 insertions(+), 46 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index ffa4d706bbad..8dac552adea8 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -768,18 +768,18 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, uint64_t reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - dev->bars[0].type = NPU_OCAPI_MMIO; - dev->bars[0].index = dev->brick_index; - dev->bars[0].reg = NPU2_REG_OFFSET(stack, 0, offset); - dev->bars[0].enabled = true; - npu2_get_bar(gcid, &dev->bars[0]); + dev->ntl_bar.type = NPU_OCAPI_MMIO; + dev->ntl_bar.index = dev->brick_index; + dev->ntl_bar.reg = NPU2_REG_OFFSET(stack, 0, offset); + dev->ntl_bar.enabled = true; + npu2_get_bar(gcid, &dev->ntl_bar); prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", - dev->bars[0].base, dev->bars[0].size); - npu2_write_bar(NULL, &dev->bars[0], gcid, scom_base); + dev->ntl_bar.base, dev->ntl_bar.size); + npu2_write_bar(NULL, &dev->ntl_bar, gcid, scom_base); - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->bars[0].base >> 16); - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->bars[0].size >> 16)); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->ntl_bar.base >> 16); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->ntl_bar.size >> 16)); prlog(PR_DEBUG, "OCAPI: PA translation %llx\n", reg); npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, @@ -795,13 +795,13 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, int stack_num = stack - NPU2_STACK_STCK_0; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); - dev->bars[1].type = NPU_GENID; - dev->bars[1].index = stack_num; - dev->bars[1].reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); - dev->bars[1].enabled = true; - npu2_get_bar(gcid, &dev->bars[1]); - prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->bars[1].base); - npu2_write_bar(NULL, &dev->bars[1], gcid, scom_base); + dev->genid_bar.type = NPU_GENID; + dev->genid_bar.index = stack_num; + dev->genid_bar.reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); + dev->genid_bar.enabled = true; + npu2_get_bar(gcid, &dev->genid_bar); + prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->genid_bar.base); + npu2_write_bar(NULL, &dev->genid_bar, gcid, scom_base); } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, @@ -1261,7 +1261,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].base + + genid_base = dev->genid_bar.base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; @@ -1319,7 +1319,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].base + + genid_base = dev->genid_bar.base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; diff --git a/hw/npu2.c b/hw/npu2.c index 8176a81262f0..60619f381301 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -114,7 +114,6 @@ static int64_t npu2_cfg_write_cmd(void *dev, { struct pci_virt_device *pvd = dev; struct npu2_dev *ndev = pvd->data; - struct npu2_bar *ntl_npu_bar, *genid_npu_bar; bool enabled; if (!write) @@ -130,11 +129,9 @@ static int64_t npu2_cfg_write_cmd(void *dev, * one GENID BAR, which is exposed via the first brick. */ enabled = !!(*data & PCI_CFG_CMD_MEM_EN); - ntl_npu_bar = &ndev->bars[0]; - genid_npu_bar = &ndev->bars[1]; - ntl_npu_bar->enabled = enabled; - npu2_write_bar(ndev->npu, ntl_npu_bar, 0, 0); + ndev->ntl_bar.enabled = enabled; + npu2_write_bar(ndev->npu, &ndev->ntl_bar, 0, 0); /* * Enable/disable the GENID BAR. Two bricks share one GENID @@ -142,13 +139,14 @@ static int64_t npu2_cfg_write_cmd(void *dev, * track the enables separately. */ if (NPU2DEV_BRICK(ndev)) - genid_npu_bar->enabled1 = enabled; + ndev->genid_bar.enabled1 = enabled; else - genid_npu_bar->enabled0 = enabled; + ndev->genid_bar.enabled0 = enabled; /* Enable the BAR if either device requests it enabled, otherwise disable it */ - genid_npu_bar->enabled = genid_npu_bar->enabled0 || genid_npu_bar->enabled1; - npu2_write_bar(ndev->npu, genid_npu_bar, 0, 0); + ndev->genid_bar.enabled = ndev->genid_bar.enabled0 || + ndev->genid_bar.enabled1; + npu2_write_bar(ndev->npu, &ndev->genid_bar, 0, 0); return OPAL_PARTIAL; } @@ -1606,7 +1604,7 @@ static void npu2_populate_cfg(struct npu2_dev *dev) PCI_VIRT_CFG_INIT_RO(pvd, PCI_CFG_CACHE_LINE_SIZE, 4, 0x00800000); /* 0x10/14 - BAR#0, NTL BAR */ - bar = &dev->bars[0]; + bar = &dev->ntl_bar; PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR0, 4, (bar->base & 0xfffffff0) | bar->flags, 0x0000000f, 0x00000000); @@ -1617,7 +1615,7 @@ static void npu2_populate_cfg(struct npu2_dev *dev) npu2_dev_cfg_bar, bar); /* 0x18/1c - BAR#1, GENID BAR */ - bar = &dev->bars[1]; + bar = &dev->genid_bar; if (NPU2DEV_BRICK(dev) == 0) PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->base & 0xfffffff0) | bar->flags, @@ -1696,7 +1694,6 @@ static void npu2_populate_devices(struct npu2 *p, p->phb_nvlink.scan_map = 0; dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { uint32_t group_id; - struct npu2_bar *npu2_bar; dev = &p->devices[index]; dev->type = NPU2_DEV_TYPE_NVLINK; @@ -1718,28 +1715,29 @@ static void npu2_populate_devices(struct npu2 *p, /* Populate BARs. BAR0/1 is the NTL bar. */ stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); - npu2_bar = &dev->bars[0]; - npu2_bar->type = NPU_NTL; - npu2_bar->index = dev->brick_index; - npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? - NPU2_NTL0_BAR : NPU2_NTL1_BAR); - npu2_get_bar(p->chip_id, npu2_bar); + dev->ntl_bar.type = NPU_NTL; + dev->ntl_bar.index = dev->brick_index; + dev->ntl_bar.reg = NPU2_REG_OFFSET(stack, 0, + NPU2DEV_BRICK(dev) == 0 ? + NPU2_NTL0_BAR : + NPU2_NTL1_BAR); + npu2_get_bar(p->chip_id, &dev->ntl_bar); - dev->bars[0].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + dev->ntl_bar.flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; /* BAR2/3 is the GENID bar. */ - npu2_bar = &dev->bars[1]; - npu2_bar->type = NPU_GENID; - npu2_bar->index = NPU2DEV_STACK(dev); - npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); - npu2_get_bar(p->chip_id, npu2_bar); + dev->genid_bar.type = NPU_GENID; + dev->genid_bar.index = NPU2DEV_STACK(dev); + dev->genid_bar.reg = NPU2_REG_OFFSET(stack, 0, + NPU2_GENID_BAR); + npu2_get_bar(p->chip_id, &dev->genid_bar); /* The GENID is a single physical BAR that we split * for each emulated device */ - npu2_bar->size = 0x10000; + dev->genid_bar.size = 0x10000; if (NPU2DEV_BRICK(dev)) - npu2_bar->base += 0x10000; - dev->bars[1].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + dev->genid_bar.base += 0x10000; + dev->genid_bar.flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; /* Initialize PCI virtual device */ dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); diff --git a/include/npu2.h b/include/npu2.h index b2a3ead858d2..c2ba4370c063 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -116,7 +116,8 @@ struct npu2_dev { uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; - struct npu2_bar bars[2]; + struct npu2_bar ntl_bar; + struct npu2_bar genid_bar; struct npu2 *npu; uint32_t bdfn; From patchwork Fri Jan 11 01:09:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023319 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPw91RnFz9sCh for ; Fri, 11 Jan 2019 12:11:53 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPw90SVzzDqyk for ; Fri, 11 Jan 2019 12:11:53 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsv2nYVzDqws for ; Fri, 11 Jan 2019 12:09:55 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14YTk049045 for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxc52b4dh-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:47 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19kcg721222 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:46 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4C3FDA405B; Fri, 11 Jan 2019 01:09:46 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A5B66A4040; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:45 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id C876BA03A5; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:24 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0016-0000-0000-00000243A7EB X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0017-0000-0000-0000329DAF09 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 04/15] hw/npu2: Move PHY/NTL/GENID BAR assignment to common code X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Assignment of PHY/NTL/GENID BARs is currently duplicated between NVLink and OpenCAPI. This is going to cause us particular issues later on when we implement support for mixed-mode setups with NVLink and OpenCAPI on the same NPU. Centralise the assignment of PHY/NTL/GENID BARs in common code. Signed-off-by: Andrew Donnellan --- v1->v2: - keep config bar enablement in separate function (Fred) - fix GENID BAR assignment for OpenCAPI OTL1 - split patch (Alexey) --- hw/npu2-common.c | 78 +++++++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 61 +++-------------------------- hw/npu2.c | 98 ++--------------------------------------------- include/npu2.h | 1 +- 4 files changed, 91 insertions(+), 147 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 3446acb45bea..944719d2489a 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -174,6 +175,81 @@ void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, } } +static void assign_bars(struct npu2 *npu) +{ + uint32_t i; + struct npu2_bar *bar; + struct npu2_dev *dev; + struct npu2_bar phy_bars[] = { + /* NPU_REGS must be first in this list */ + { .type = NPU_REGS, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 1, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), + .enabled = true }, + }; + struct npu2_bar last_bar = + { .type = NPU_GENID, .index = 2, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_GENID_BAR) }; + + /* Set common PHY BARs */ + for (i = 0; i < ARRAY_SIZE(phy_bars); i++) { + bar = &phy_bars[i]; + npu2_get_bar(npu->chip_id, bar); + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + } + + /* Device BARs */ + for (i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type == NPU2_DEV_TYPE_UNKNOWN) + continue; + + /* NTL / OCAPI_MMIO BAR */ + bar = &dev->ntl_bar; + if (dev->type == NPU2_DEV_TYPE_NVLINK) + bar->type = NPU_NTL; + else if (dev->type == NPU2_DEV_TYPE_OPENCAPI) + bar->type = NPU_OCAPI_MMIO; + bar->index = dev->brick_index; + bar->reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev), + 0, NPU2DEV_BRICK(dev) == 0 ? + NPU2_NTL0_BAR : NPU2_NTL1_BAR); + bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + npu2_get_bar(npu->chip_id, bar); + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + + /* GENID BAR */ + bar = &dev->genid_bar; + bar->type = NPU_GENID; + bar->index = NPU2DEV_STACK(dev); + bar->reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev), + 0, NPU2_GENID_BAR); + bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + npu2_get_bar(npu->chip_id, bar); + if (dev->type == NPU2_DEV_TYPE_NVLINK) { + bar->size = 0x10000; + if (NPU2DEV_BRICK(dev)) + bar->base += 0x10000; + } + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + }; + + /* Global MMIO BAR */ + npu->regs = (uint64_t *)phy_bars[0].base; + npu->regs_size = phy_bars[0].size; + + /* NTL and GENID BARs are exposed to kernel via the mm + * window */ + npu2_get_bar(npu->chip_id, &last_bar); + npu->mm_base = phy_bars[2].base + phy_bars[2].size; + npu->mm_size = last_bar.base + last_bar.size - npu->mm_base; +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; @@ -350,6 +426,8 @@ static void setup_devices(struct npu2 *npu) return; } + assign_bars(npu); + if (nvlink_detected) npu2_nvlink_init_npu(npu); else if (ocapi_detected) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 8dac552adea8..17a16be00eba 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -730,53 +730,19 @@ static void address_translation_config(uint32_t gcid, uint32_t scom_base, } } -static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, - uint64_t reg[]) -{ - struct npu2_bar *bar; - struct npu2_bar phy_bars[] = { - { .type = NPU_PHY, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_REGS, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .enabled = true }, - }; - - for (int i = 0; i < ARRAY_SIZE(phy_bars); i++) { - bar = &phy_bars[i]; - npu2_get_bar(gcid, bar); - npu2_write_bar(NULL, bar, gcid, scom_base); - } - reg[0] = phy_bars[2].base; - reg[1] = phy_bars[2].size; -} - /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { uint64_t stack = index_to_stack(dev->brick_index); - uint64_t offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? - NPU2_NTL0_BAR : NPU2_NTL1_BAR; uint64_t pa_offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; uint64_t reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - dev->ntl_bar.type = NPU_OCAPI_MMIO; - dev->ntl_bar.index = dev->brick_index; - dev->ntl_bar.reg = NPU2_REG_OFFSET(stack, 0, offset); dev->ntl_bar.enabled = true; - npu2_get_bar(gcid, &dev->ntl_bar); - - prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", - dev->ntl_bar.base, dev->ntl_bar.size); - npu2_write_bar(NULL, &dev->ntl_bar, gcid, scom_base); + npu2_write_bar(dev->npu, &dev->ntl_bar, gcid, scom_base); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->ntl_bar.base >> 16); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->ntl_bar.size >> 16)); @@ -788,20 +754,12 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, } /* Procedure 13.1.3.9 - AFU Config BARs */ -static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, - struct npu2_dev *dev) +static void setup_afu_config_bars(struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->brick_index); - int stack_num = stack - NPU2_STACK_STCK_0; - prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); - dev->genid_bar.type = NPU_GENID; - dev->genid_bar.index = stack_num; - dev->genid_bar.reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); dev->genid_bar.enabled = true; - npu2_get_bar(gcid, &dev->genid_bar); - prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->genid_bar.base); - npu2_write_bar(NULL, &dev->genid_bar, gcid, scom_base); + npu2_write_bar(dev->npu, &dev->genid_bar, dev->npu->chip_id, + dev->npu->xscom_base); } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, @@ -1580,8 +1538,8 @@ static void setup_device(struct npu2_dev *dev) uint64_t mm_win[2]; /* Populate PHB device node */ - phys_map_get(dev->npu->chip_id, NPU_OCAPI_MMIO, dev->brick_index, &mm_win[0], - &mm_win[1]); + mm_win[0] = dev->ntl_bar.base; + mm_win[1] = dev->ntl_bar.size; prlog(PR_DEBUG, "OCAPI: Setting MMIO window to %016llx + %016llx\n", mm_win[0], mm_win[1]); dn_phb = dt_new_addr(dt_root, "pciex", mm_win[0]); @@ -1627,7 +1585,7 @@ static void setup_device(struct npu2_dev *dev) /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ setup_afu_mmio_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); /* Procedure 13.1.3.9 - AFU Config BARs */ - setup_afu_config_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); + setup_afu_config_bars(dev); set_fence_control(dev->npu->chip_id, dev->npu->xscom_base, dev->brick_index, 0b00); @@ -1668,7 +1626,6 @@ static void read_nvram_training_state(void) int npu2_opencapi_init_npu(struct npu2 *npu) { struct npu2_dev *dev; - uint64_t reg[2]; int rc; assert(platform.ocapi); @@ -1677,10 +1634,6 @@ int npu2_opencapi_init_npu(struct npu2 *npu) /* TODO: Test OpenCAPI with fast reboot and make it work */ disable_fast_reboot("OpenCAPI device enabled"); - setup_global_mmio_bar(npu->chip_id, npu->xscom_base, reg); - - npu->regs = (void *)reg[0]; - for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; if (dev->type != NPU2_DEV_TYPE_OPENCAPI) diff --git a/hw/npu2.c b/hw/npu2.c index 60619f381301..b920edef3265 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -139,7 +139,7 @@ static int64_t npu2_cfg_write_cmd(void *dev, * track the enables separately. */ if (NPU2DEV_BRICK(ndev)) - ndev->genid_bar.enabled1 = enabled; + ndev->genid_bar.enabled1 = enabled; // TODO FIXME TABS GAHHHHH else ndev->genid_bar.enabled0 = enabled; @@ -1314,59 +1314,6 @@ static const struct phb_ops npu_ops = { .tce_kill = npu2_tce_kill, }; -static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint64_t mm_win[2]) -{ - uint32_t i; - struct npu2_bar *bar; - struct npu2_bar npu2_bars[] = { - /* NPU_REGS must be first in this list */ - { .type = NPU_REGS, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_NTL, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL1_BAR) }, - { .type = NPU_NTL, .index = 2, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 3, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_NTL1_BAR) }, - { .type = NPU_NTL, .index = 4, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 5, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_NTL1_BAR) }, - { .type = NPU_GENID, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_GENID_BAR) }, - { .type = NPU_GENID, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_GENID_BAR) }, - { .type = NPU_GENID, .index = 2, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_GENID_BAR) }, - }; - - for (i = 0; i < ARRAY_SIZE(npu2_bars); i++) { - bar = &npu2_bars[i]; - npu2_get_bar(gcid, bar); - npu2_write_bar(NULL, bar, gcid, scom); - } - - /* Global MMIO BAR */ - reg[0] = npu2_bars[0].base; - reg[1] = npu2_bars[0].size; - - /* NTL and GENID BARs are exposed to kernel via the mm - * window */ - mm_win[0] = npu2_bars[3].base; - mm_win[1] = npu2_bars[ARRAY_SIZE(npu2_bars) - 1].base + - npu2_bars[ARRAY_SIZE(npu2_bars) - 1].size - - mm_win[0]; -} - /* * Set up NPU for NVLink and create PCI root device node * accordingly. @@ -1428,19 +1375,9 @@ int npu2_nvlink_init_npu(struct npu2 *npu) xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); - /* Reassign the BARs */ - assign_mmio_bars(npu->chip_id, npu->xscom_base, reg, mm_win); - npu->regs = (void *)reg[0]; - npu->mm_base = mm_win[0]; - npu->mm_size = mm_win[1]; - - if (reg[0] && reg[1]) - prlog(PR_INFO, " Global MMIO BAR: %016llx (%lldMB)\n", - reg[0], reg[1] >> 20); - else - prlog(PR_ERR, " Global MMIO BAR: Disabled\n"); - /* Populate PCI root device node */ + reg[0] = (uint64_t)npu->regs; + reg[1] = npu->regs_size; np = dt_new_addr(dt_root, "pciex", reg[0]); assert(np); dt_add_property_strings(np, @@ -1455,6 +1392,8 @@ int npu2_nvlink_init_npu(struct npu2 *npu) dt_add_property_cells(np, "ibm,xscom-base", npu->xscom_base); dt_add_property_cells(np, "ibm,npcq", npu->dt_node->phandle); dt_add_property_cells(np, "ibm,links", npu->total_devices); + mm_win[0] = npu->mm_base; + mm_win[1] = npu->mm_size; dt_add_property(np, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(np, "ibm,phb-diag-data-size", 0); @@ -1679,7 +1618,6 @@ static void npu2_populate_devices(struct npu2 *p, struct npu2_dev *dev; struct dt_node *npu2_dn, *link; uint32_t npu_phandle, index = 0; - int stack; /* * Get the npu node which has the links which we expand here @@ -1713,32 +1651,6 @@ static void npu2_populate_devices(struct npu2 *p, dev->pl_xscom_base = dt_prop_get_u64(link, "ibm,npu-phy"); dev->lane_mask = dt_prop_get_u32(link, "ibm,npu-lane-mask"); - /* Populate BARs. BAR0/1 is the NTL bar. */ - stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); - dev->ntl_bar.type = NPU_NTL; - dev->ntl_bar.index = dev->brick_index; - dev->ntl_bar.reg = NPU2_REG_OFFSET(stack, 0, - NPU2DEV_BRICK(dev) == 0 ? - NPU2_NTL0_BAR : - NPU2_NTL1_BAR); - npu2_get_bar(p->chip_id, &dev->ntl_bar); - - dev->ntl_bar.flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; - - /* BAR2/3 is the GENID bar. */ - dev->genid_bar.type = NPU_GENID; - dev->genid_bar.index = NPU2DEV_STACK(dev); - dev->genid_bar.reg = NPU2_REG_OFFSET(stack, 0, - NPU2_GENID_BAR); - npu2_get_bar(p->chip_id, &dev->genid_bar); - - /* The GENID is a single physical BAR that we split - * for each emulated device */ - dev->genid_bar.size = 0x10000; - if (NPU2DEV_BRICK(dev)) - dev->genid_bar.base += 0x10000; - dev->genid_bar.flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; - /* Initialize PCI virtual device */ dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); if (dev->nvlink.pvd) diff --git a/include/npu2.h b/include/npu2.h index c2ba4370c063..31d9a4a1053f 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -147,6 +147,7 @@ struct npu2 { uint32_t chip_id; uint64_t xscom_base; void *regs; + uint64_t regs_size; uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; From patchwork Fri Jan 11 01:09:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023316 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPvM4KcWz9sCh for ; Fri, 11 Jan 2019 12:11:11 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPvM3DxyzDqwm for ; Fri, 11 Jan 2019 12:11:11 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPst5DRrzDqwP for ; Fri, 11 Jan 2019 12:09:54 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14Ibi091634 for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxd32rke9-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:48 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19lQ38913282 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:47 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C4607AE045; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2AF6AAE04D; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id D782FA03B0; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:25 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0016-0000-0000-00000243A7EC X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0017-0000-0000-0000329DAF0B Message-Id: <6064e343ca55b146c3ba3f7c213209c8a87bdc9b.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 05/15] hw/npu2: Simplify npu2_write_bar() X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Now that we've moved most of the BAR assignment code into common code and we have an existing struct npu2 everywhere we need it, we don't need the gcid and scom parameters to npu2_write_bar() any more. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Reviewed-by: Alexey Kardashevskiy --- v1->v2: - afu config bar enablement is in separate function --- hw/npu2-common.c | 14 +++++--------- hw/npu2-opencapi.c | 5 ++--- hw/npu2.c | 6 +++--- include/npu2.h | 3 +-- 4 files changed, 11 insertions(+), 17 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 944719d2489a..90c7f0ac4d5d 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -141,8 +141,7 @@ void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) } } -void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, - uint32_t scom) +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar) { uint64_t reg, val; int block; @@ -168,10 +167,7 @@ void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { reg = NPU2_REG_OFFSET(0, block, bar->reg); - if (p) - npu2_write(p, reg, val); - else - npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); + npu2_write(p, reg, val); } } @@ -200,7 +196,7 @@ static void assign_bars(struct npu2 *npu) for (i = 0; i < ARRAY_SIZE(phy_bars); i++) { bar = &phy_bars[i]; npu2_get_bar(npu->chip_id, bar); - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); } /* Device BARs */ @@ -221,7 +217,7 @@ static void assign_bars(struct npu2 *npu) NPU2_NTL0_BAR : NPU2_NTL1_BAR); bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; npu2_get_bar(npu->chip_id, bar); - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); /* GENID BAR */ bar = &dev->genid_bar; @@ -236,7 +232,7 @@ static void assign_bars(struct npu2 *npu) if (NPU2DEV_BRICK(dev)) bar->base += 0x10000; } - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); }; /* Global MMIO BAR */ diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 17a16be00eba..2a319979eccc 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -742,7 +742,7 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); dev->ntl_bar.enabled = true; - npu2_write_bar(dev->npu, &dev->ntl_bar, gcid, scom_base); + npu2_write_bar(dev->npu, &dev->ntl_bar); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->ntl_bar.base >> 16); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->ntl_bar.size >> 16)); @@ -758,8 +758,7 @@ static void setup_afu_config_bars(struct npu2_dev *dev) { prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); dev->genid_bar.enabled = true; - npu2_write_bar(dev->npu, &dev->genid_bar, dev->npu->chip_id, - dev->npu->xscom_base); + npu2_write_bar(dev->npu, &dev->genid_bar); } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, diff --git a/hw/npu2.c b/hw/npu2.c index b920edef3265..1c7af14958e8 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -131,7 +131,7 @@ static int64_t npu2_cfg_write_cmd(void *dev, enabled = !!(*data & PCI_CFG_CMD_MEM_EN); ndev->ntl_bar.enabled = enabled; - npu2_write_bar(ndev->npu, &ndev->ntl_bar, 0, 0); + npu2_write_bar(ndev->npu, &ndev->ntl_bar); /* * Enable/disable the GENID BAR. Two bricks share one GENID @@ -146,7 +146,7 @@ static int64_t npu2_cfg_write_cmd(void *dev, /* Enable the BAR if either device requests it enabled, otherwise disable it */ ndev->genid_bar.enabled = ndev->genid_bar.enabled0 || ndev->genid_bar.enabled1; - npu2_write_bar(ndev->npu, &ndev->genid_bar, 0, 0); + npu2_write_bar(ndev->npu, &ndev->genid_bar); return OPAL_PARTIAL; } @@ -216,7 +216,7 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, return OPAL_HARDWARE; } - npu2_write_bar(dev->npu, bar, 0, 0); + npu2_write_bar(dev->npu, bar); } /* To update the config cache */ diff --git a/include/npu2.h b/include/npu2.h index 31d9a4a1053f..8d739d8d5659 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -208,8 +208,7 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask); void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar); void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar); -void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, - uint32_t scom); +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar); int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); From patchwork Fri Jan 11 01:09:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023314 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPtn2BQZz9sCh for ; Fri, 11 Jan 2019 12:10:41 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPtn19pCzDqxH for ; Fri, 11 Jan 2019 12:10:41 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPst6CPBzDqwh for ; Fri, 11 Jan 2019 12:09:54 +1100 (AEDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14D5C144845 for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pxfbjktdg-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:48 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19luM64225366 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:47 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C5340A405C; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2C7CCA405B; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id DD305A03B1; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:26 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0012-0000-0000-000002E6349C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0013-0000-0000-0000211D3CAC Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=960 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 06/15] hw/npu2: Don't repopulate NPU devices in NVLink init path X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" In 68415d5e38ef ("hw/npu2: Common NPU2 init routine between NVLink and OpenCAPI") we refactored a large chunk of the NPU init path into common code, including walking the device tree to setup npu2_devs based on the ibm,npu-link device tree nodes. We didn't actually remove the code that does that in npu2_populate_devices(), so currently we populate the devices in the common setup path, then repopulate them incorrectly in the NVLink setup path. Currently this is fine, because we don't support having both NVLink and OpenCAPI devices on the same NPU, but when we do, this will be a problem. Fix npu2_populate_devices() to only populate additional fields as required for NVLink devices. Rename it to npu2_configure_devices() to better reflect what it now does. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - remove unneeded scan_map assignment, it's zalloced anyway (Alexey) - rebase on Reza's cleanups --- hw/npu2.c | 42 ++++++++++-------------------------------- 1 file changed, 10 insertions(+), 32 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 1c7af14958e8..106b32150994 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1599,12 +1599,12 @@ static void npu2_populate_cfg(struct npu2_dev *dev) PCI_VIRT_CFG_INIT_RO(pvd, pos + 1, 1, 0); } -static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group) +static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group, int size) { int i; int bdfn = (group << 3); - for (i = 0; i < p->total_devices; i++) { + for (i = 0; i < size; i++) { if ((p->devices[i].bdfn & 0xf8) == (bdfn & 0xf8)) bdfn++; } @@ -1612,51 +1612,29 @@ static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group) return bdfn; } -static void npu2_populate_devices(struct npu2 *p, - struct dt_node *dn) +static void npu2_configure_devices(struct npu2 *p) { struct npu2_dev *dev; - struct dt_node *npu2_dn, *link; - uint32_t npu_phandle, index = 0; - - /* - * Get the npu node which has the links which we expand here - * into pci like devices attached to our emulated phb. - */ - npu_phandle = dt_prop_get_u32(dn, "ibm,npcq"); - npu2_dn = dt_find_by_phandle(dt_root, npu_phandle); - assert(npu2_dn); + uint32_t index = 0; - /* Walk the link@x nodes to initialize devices */ - p->total_devices = 0; - p->phb_nvlink.scan_map = 0; - dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { + for (index = 0; index < p->total_devices; index++) { uint32_t group_id; dev = &p->devices[index]; - dev->type = NPU2_DEV_TYPE_NVLINK; - dev->npu = p; - dev->dt_node = link; - dev->link_index = dt_prop_get_u32(link, "ibm,npu-link-index"); - dev->brick_index = dev->link_index; + if (dev->type != NPU2_DEV_TYPE_NVLINK) + continue; - group_id = dt_prop_get_u32(link, "ibm,npu-group-id"); - dev->bdfn = npu_allocate_bdfn(p, group_id); + group_id = dt_prop_get_u32(dev->dt_node, "ibm,npu-group-id"); + dev->bdfn = npu_allocate_bdfn(p, group_id, index); /* This must be done after calling * npu_allocate_bdfn() */ - p->total_devices++; p->phb_nvlink.scan_map |= 0x1 << ((dev->bdfn & 0xf8) >> 3); - dev->pl_xscom_base = dt_prop_get_u64(link, "ibm,npu-phy"); - dev->lane_mask = dt_prop_get_u32(link, "ibm,npu-lane-mask"); - /* Initialize PCI virtual device */ dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); if (dev->nvlink.pvd) npu2_populate_cfg(dev); - - index++; } } @@ -1857,7 +1835,7 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) list_head_init(&npu->phb_nvlink.virt_devices); npu2_setup_irqs(npu); - npu2_populate_devices(npu, dn); + npu2_configure_devices(npu); npu2_add_interrupt_map(npu, dn); npu2_add_phb_properties(npu); From patchwork Fri Jan 11 01:09:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023313 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPtR3QFJz9sN1 for ; Fri, 11 Jan 2019 12:10:23 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPtR1dr2zDqwr for ; Fri, 11 Jan 2019 12:10:23 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPst5DStzDqwL for ; Fri, 11 Jan 2019 12:09:54 +1100 (AEDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14Dgo144851 for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pxfbjktdc-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:52 -0500 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mIj60555458 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 15E12AE059; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 713A8AE055; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id E2B40A03B2; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:27 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0012-0000-0000-000002E6349B X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0013-0000-0000-0000211D3CAD Message-Id: <2f690f27c339a44d166826f820eb4d3d25a6063f.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 07/15] hw/npu2: Rework npu2_add_interrupt_map() and skip non-NVLink devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Rework npu2_add_interrupt_map() so it only includes NVLink devices. Use the existing struct npu2_devs rather than accessing the device tree. Use each device's brick index to determine its interrupt number, as interrupt levels depend on the ODL number and thus the brick index rather than link index. Signed-off-by: Andrew Donnellan --- This is difficult to test as I don't know how to inject faults in NVLink. v1->v2: - add more info to commit message and comment about interrupt level number (Alexey) --- hw/npu2.c | 41 +++++++++++++++++++++++++---------------- 1 file changed, 25 insertions(+), 16 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 106b32150994..78233579d3b7 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1638,11 +1638,11 @@ static void npu2_configure_devices(struct npu2 *p) } } -static void npu2_add_interrupt_map(struct npu2 *p, - struct dt_node *dn) +static void npu2_add_interrupt_map(struct npu2 *p) { - struct dt_node *npu2_dn, *link, *phb_dn; - uint32_t npu2_phandle, index = 0, i; + struct dt_node *phb_dn; + struct npu2_dev *dev; + int index, i = 0, nv_devices = 0; uint32_t icsp = get_ics_phandle(); uint32_t *map; size_t map_size; @@ -1651,23 +1651,32 @@ static void npu2_add_interrupt_map(struct npu2 *p, assert(p->phb_nvlink.dt_node); phb_dn = p->phb_nvlink.dt_node; - npu2_phandle = dt_prop_get_u32(dn, "ibm,npcq"); - npu2_dn = dt_find_by_phandle(dt_root, npu2_phandle); - assert(npu2_dn); - map_size = 7 * sizeof(*map) * p->total_devices; + for (index = 0; index < p->total_devices; index++) { + if (p->devices[index].type == NPU2_DEV_TYPE_NVLINK) + nv_devices++; + } + + map_size = 7 * sizeof(*map) * nv_devices; map = malloc(map_size); - index = 0; - dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { - i = index * 7; - map[i + 0] = (p->devices[index].bdfn << 8); + + for (index = 0; index < p->total_devices; index++) { + dev = &p->devices[index]; + if (dev->type != NPU2_DEV_TYPE_NVLINK) + continue; + + map[i + 0] = (dev->bdfn << 8); map[i + 1] = 0; map[i + 2] = 0; - map[i + 3] = 1; /* INT A */ map[i + 4] = icsp; /* interrupt-parent */ - map[i + 5] = p->base_lsi + (index * 2) + 1; /* NDL No-Stall Event */ + + /* + * NDL No-Stall Event - the interrupt level is determined by + * the brick number, per the NPU workbook + */ + map[i + 5] = p->base_lsi + (dev->brick_index * 2) + 1; map[i + 6] = 0; /* 0 = EDGE, 1 = LEVEL. */ - index++; + i += 7; } dt_add_property(phb_dn, "interrupt-map", map, map_size); free(map); @@ -1836,7 +1845,7 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) npu2_setup_irqs(npu); npu2_configure_devices(npu); - npu2_add_interrupt_map(npu, dn); + npu2_add_interrupt_map(npu); npu2_add_phb_properties(npu); slot = npu2_slot_create(&npu->phb_nvlink); From patchwork Fri Jan 11 01:09:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023321 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPwk6HTYz9sCh for ; Fri, 11 Jan 2019 12:12:22 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPwk4vKJzDqwN for ; Fri, 11 Jan 2019 12:12:22 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsv6CG0zDqwh for ; Fri, 11 Jan 2019 12:09:55 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14FWP091360 for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxd32rkeq-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:51 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mo22621758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 17B2A42049; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7265F42041; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id E7A09A03B5; Fri, 11 Jan 2019 12:09:43 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:28 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0028-0000-0000-000003375C7E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0029-0000-0000-000023F4734D Message-Id: <73462a672811115afadcd984d921dda6fd71b91e.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 08/15] hw/npu2: Make IRQ setup code common X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Frederic Barrat The NPU IRQ setup code is currently duplicated between NVLink and OpenCAPI. Apart from the NPU2_MISC_IRQ_ENABLE2 register, the setup of IRQs is basically identical between NVLink and OpenCAPI. Move the NVLink IRQ setup code into the common path and get rid of all the OpenCAPI IRQ setup apart from the enable mask. We allocate a few more interrupts per NPU as OpenCAPI is a bit more greedy than NVLink (35 IRQs vs 23 previously). Signed-off-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- v1->v2: - updated commit message (Fred) Might need some testing --- hw/npu2-common.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 50 +-------------------- hw/npu2.c | 100 +----------------------------------------- include/npu2.h | 1 +- 4 files changed, 116 insertions(+), 147 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 90c7f0ac4d5d..282e4873125a 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -24,6 +24,12 @@ #include #include #include +#include +#include + +#define NPU2_IRQ_BASE_SHIFT 13 +#define NPU2_N_DL_IRQS 35 +#define NPU2_N_DL_IRQS_ALIGN 64 /* * We use the indirect method because it uses the same addresses as @@ -246,6 +252,111 @@ static void assign_bars(struct npu2 *npu) npu->mm_size = last_bar.base + last_bar.size - npu->mm_base; } +static uint64_t npu2_ipi_attributes(struct irq_source *is __unused, uint32_t isn __unused) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + + if (idx == 18) + /* TCE Interrupt - used to detect a frozen PE */ + return IRQ_ATTR_TARGET_OPAL | IRQ_ATTR_TARGET_RARE | IRQ_ATTR_TYPE_MSI; + else + return IRQ_ATTR_TARGET_LINUX; +} + +static char *npu2_ipi_name(struct irq_source *is, uint32_t isn) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + const char *name; + + switch (idx) { + case 0: name = "NDL 0 Stall Event (brick 0)"; break; + case 1: name = "NDL 0 No-Stall Event (brick 0)"; break; + case 2: name = "NDL 1 Stall Event (brick 1)"; break; + case 3: name = "NDL 1 No-Stall Event (brick 1)"; break; + case 4: name = "NDL 2 Stall Event (brick 2)"; break; + case 5: name = "NDL 2 No-Stall Event (brick 2)"; break; + case 6: name = "NDL 5 Stall Event (brick 3)"; break; + case 7: name = "NDL 5 No-Stall Event (brick 3)"; break; + case 8: name = "NDL 4 Stall Event (brick 4)"; break; + case 9: name = "NDL 4 No-Stall Event (brick 4)"; break; + case 10: name = "NDL 3 Stall Event (brick 5)"; break; + case 11: name = "NDL 3 No-Stall Event (brick 5)"; break; + case 12: name = "NTL 0 Event"; break; + case 13: name = "NTL 1 Event"; break; + case 14: name = "NTL 2 Event"; break; + case 15: name = "NTL 3 Event"; break; + case 16: name = "NTL 4 Event"; break; + case 17: name = "NTL 5 Event"; break; + case 18: name = "TCE Event"; break; + case 19: name = "ATS Event"; break; + case 20: name = "CQ Event"; break; + case 21: name = "MISC Event"; break; + case 22: name = "NMMU Local Xstop"; break; + case 23: name = "Translate Fail (brick 2)"; break; + case 24: name = "Translate Fail (brick 3)"; break; + case 25: name = "Translate Fail (brick 4)"; break; + case 26: name = "Translate Fail (brick 5)"; break; + case 27: name = "OTL Event (brick 2)"; break; + case 28: name = "OTL Event (brick 3)"; break; + case 29: name = "OTL Event (brick 4)"; break; + case 30: name = "OTL Event (brick 5)"; break; + case 31: name = "XSL Event (brick 2)"; break; + case 32: name = "XSL Event (brick 3)"; break; + case 33: name = "XSL Event (brick 4)"; break; + case 34: name = "XSL Event (brick 5)"; break; + default: name = "Unknown"; + } + return strdup(name); +} + +static void npu2_err_interrupt(struct irq_source *is, uint32_t isn) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + + if (idx != 18) { + prerror("OPAL received unknown NPU2 interrupt %d\n", idx); + return; + } + + opal_update_pending_evt(OPAL_EVENT_PCI_ERROR, + OPAL_EVENT_PCI_ERROR); +} + +static const struct irq_source_ops npu2_ipi_ops = { + .interrupt = npu2_err_interrupt, + .attributes = npu2_ipi_attributes, + .name = npu2_ipi_name, +}; + +static void setup_irqs(struct npu2 *p) +{ + uint64_t reg, val; + void *tp; + + p->base_lsi = xive_alloc_ipi_irqs(p->chip_id, NPU2_N_DL_IRQS, NPU2_N_DL_IRQS_ALIGN); + if (p->base_lsi == XIVE_IRQ_ERROR) { + prlog(PR_ERR, "NPU: Failed to allocate interrupt sources\n"); + return; + } + xive_register_ipi_source(p->base_lsi, NPU2_N_DL_IRQS, p, &npu2_ipi_ops); + + /* Set IPI configuration */ + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_CFG); + val = npu2_read(p, reg); + val = SETFIELD(NPU2_MISC_CFG_IPI_PS, val, NPU2_MISC_CFG_IPI_PS_64K); + val = SETFIELD(NPU2_MISC_CFG_IPI_OS, val, NPU2_MISC_CFG_IPI_OS_AIX); + npu2_write(p, reg, val); + + /* Set IRQ base */ + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_IRQ_BASE); + tp = xive_get_trigger_port(p->base_lsi); + val = ((uint64_t)tp) << NPU2_IRQ_BASE_SHIFT; + npu2_write(p, reg, val); +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; @@ -423,6 +534,7 @@ static void setup_devices(struct npu2 *npu) } assign_bars(npu); + setup_irqs(npu); if (nvlink_detected) npu2_nvlink_init_npu(npu); diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 2a319979eccc..1b1a06ff81ad 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -48,11 +48,9 @@ #include #include #include -#include #include #include -#define NPU_IRQ_LEVELS 35 #define NPU_IRQ_LEVELS_XSL 23 #define MAX_PE_HANDLE ((1 << 15) - 1) #define TL_MAX_TEMPLATE 63 @@ -1390,7 +1388,7 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, * Pass the hw irq number for the translation fault irq * irq levels 23 -> 26 are for translation faults, 1 per brick */ - irq = dev->npu->irq_base + NPU_IRQ_LEVELS_XSL; + irq = dev->npu->base_lsi + NPU_IRQ_LEVELS_XSL; if (stacku == NPU2_STACK_STCK_2U) irq += 2; if (block == NPU2_BLOCK_OTL1) @@ -1463,43 +1461,9 @@ static void mask_nvlink_fir(struct npu2 *p) NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); } -static int setup_irq(struct npu2 *p) +static int enable_xsl_irq(struct npu2 *p) { - uint64_t reg, mmio_addr; - uint32_t base; - - base = xive_alloc_ipi_irqs(p->chip_id, NPU_IRQ_LEVELS, 64); - if (base == XIVE_IRQ_ERROR) { - /** - * @fwts-label OCAPIIRQAllocationFailed - * @fwts-advice OpenCAPI IRQ setup failed. This is probably - * a firmware bug. OpenCAPI functionality will be broken. - */ - prlog(PR_ERR, "OCAPI: Couldn't allocate interrupts for NPU\n"); - return -1; - } - p->irq_base = base; - - xive_register_ipi_source(base, NPU_IRQ_LEVELS, NULL, NULL); - mmio_addr = (uint64_t ) xive_get_trigger_port(base); - prlog(PR_DEBUG, "OCAPI: NPU base irq %d @%llx\n", base, mmio_addr); - reg = (mmio_addr & NPU2_MISC_IRQ_BASE_MASK) << 13; - npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_BASE, - NPU2_MISC_DA_LEN_8B, reg); - /* - * setup page size = 64k - * - * OS type is set to AIX: opal also runs with 2 pages per interrupt, - * so to cover the max offset for 35 levels of interrupt, we need - * bits 41 to 46, which is what the AIX setting does. There's no - * other meaning for that AIX setting. - */ - reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_CFG, - NPU2_MISC_DA_LEN_8B); - reg |= NPU2_MISC_CFG_IPI_PS; - reg &= ~NPU2_MISC_CFG_IPI_OS; - npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_CFG, - NPU2_MISC_DA_LEN_8B, reg); + uint64_t reg; /* enable translation interrupts for all bricks */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE2, @@ -1625,7 +1589,6 @@ static void read_nvram_training_state(void) int npu2_opencapi_init_npu(struct npu2 *npu) { struct npu2_dev *dev; - int rc; assert(platform.ocapi); read_nvram_training_state(); @@ -1651,10 +1614,7 @@ int npu2_opencapi_init_npu(struct npu2 *npu) address_translation_config(npu->chip_id, npu->xscom_base, dev->brick_index); } - /* Procedure 13.1.3.10 - Interrupt Configuration */ - rc = setup_irq(npu); - if (rc) - goto failed; + enable_xsl_irq(npu); for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; @@ -1664,8 +1624,6 @@ int npu2_opencapi_init_npu(struct npu2 *npu) } return 0; -failed: - return -1; } static const struct phb_ops npu2_opencapi_ops = { diff --git a/hw/npu2.c b/hw/npu2.c index 78233579d3b7..6c720764e614 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include #include @@ -36,14 +35,9 @@ #include #include #include -#include #include #include -#define NPU2_IRQ_BASE_SHIFT 13 -#define NPU2_N_DL_IRQS 23 -#define NPU2_N_DL_IRQS_ALIGN 64 - #define VENDOR_CAP_START 0x80 #define VENDOR_CAP_END 0x90 #define VENDOR_CAP_LEN 0x10 @@ -1737,99 +1731,6 @@ static void npu2_add_phb_properties(struct npu2 *p) hi32(mm_size), lo32(mm_size)); } -static uint64_t npu2_ipi_attributes(struct irq_source *is __unused, uint32_t isn __unused) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - - if (idx == 18) - /* TCE Interrupt - used to detect a frozen PE */ - return IRQ_ATTR_TARGET_OPAL | IRQ_ATTR_TARGET_RARE | IRQ_ATTR_TYPE_MSI; - else - return IRQ_ATTR_TARGET_LINUX; -} - -static char *npu2_ipi_name(struct irq_source *is, uint32_t isn) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - const char *name; - - switch (idx) { - case 0: name = "NDL 0 Stall Event (brick 0)"; break; - case 1: name = "NDL 0 No-Stall Event (brick 0)"; break; - case 2: name = "NDL 1 Stall Event (brick 1)"; break; - case 3: name = "NDL 1 No-Stall Event (brick 1)"; break; - case 4: name = "NDL 2 Stall Event (brick 2)"; break; - case 5: name = "NDL 2 No-Stall Event (brick 2)"; break; - case 6: name = "NDL 5 Stall Event (brick 3)"; break; - case 7: name = "NDL 5 No-Stall Event (brick 3)"; break; - case 8: name = "NDL 4 Stall Event (brick 4)"; break; - case 9: name = "NDL 4 No-Stall Event (brick 4)"; break; - case 10: name = "NDL 3 Stall Event (brick 5)"; break; - case 11: name = "NDL 3 No-Stall Event (brick 5)"; break; - case 12: name = "NTL 0 Event"; break; - case 13: name = "NTL 1 Event"; break; - case 14: name = "NTL 2 Event"; break; - case 15: name = "NTL 3 Event"; break; - case 16: name = "NTL 4 Event"; break; - case 17: name = "NTL 5 Event"; break; - case 18: name = "TCE Event"; break; - case 19: name = "ATS Event"; break; - case 20: name = "CQ Event"; break; - case 21: name = "MISC Event"; break; - case 22: name = "NMMU Local Xstop"; break; - default: name = "Unknown"; - } - return strdup(name); -} - -static void npu2_err_interrupt(struct irq_source *is, uint32_t isn) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - - if (idx != 18) { - prerror("OPAL received unknown NPU2 interrupt %d\n", idx); - return; - } - - opal_update_pending_evt(OPAL_EVENT_PCI_ERROR, - OPAL_EVENT_PCI_ERROR); -} - -static const struct irq_source_ops npu2_ipi_ops = { - .interrupt = npu2_err_interrupt, - .attributes = npu2_ipi_attributes, - .name = npu2_ipi_name, -}; - -static void npu2_setup_irqs(struct npu2 *p) -{ - uint64_t reg, val; - void *tp; - - p->base_lsi = xive_alloc_ipi_irqs(p->chip_id, NPU2_N_DL_IRQS, NPU2_N_DL_IRQS_ALIGN); - if (p->base_lsi == XIVE_IRQ_ERROR) { - prlog(PR_ERR, "NPU: Failed to allocate interrupt sources, IRQs for NDL No-stall events will not be available.\n"); - return; - } - xive_register_ipi_source(p->base_lsi, NPU2_N_DL_IRQS, p, &npu2_ipi_ops ); - - /* Set IPI configuration */ - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_CFG); - val = npu2_read(p, reg); - val = SETFIELD(NPU2_MISC_CFG_IPI_PS, val, NPU2_MISC_CFG_IPI_PS_64K); - val = SETFIELD(NPU2_MISC_CFG_IPI_OS, val, NPU2_MISC_CFG_IPI_OS_AIX); - npu2_write(p, reg, val); - - /* Set IRQ base */ - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_IRQ_BASE); - tp = xive_get_trigger_port(p->base_lsi); - val = ((uint64_t)tp) << NPU2_IRQ_BASE_SHIFT; - npu2_write(p, reg, val); -} - void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) { struct pci_slot *slot; @@ -1843,7 +1744,6 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) list_head_init(&npu->phb_nvlink.devices); list_head_init(&npu->phb_nvlink.virt_devices); - npu2_setup_irqs(npu); npu2_configure_devices(npu); npu2_add_interrupt_map(npu); npu2_add_phb_properties(npu); diff --git a/include/npu2.h b/include/npu2.h index 8d739d8d5659..75c11ff3c26f 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -151,7 +151,6 @@ struct npu2 { uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; - uint32_t irq_base; uint32_t total_devices; struct npu2_dev *devices; enum phys_map_type gpu_map_type; From patchwork Fri Jan 11 01:09:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023324 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPxV5wnNz9sCh for ; Fri, 11 Jan 2019 12:13:02 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPxV4tvxzDqwv for ; Fri, 11 Jan 2019 12:13:02 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsw5q0tzDqwp for ; Fri, 11 Jan 2019 12:09:56 +1100 (AEDT) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14G8d085073 for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxbw23ps7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:52 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mHr58654796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1B885AE045; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7561BAE058; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 07C15A03B6; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:29 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0028-0000-0000-000003375C81 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0029-0000-0000-000023F4734E Message-Id: <38fdb9d95fc9a27b2caf4836bfe760b8f11ee7f9.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 09/15] platforms/astbmc/witherspoon: Rework NPU presence detection X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Rework NPU presence detection in preparation for supporting both NVLink and OpenCAPI devices operating simultaneously on the same NPU. If an OpenCAPI card is connected to GPU#0, and an NVLink GPU is connected to GPU#1, the GPU will only receive 2 links rather than the usual 3. The reason for this is that without the OpenCAPI card, the GPU would be use links 3-5, connected to NPU bricks 3-5, which needs both stacks 1 and 2 to be in NVLink mode. However, with an OpenCAPI card in the GPU#0 slot that uses links 0-1, we need to use NPU bricks 2-3, which means stack 1 must be set in OpenCAPI mode. As such, the GPU will be restricted to using links 4 and 5. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- platforms/astbmc/witherspoon.c | 62 ++++++++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 15 deletions(-) diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index fe138991696f..c41f0c5b1971 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -233,6 +233,8 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) int rc; bool gpu0_present, gpu1_present; + enum npu2_dev_type gpu0_type = NPU2_DEV_TYPE_UNKNOWN; + enum npu2_dev_type gpu1_type = NPU2_DEV_TYPE_UNKNOWN; if (witherspoon_type != WITHERSPOON_TYPE_REDBUD) { prlog(PR_DEBUG, "PLAT: Setting all NPU links to NVLink, OpenCAPI only supported on Redbud\n"); @@ -298,19 +300,11 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) if (state & (1 << 0)) { prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is OpenCAPI\n", chip->id); - /* - * On witherspoon, bricks 2 and 3 are connected to - * the lanes matching links 0 and 1 in OpenCAPI mode. - */ - set_link_details(npu, 1, 3, NPU2_DEV_TYPE_OPENCAPI); - /* We current don't support using the second link */ - set_link_details(npu, 0, 2, NPU2_DEV_TYPE_UNKNOWN); + gpu0_type = NPU2_DEV_TYPE_OPENCAPI; } else { prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is NVLink\n", chip->id); - set_link_details(npu, 0, 0, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 1, 1, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 2, 2, NPU2_DEV_TYPE_NVLINK); + gpu0_type = NPU2_DEV_TYPE_NVLINK; } } @@ -318,16 +312,54 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) if (state & (1 << 1)) { prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is OpenCAPI\n", chip->id); - set_link_details(npu, 4, 4, NPU2_DEV_TYPE_OPENCAPI); - /* We current don't support using the second link */ - set_link_details(npu, 5, 5, NPU2_DEV_TYPE_UNKNOWN); + gpu1_type = NPU2_DEV_TYPE_OPENCAPI; } else { prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is NVLink\n", chip->id); + gpu1_type = NPU2_DEV_TYPE_NVLINK; + } + } + + if (gpu0_type == NPU2_DEV_TYPE_OPENCAPI) { + set_link_details(npu, 1, 3, NPU2_DEV_TYPE_OPENCAPI); + /* We currently don't support using the second link */ + set_link_details(npu, 0, 2, NPU2_DEV_TYPE_UNKNOWN); + } + + if (gpu0_type == NPU2_DEV_TYPE_NVLINK) { + set_link_details(npu, 0, 0, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 1, 1, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 2, 2, NPU2_DEV_TYPE_NVLINK); + } + + if (gpu1_type == NPU2_DEV_TYPE_OPENCAPI) { + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_OPENCAPI); + /* We currently don't support using the second link */ + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_UNKNOWN); + } + + /* + * If an OpenCAPI card is connected to GPU#0, and an NVLink GPU is + * connected to GPU#1, the GPU will only receive 2 links rather than the + * usual 3. + * + * The reason for this is that without the OpenCAPI card, the GPU would + * be use links 3-5, connected to NPU bricks 3-5, which needs both + * stacks 1 and 2 to be in NVLink mode. + * + * However, with an OpenCAPI card in the GPU#0 slot that uses links 0-1, + * we need to use NPU bricks 2-3, which means stack 1 must be set in + * OpenCAPI mode. As such, the GPU will be restricted to using links 4 + * and 5. + */ + if (gpu1_type == NPU2_DEV_TYPE_NVLINK) { + if (gpu0_type == NPU2_DEV_TYPE_OPENCAPI) { + prlog(PR_WARNING, "PLAT: Chip %d GPU#1 will operate at reduced performance due to presence of OpenCAPI device. For optimal performance, swap device locations\n", chip->id); + } else { set_link_details(npu, 3, 3, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 4, 4, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 5, 5, NPU2_DEV_TYPE_NVLINK); } + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_NVLINK); } return; From patchwork Fri Jan 11 01:09:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023327 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPyM559Pz9sCh for ; Fri, 11 Jan 2019 12:13:47 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPyM3qFpzDr0V for ; Fri, 11 Jan 2019 12:13:47 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsy6chQzDqwm for ; Fri, 11 Jan 2019 12:09:58 +1100 (AEDT) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14KMl061788 for ; Thu, 10 Jan 2019 20:09:55 -0500 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxbxxukpn-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:55 -0500 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:50 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mE739452758 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 60235AE045; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 740F6AE057; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 165CEA03BA; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:30 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0008-0000-0000-000002AF363E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0009-0000-0000-0000221B3E96 Message-Id: <1029ee5451697c8a14fdf29710543998582c21fe.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 10/15] hw/npu2: Combined NPU brick configuration procedure X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" To support the case where we have both OpenCAPI and NVLink devices connected to the same NPU, we need to be able to configure each NPU stack/brick separately depending on the device type it is connected to. Refactor the existing code so that we can set the transport muxes and other relevant NPU registers on a per-brick/stack basis. Replace the existing hard-coded register values for NVLink setup with code to do it properly only on the devices that need it. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - commit message (Fred) --- hw/npu2-common.c | 126 +++++++++++++++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 109 +--------------------------------------- hw/npu2.c | 121 +++++++++++++++++++++++-------------------- include/npu2-regs.h | 4 +- include/npu2.h | 1 +- 5 files changed, 196 insertions(+), 165 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 282e4873125a..e323e71cb8d5 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -411,6 +411,130 @@ void npu2_i2c_presence_detect(struct npu2 *npu) } } +/* Procedure 13.1.3.1, OpenCAPI NPU Workbook - Select OCAPI vs NVLink */ +static void set_brick_config(struct npu2 *npu) +{ + /* Step 1 - Set Transport MUX controls to select correct OTL or NTL */ + uint64_t reg; + uint64_t ndlmux_brk0to2, ocmux_brk0to1, ocmux_brk4to5; + uint64_t phy_config_scom; + struct npu2_dev *dev; + + prlog(PR_DEBUG, "NPU: %s: Setting transport mux controls\n", __func__); + + /* + * Disable NVLink link layers on PHYs being used for OpenCAPI. + * Experience suggests that this needs to be done early + * (before setting the transport muxes) otherwise we get link + * internal errors. + */ + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + switch (dev->brick_index) { + case 2: + case 3: + phy_config_scom = OBUS_LL0_IOOL_PHY_CONFIG; + break; + case 4: + case 5: + phy_config_scom = OBUS_LL3_IOOL_PHY_CONFIG; + break; + default: + assert(false); + } + /* Disable NV-Link link layers */ + xscom_read(npu->chip_id, phy_config_scom, ®); + reg &= ~OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED; + xscom_write(npu->chip_id, phy_config_scom, reg); + } + + /* Optical IO Transport Mux Config for Bricks 0-2 and 4-5 */ + reg = npu2_scom_read(npu->chip_id, npu->xscom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B); + ndlmux_brk0to2 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); + ocmux_brk0to1 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); + ocmux_brk4to5 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type == NPU2_DEV_TYPE_UNKNOWN) + continue; + + switch (dev->brick_index) { + case 0: /* NTL0.0 */ + assert(dev->type == NPU2_DEV_TYPE_NVLINK); + ndlmux_brk0to2 |= 0b100; + break; + case 1: /* NTL0.1 */ + assert(dev->type == NPU2_DEV_TYPE_NVLINK); + ndlmux_brk0to2 |= 0b010; + break; + case 2: /* NTL1.0 / OTL1.0 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ndlmux_brk0to2 &= ~0b100; + ocmux_brk0to1 |= 0b10; + } else { + ndlmux_brk0to2 |= 0b001; + } + break; + case 3: /* NTL1.1 / OTL1.1 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ndlmux_brk0to2 &= ~0b010; + ocmux_brk0to1 |= 0b01; + } + break; + case 4: /* NTL2.0 / OTL2.0 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ocmux_brk4to5 |= 0b10; + } else { + ocmux_brk4to5 &= ~0b10; + } + break; + case 5: /* NTL2.1 / OTL2.1 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ocmux_brk4to5 |= 0b01; + } else { + ocmux_brk4to5 &= ~0b01; + } + break; + default: + assert(false); + } + } + + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, ndlmux_brk0to2); + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, ocmux_brk0to1); + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, ocmux_brk4to5); + npu2_scom_write(npu->chip_id, npu->xscom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B, reg); + + /* + * PowerBus Optical Miscellaneous Config Register + */ + xscom_read(npu->chip_id, PU_IOE_PB_MISC_CFG, ®); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + switch (dev->brick_index) { + case 3: + if (dev->type == NPU2_DEV_TYPE_NVLINK) + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_03_NPU_NOT_PB, reg, 1); + break; + case 4: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB, reg, 1); + break; + case 5: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB, reg, 1); + break; + default: + break; + } + } + xscom_write(npu->chip_id, PU_IOE_PB_MISC_CFG, reg); +} + static struct npu2 *setup_npu(struct dt_node *dn) { struct npu2 *npu; @@ -471,6 +595,7 @@ static struct npu2 *setup_npu(struct dt_node *dn) dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); /* May be overridden by platform presence detection */ dev->brick_index = dev->link_index; + dev->group_id = dt_prop_get_u32(np, "ibm,npu-group-id"); /* Will be overridden by presence detection */ dev->type = NPU2_DEV_TYPE_UNKNOWN; dev->npu = npu; @@ -575,6 +700,7 @@ void probe_npu2(void) if (!npu) continue; platform.npu2_device_detect(npu); + set_brick_config(npu); setup_devices(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 1b1a06ff81ad..d3e95a45e9a4 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -196,112 +196,6 @@ static uint64_t get_odl_endpoint_info(uint32_t gcid, uint64_t index) return reg; } -static void disable_nvlink(uint32_t gcid, int index) -{ - uint64_t phy_config_scom, reg; - - switch (index) { - case 2: - case 3: - phy_config_scom = OBUS_LL0_IOOL_PHY_CONFIG; - break; - case 4: - case 5: - phy_config_scom = OBUS_LL3_IOOL_PHY_CONFIG; - break; - default: - assert(false); - } - /* Disable NV-Link link layers */ - xscom_read(gcid, phy_config_scom, ®); - reg &= ~OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED; - reg &= ~OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED; - reg &= ~OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED; - xscom_write(gcid, phy_config_scom, reg); -} - -/* Procedure 13.1.3.1 - select OCAPI vs NVLink for bricks 2-3/4-5 */ - -static void set_transport_mux_controls(uint32_t gcid, uint32_t scom_base, - int index, enum npu2_dev_type type) -{ - /* Step 1 - Set Transport MUX controls to select correct OTL or NTL */ - uint64_t reg; - uint64_t field; - - /* TODO: Rework this to select for NVLink too */ - assert(type == NPU2_DEV_TYPE_OPENCAPI); - - prlog(PR_DEBUG, "OCAPI: %s: Setting transport mux controls\n", __func__); - - /* Optical IO Transport Mux Config for Bricks 0-2 and 4-5 */ - reg = npu2_scom_read(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, - NPU2_MISC_DA_LEN_8B); - switch (index) { - case 0: - case 1: - /* not valid for OpenCAPI */ - assert(false); - break; - case 2: /* OTL1.0 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); - field &= ~0b100; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, - field); - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); - field |= 0b10; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, - field); - break; - case 3: /* OTL1.1 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); - field &= ~0b010; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, - field); - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); - field |= 0b01; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, - field); - break; - case 4: /* OTL2.0 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); - field |= 0b10; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, - field); - break; - case 5: /* OTL2.1 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); - field |= 0b01; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, - field); - break; - default: - assert(false); - } - npu2_scom_write(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, - NPU2_MISC_DA_LEN_8B, reg); - - /* - * PowerBus Optical Miscellaneous Config Register - select - * OpenCAPI for b4/5 and A-Link for b3 - */ - xscom_read(gcid, PU_IOE_PB_MISC_CFG, ®); - switch (index) { - case 0: - case 1: - case 2: - case 3: - break; - case 4: - reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB, reg, 1); - break; - case 5: - reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB, reg, 1); - break; - } - xscom_write(gcid, PU_IOE_PB_MISC_CFG, reg); -} - static void enable_odl_phy_mux(uint32_t gcid, int index) { uint64_t reg; @@ -579,9 +473,6 @@ static void brick_config(uint32_t gcid, uint32_t scom_base, int index) * We assume at this point that the PowerBus Hotplug Mode Control * register is correctly set by Hostboot */ - disable_nvlink(gcid, index); - set_transport_mux_controls(gcid, scom_base, index, - NPU2_DEV_TYPE_OPENCAPI); enable_odl_phy_mux(gcid, index); disable_alink_fp(gcid); enable_xsl_clocks(gcid, scom_base, index); diff --git a/hw/npu2.c b/hw/npu2.c index 6c720764e614..a8744f4af7f7 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1308,6 +1308,62 @@ static const struct phb_ops npu_ops = { .tce_kill = npu2_tce_kill, }; +static void enable_nvlink(struct npu2_dev *dev) +{ + struct npu2 *npu = dev->npu; + int stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); + int block; + uint64_t addr, val; + + /* CQ_SM Misc Config #0 - enable NVLink mode */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + addr = NPU2_REG_OFFSET(stack, block, NPU2_CQ_SM_MISC_CFG0); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_SM_MISC_CFG0_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + } + + /* CQ_CTL Misc Config #0 - enable NVLink mode */ + addr = NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, NPU2_CQ_CTL_MISC_CFG); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_CTL_MISC_CFG_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* CQ_DAT Misc Config #1 - enable NVLink mode */ + addr = NPU2_REG_OFFSET(stack, NPU2_BLOCK_DAT, NPU2_CQ_DAT_MISC_CFG); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_DAT_MISC_CFG_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* NTL Misc Config 2 - enable NTL brick and checks */ + addr = NPU2_NTL_MISC_CFG2(dev); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_NTL_MISC_CFG2_BRICK_ENABLE; + val |= NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA; + val |= NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA; + val |= NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* High Water Marks */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + addr = NPU2_REG_OFFSET(stack, block, NPU2_HIGH_WATER_MARKS); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val &= ~NPU2_HIGH_WATER_MARKS_PWR0; + val |= PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + } +} + /* * Set up NPU for NVLink and create PCI root device node * accordingly. @@ -1315,59 +1371,12 @@ static const struct phb_ops npu_ops = { int npu2_nvlink_init_npu(struct npu2 *npu) { struct dt_node *np; - uint64_t reg[2], mm_win[2], val; + uint64_t reg[2], mm_win[2]; - /* TODO: Clean this up with register names, etc. when we get - * time. This just turns NVLink mode on in each brick and should - * get replaced with a patch from ajd once we've worked out how - * things are going to work there. - * - * Obviously if the year is now 2020 that didn't happen and you - * should fix this :-) */ - xscom_write_mask(npu->chip_id, 0x5011000, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011030, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011060, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011090, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011200, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011230, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011260, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011290, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011400, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011430, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011460, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011490, PPC_BIT(58), PPC_BIT(58)); - - xscom_write_mask(npu->chip_id, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(npu->chip_id, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(npu->chip_id, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); - - val = NPU2_NTL_MISC_CFG2_BRICK_ENABLE | - NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA | - NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA | - NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; - xscom_write_mask(npu->chip_id, 0x5011110, val, val); - xscom_write_mask(npu->chip_id, 0x5011130, val, val); - xscom_write_mask(npu->chip_id, 0x5011310, val, val); - xscom_write_mask(npu->chip_id, 0x5011330, val, val); - xscom_write_mask(npu->chip_id, 0x5011510, val, val); - xscom_write_mask(npu->chip_id, 0x5011530, val, val); - - val = PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); - xscom_write_mask(npu->chip_id, 0x5011009, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011039, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011069, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011099, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011209, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011239, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011269, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011299, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011409, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011439, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); + for (int i = 0; i < npu->total_devices; i++) { + if (npu->devices[i].type == NPU2_DEV_TYPE_NVLINK) + enable_nvlink(&npu->devices[i]); + } /* Populate PCI root device node */ reg[0] = (uint64_t)npu->regs; @@ -1599,6 +1608,9 @@ static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group, int size) int bdfn = (group << 3); for (i = 0; i < size; i++) { + if (p->devices[i].type != NPU2_DEV_TYPE_NVLINK) + continue; + if ((p->devices[i].bdfn & 0xf8) == (bdfn & 0xf8)) bdfn++; } @@ -1612,14 +1624,11 @@ static void npu2_configure_devices(struct npu2 *p) uint32_t index = 0; for (index = 0; index < p->total_devices; index++) { - uint32_t group_id; - dev = &p->devices[index]; if (dev->type != NPU2_DEV_TYPE_NVLINK) continue; - group_id = dt_prop_get_u32(dev->dt_node, "ibm,npu-group-id"); - dev->bdfn = npu_allocate_bdfn(p, group_id, index); + dev->bdfn = npu_allocate_bdfn(p, dev->group_id, index); /* This must be done after calling * npu_allocate_bdfn() */ diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 8273b2be26a2..a8f59dcdbe2f 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -119,6 +119,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_SM_MISC_CFG0 0x000 #define NPU2_CQ_SM_MISC_CFG0_CONFIG_ENABLE_PBUS PPC_BIT(38) #define NPU2_CQ_SM_MISC_CFG0_CONFIG_OCAPI_MODE PPC_BIT(57) +#define NPU2_CQ_SM_MISC_CFG0_CONFIG_NVLINK_MODE PPC_BIT(58) #define NPU2_CQ_SM_MISC_CFG1 0x008 #define NPU2_CQ_SM_MISC_CFG2 0x148 #define NPU2_PB_EPSILON 0x010 @@ -152,6 +153,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_LOW_WATER_MARKS 0x040 #define NPU2_LOW_WATER_MARKS_ENABLE_MACHINE_ALLOC PPC_BIT(51) #define NPU2_HIGH_WATER_MARKS 0x048 +#define NPU2_HIGH_WATER_MARKS_PWR0 PPC_BITMASK(6, 11); #define NPU2_RELAXED_ORDERING_CFG(n) (0x050 + (n)*8) #define NPU2_RELAXED_ORDERING_SOURCE(n) (PPC_BITMASK(0,31) >> ((n)*32)) #define NPU2_RELAXED_ORDERING_SOURCE_ENA PPC_BITMASK32(0,3) @@ -207,6 +209,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* CTL block registers */ #define NPU2_CQ_CTL_MISC_CFG 0x000 #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(52) +#define NPU2_CQ_CTL_MISC_CFG_CONFIG_NVLINK_MODE PPC_BIT(53) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL0_ENABLE PPC_BIT(55) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL1_ENABLE PPC_BIT(56) #define NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG 0x0B0 @@ -261,6 +264,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* DAT block registers */ #define NPU2_CQ_DAT_MISC_CFG 0x008 #define NPU2_CQ_DAT_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(40) +#define NPU2_CQ_DAT_MISC_CFG_CONFIG_NVLINK_MODE PPC_BIT(41) #define NPU2_CQ_DAT_ECC_CFG 0x010 #define NPU2_CQ_DAT_SCRATCH0 0x018 #define NPU2_CQ_DAT_ECC_STATUS 0x020 diff --git a/include/npu2.h b/include/npu2.h index 75c11ff3c26f..347a69b30d7d 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -114,6 +114,7 @@ struct npu2_dev { enum npu2_dev_type type; uint32_t link_index; uint32_t brick_index; + uint32_t group_id; uint64_t pl_xscom_base; struct dt_node *dt_node; struct npu2_bar ntl_bar; From patchwork Fri Jan 11 01:09:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023318 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPvt1GZJz9sCh for ; Fri, 11 Jan 2019 12:11:38 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPvs71zPzDqyZ for ; Fri, 11 Jan 2019 12:11:37 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsv4kjJzDqwt for ; Fri, 11 Jan 2019 12:09:55 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14ZvJ049164 for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxc52b4ds-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:51 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mRZ7930106 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 17736A405C; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 740D8A4060; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 2BA67A03BC; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:31 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0020-0000-0000-00000304A96C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0021-0000-0000-00002155AFB0 Message-Id: <336d5f4ecbec960b04ac2dff16e4d99b2b25bab7.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=277 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 11/15] hw/npu2-opencapi: FIR masking for mixed setups X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" When setting up an NPU with an OpenCAPI device, we need to mask the FIR bits for NDL Stall/NoStall, which are used for a different purpose on OpenCAPI vs NVLink. Currently, we just mask the bits for all links/DLs. When we support mixed setups of OpenCAPI + NVLink on the same NPU, we don't want to mask all the bits. Only mask the FIR bits for the specific links which are OpenCAPI. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Reviewed-by: Alexey Kardashevskiy --- hw/npu2-opencapi.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index d3e95a45e9a4..21dfbdd24989 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1310,7 +1310,8 @@ static void npu2_opencapi_final_fixup(struct phb *phb) static void mask_nvlink_fir(struct npu2 *p) { - uint64_t reg; + uint64_t reg, mask = 0ull; + int link_num; /* * From section 13.1.3.10 of the NPU workbook: "the NV-Link @@ -1319,35 +1320,46 @@ static void mask_nvlink_fir(struct npu2 *p) * OpenCAPI. Therefore, the corresponding bits in NPU FIR * Register 1 must be masked and configured to NOT cause the * NPU to go into Freeze or Fence mode or send an Interrupt." - * - * FIXME: will need to revisit when mixing nvlink with - * opencapi. Assumes an opencapi-only setup on both PHYs for - * now. */ + for (int i = 0; i < p-> total_devices; i++) { + struct npu2_dev *dev = &p->devices[i]; + /* Only mask OpenCAPI links */ + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + + if (dev->brick_index == 2 || dev->brick_index == 3) { + link_num = dev->brick_index - 2; + } else { + link_num = dev->brick_index; + } + mask = SETFIELD(PPC_BITMASK(link_num * 2, + link_num * 2 + 1), + mask, 0b11); + } /* Mask FIRs */ xscom_read(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, ®); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0xFFF); + reg |= mask; xscom_write(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, reg); /* freeze disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* fence disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* irq disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); } From patchwork Fri Jan 11 01:09:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023325 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPxq46PQz9sCh for ; Fri, 11 Jan 2019 12:13:19 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPxq2x3LzDqsh for ; Fri, 11 Jan 2019 12:13:19 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsw5FnlzDqwb for ; Fri, 11 Jan 2019 12:09:56 +1100 (AEDT) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14ES4075474 for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxcyarvvw-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:52 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mCl56950796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1AE67A405F; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 75B5FA4062; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 2E9F9A03BE; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:32 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0016-0000-0000-00000243A7ED X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0017-0000-0000-0000329DAF0C Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 12/15] hw/npu2: Fix OpenCAPI PE assignment X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" When we support mixing NVLink and OpenCAPI devices on the same NPU, we're going to have to share the same range of 16 PE numbers between NVLink and OpenCAPI PHBs. For OpenCAPI devices, PE assignment is only significant for determining which System Interrupt Log register is used for a particular brick - unlike NVLink, it doesn't play any role in determining how links are fenced. Split the PE range into a lower half which is used for NVLink, and an upper half that is used for OpenCAPI, with a fixed PE number assigned per brick. As the PE assignment for OpenCAPI devices is fixed, set the PE once during device init and then ignore calls to the set_pe() operation. Suggested-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- v1->v2: - Rebase on Reza's cleanups --- hw/npu2-opencapi.c | 74 +++++++++++++++++++++-------------------------- include/npu2.h | 21 +++++++++++-- 2 files changed, 52 insertions(+), 43 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 21dfbdd24989..0c7d4f9ff148 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -482,6 +482,20 @@ static void brick_config(uint32_t gcid, uint32_t scom_base, int index) enable_pb_snooping(gcid, scom_base, index); } +/* Procedure 13.1.3.4 - Brick to PE Mapping */ +static void pe_config(struct npu2_dev *dev) +{ + /* We currently use a fixed PE assignment per brick */ + uint64_t val, reg; + val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, NPU2_OCAPI_PE(dev)); + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, 0); + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, + NPU2_MISC_BRICK0_BDF2PE_MAP0 + + (dev->brick_index * 0x18)); + npu2_write(dev->npu, reg, val); +} + /* Procedure 13.1.3.5 - TL Configuration */ static void tl_config(uint32_t gcid, uint32_t scom_base, uint64_t index) { @@ -1222,47 +1236,18 @@ static int64_t npu2_opencapi_ioda_reset(struct phb __unused *phb, return OPAL_SUCCESS; } -static int64_t npu2_opencapi_set_pe(struct phb *phb, - uint64_t pe_num, - uint64_t bdfn, - uint8_t bcompare, - uint8_t dcompare, - uint8_t fcompare, - uint8_t action) +static int64_t npu2_opencapi_set_pe(struct phb __unused *phb, + uint64_t __unused pe_num, + uint64_t __unused bdfn, + uint8_t __unused bcompare, + uint8_t __unused dcompare, + uint8_t __unused fcompare, + uint8_t __unused action) { - struct npu2 *p; - struct npu2_dev *dev; - uint64_t reg, val, pe_bdfn; - - /* Sanity check */ - if (action != OPAL_MAP_PE && action != OPAL_UNMAP_PE) - return OPAL_PARAMETER; - if (pe_num >= NPU2_MAX_PE_NUM) - return OPAL_PARAMETER; - if (bdfn >> 8) - return OPAL_PARAMETER; - if (bcompare != OpalPciBusAll || - dcompare != OPAL_COMPARE_RID_DEVICE_NUMBER || - fcompare != OPAL_COMPARE_RID_FUNCTION_NUMBER) - return OPAL_UNSUPPORTED; - - /* Get the NPU2 device */ - dev = phb_to_npu2_dev_ocapi(phb); - if (!dev) - return OPAL_PARAMETER; - - p = dev->npu; - - pe_bdfn = dev->bdfn; - - val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; - val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); - val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, pe_bdfn); - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + - (dev->brick_index * 0x18)); - npu2_write(p, reg, val); - + /* + * Ignored on OpenCAPI - we use fixed PE assignments. May need + * addressing when we support dual-link devices. + */ return OPAL_SUCCESS; } @@ -1431,7 +1416,13 @@ static void setup_device(struct npu2_dev *dev) dt_add_property_cells(dn_phb, "ibm,links", 1); dt_add_property(dn_phb, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(dn_phb, "ibm,phb-diag-data-size", 0); + + /* + * We ignore whatever PE numbers Linux tries to set, so we just + * advertise enough that Linux won't complain + */ dt_add_property_cells(dn_phb, "ibm,opal-num-pes", NPU2_MAX_PE_NUM); + dt_add_property_cells(dn_phb, "ibm,opal-reserved-pe", NPU2_RESERVED_PE_NUM); dt_add_property_cells(dn_phb, "ranges", 0x02000000, hi32(mm_win[0]), lo32(mm_win[0]), @@ -1510,6 +1501,9 @@ int npu2_opencapi_init_npu(struct npu2 *npu) /* Procedure 13.1.3.1 - Select OCAPI vs NVLink */ brick_config(npu->chip_id, npu->xscom_base, dev->brick_index); + /* Procedure 13.1.3.4 - Brick to PE Mapping */ + pe_config(dev); + /* Procedure 13.1.3.5 - Transaction Layer Configuration */ tl_config(npu->chip_id, npu->xscom_base, dev->brick_index); diff --git a/include/npu2.h b/include/npu2.h index 347a69b30d7d..49b8e1dfcf7c 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -45,9 +45,24 @@ dev->npu->chip_id, dev->brick_index, ## a) -/* Number of PEs supported */ -#define NPU2_MAX_PE_NUM 16 -#define NPU2_RESERVED_PE_NUM 15 +/* + * Number of PEs supported + * + * The NPU supports PE numbers from 0-15. At present, we only assign a maximum + * of 1 PE per brick. + * + * NVLink devices are currently exposed to Linux underneath a single virtual + * PHB. Therefore, we give NVLink half the available PEs, which is enough for + * 6 bricks plus 1 reserved PE. + * + * For OpenCAPI, the BDF-to-PE registers are used exclusively for mapping + * bricks to System Interrupt Log registers (the BDF component of those + * registers is ignored). Currently, we allocate a fixed PE based on the brick + * index in the upper half of the PE namespace. + */ +#define NPU2_MAX_PE_NUM 8 +#define NPU2_RESERVED_PE_NUM 7 +#define NPU2_OCAPI_PE(ndev) ((ndev)->brick_index + NPU2_MAX_PE_NUM) #define NPU2_LINKS_PER_CHIP 6 From patchwork Fri Jan 11 01:09:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023322 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPwz2rb5z9sCh for ; Fri, 11 Jan 2019 12:12:35 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPwz1RRtzDr0f for ; Fri, 11 Jan 2019 12:12:35 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsw1GSyzDqwP for ; Fri, 11 Jan 2019 12:09:55 +1100 (AEDT) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14E2e075492 for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxcyarvvr-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:51 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:48 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19lFU2621752 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:47 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CA718A405F; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 787AEA4064; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 3BB7CA03BF; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:33 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0028-0000-0000-000003375C7F X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0029-0000-0000-000023F4734B Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=807 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 13/15] hw/npu2: Always check device type when looping through NPU devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Fix all the remaining places in the NVLink code where we loop through all the devices on an NPU and don't check whether it's an NVLink device - triggering NTL resets, setting up relaxed ordering, and setting up LPAR mapping. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Reviewed-by: Alexey Kardashevskiy --- hw/npu2.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index a8744f4af7f7..411028189975 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1120,7 +1120,7 @@ static int64_t npu2_hreset(struct pci_slot *slot __unused) for (i = 0; i < p->total_devices; i++) { ndev = &p->devices[i]; - if (ndev) { + if (ndev && ndev->type == NPU2_DEV_TYPE_NVLINK) { NPU2DEVINF(ndev, "Resetting device\n"); reset_ntl(ndev); } @@ -1144,7 +1144,7 @@ static int64_t npu2_creset(struct pci_slot *slot) for (i = 0; i < p->total_devices; i++) { ndev = &p->devices[i]; - if (ndev) { + if (ndev && ndev->type == NPU2_DEV_TYPE_NVLINK) { NPU2DEVINF(ndev, "Resetting device\n"); reset_ntl(ndev); } @@ -1972,7 +1972,8 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, /* Need to find an NVLink to send the ATSDs for this device over */ for (i = 0; i < p->total_devices; i++) { - if (p->devices[i].nvlink.gpu_bdfn == bdf) { + if (p->devices[i].type == NPU2_DEV_TYPE_NVLINK && + p->devices[i].nvlink.gpu_bdfn == bdf) { ndev = &p->devices[i]; break; } @@ -2156,6 +2157,9 @@ static int npu2_set_relaxed_ordering(uint32_t gcid, int pec, bool enable) npu = phb_to_npu2_nvlink(phb); for (int i = 0; i < npu->total_devices; i++) { ndev = &npu->devices[i]; + if (ndev->type != NPU2_DEV_TYPE_NVLINK) + continue; + if (enable) rc = npu2_enable_relaxed_ordering(ndev, gcid, pec); else From patchwork Fri Jan 11 01:09:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023323 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPxD0Qrwz9sCh for ; Fri, 11 Jan 2019 12:12:48 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPxC6JS3zDr0Q for ; Fri, 11 Jan 2019 12:12:47 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsw5YPjzDqwm for ; Fri, 11 Jan 2019 12:09:56 +1100 (AEDT) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14Ell069632 for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0b-001b2d01.pphosted.com with ESMTP id 2pxg8dj0cw-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:53 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:52 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:49 -0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19mMg4522442 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:48 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1CECA52052; Fri, 11 Jan 2019 01:09:48 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 6EC4F52050; Fri, 11 Jan 2019 01:09:47 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 4A2EEA03C2; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:34 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0028-0000-0000-000003375C80 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0029-0000-0000-000023F4734C Message-Id: <2ffb70b58cfe4a2b14f891305db83d84f23ba435.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 14/15] hw/npu2-common: Allow mixed mode NPU setups X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Now that we have all the support in place for NPUs with both NVLink and OpenCAPI devices, get rid of the error that aborts NPU init when a mixed setup is detected. While we're there, rename setup_devices() to more accurately reflect what it does, and move the calls to the NVLink/OpenCAPI setup code out of there into the main probe function. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-common.c | 28 ++++++---------------------- hw/npu2-opencapi.c | 5 ----- 2 files changed, 6 insertions(+), 27 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index e323e71cb8d5..9e1b4bcc250e 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -620,26 +620,19 @@ failed: return NULL; } -static void setup_devices(struct npu2 *npu) +static void add_link_type_properties(struct npu2 *npu) { - bool nvlink_detected = false, ocapi_detected = false; struct npu2_dev *dev; - /* - * TODO: In future, we'll do brick configuration here to support mixed - * setups. - */ for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; switch (dev->type) { case NPU2_DEV_TYPE_NVLINK: - nvlink_detected = true; dt_add_property_strings(dev->dt_node, "ibm,npu-link-type", "nvlink"); break; case NPU2_DEV_TYPE_OPENCAPI: - ocapi_detected = true; dt_add_property_strings(dev->dt_node, "ibm,npu-link-type", "opencapi"); @@ -652,19 +645,6 @@ static void setup_devices(struct npu2 *npu) "unknown"); } } - - if (nvlink_detected && ocapi_detected) { - prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported, aborting NPU init\n"); - return; - } - - assign_bars(npu); - setup_irqs(npu); - - if (nvlink_detected) - npu2_nvlink_init_npu(npu); - else if (ocapi_detected) - npu2_opencapi_init_npu(npu); } void probe_npu2(void) @@ -700,7 +680,11 @@ void probe_npu2(void) if (!npu) continue; platform.npu2_device_detect(npu); + add_link_type_properties(npu); set_brick_config(npu); - setup_devices(npu); + assign_bars(npu); + setup_irqs(npu); + npu2_nvlink_init_npu(npu); + npu2_opencapi_init_npu(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 0c7d4f9ff148..7acf3837fd40 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -19,15 +19,10 @@ * * This file provides support for OpenCAPI as implemented on POWER9. * - * At present, we initialise the NPU separately from the NVLink code in npu2.c. - * As such, we don't currently support mixed NVLink and OpenCAPI configurations - * on the same NPU for machines such as Witherspoon. - * * Procedure references in this file are to the POWER9 OpenCAPI NPU Workbook * (IBM internal document). * * TODO: - * - Support for mixed NVLink and OpenCAPI on the same NPU * - Support for link ganging (one AFU using multiple links) * - Link reset and error handling * - Presence detection From patchwork Fri Jan 11 01:09:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1023326 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43bPy50jw9z9sCh for ; Fri, 11 Jan 2019 12:13:33 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43bPy46h2gzDqy6 for ; Fri, 11 Jan 2019 12:13:32 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43bPsx1pSdzDqwh for ; Fri, 11 Jan 2019 12:09:56 +1100 (AEDT) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id x0B14ESn030976 for ; Thu, 10 Jan 2019 20:09:55 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pxe2vxa8r-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 10 Jan 2019 20:09:54 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 11 Jan 2019 01:09:53 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 11 Jan 2019 01:09:50 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x0B19nTw57278590 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Fri, 11 Jan 2019 01:09:49 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B1769AE045; Fri, 11 Jan 2019 01:09:49 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5F3B4AE04D; Fri, 11 Jan 2019 01:09:49 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 11 Jan 2019 01:09:49 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 5B60AA03C3; Fri, 11 Jan 2019 12:09:44 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 11 Jan 2019 12:09:35 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 19011101-0020-0000-0000-00000304A96E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19011101-0021-0000-0000-00002155AFB1 Message-Id: <00817c6b35b577055088a20055ce75ab6904624a.1547168645.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-01-11_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=722 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901110006 Subject: [Skiboot] [PATCH v2 15/15] hw/npu2-opencapi: Remove completed TODO X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" We support presence detection now. Remove the TODO about that. Suggested-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- hw/npu2-opencapi.c | 1 - 1 file changed, 1 deletion(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 7acf3837fd40..e268b75021b1 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -25,7 +25,6 @@ * TODO: * - Support for link ganging (one AFU using multiple links) * - Link reset and error handling - * - Presence detection * - Consume HDAT NPU information * - LPC Memory support */