From patchwork Wed Dec 12 06:58:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011607 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F78M6mMwz9s5c for ; Wed, 12 Dec 2018 18:04:03 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F78M4fq7zDqcY for ; Wed, 12 Dec 2018 18:04:03 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F76k6FlCzDqVS for ; Wed, 12 Dec 2018 18:02:38 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC72UbW076911 for ; Wed, 12 Dec 2018 02:02:36 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pasabs41w-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 02:02:31 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:30 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:27 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xQHm8782174 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:26 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A39064C059; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 09E684C04E; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:25 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 74FF3A0240; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:44 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0020-0000-0000-000002F5CB0E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0021-0000-0000-00002145F54E Message-Id: <45fc71211ef2d7b6810c7ee62cd50a51045d5bad.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 01/13] hw/npu2: Simplify BAR structures X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" At the moment, we have struct npu2_bar to represent an NPU BAR, and struct npu2_pcie_bar which wraps an npu2_bar and does nothing other than adding an additional flags field, which is exposed through the PCI virtual device. In npu2_bar, we have another flags field which is used to store state about the BAR, but it's only used internally, which means we use a lot of bitwise manipulations with no benefit other than saving a couple of bytes. Get rid of npu2_pcie_bar, convert the flags in npu2_bar::flags into individual bools, and fix references accordingly. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-opencapi.c | 12 ++--- hw/npu2.c | 104 +++++++++++++++++++++------------------------- include/npu2.h | 27 ++++-------- 3 files changed, 66 insertions(+), 77 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 65f623c731c2..b160a41741b9 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -807,8 +807,8 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, size); - dev->bars[0].npu2_bar.base = addr; - dev->bars[0].npu2_bar.size = size; + dev->bars[0].base = addr; + dev->bars[0].size = size; reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, addr >> 16); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(size >> 16)); @@ -832,8 +832,8 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", addr); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR), addr, size); - dev->bars[1].npu2_bar.base = addr; - dev->bars[1].npu2_bar.size = size; + dev->bars[1].base = addr; + dev->bars[1].size = size; } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, @@ -1293,7 +1293,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].npu2_bar.base + + genid_base = dev->bars[1].base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; @@ -1351,7 +1351,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].npu2_bar.base + + genid_base = dev->bars[1].base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; diff --git a/hw/npu2.c b/hw/npu2.c index 4e75eeb764ce..ba5575cf3097 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -114,7 +114,6 @@ static inline void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) { uint64_t reg, val; - int enabled; reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); val = npu2_read(p, reg); @@ -122,7 +121,7 @@ static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) switch (NPU2_REG(bar->reg)) { case NPU2_PHY_BAR: bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; - enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_2) /* This is the global MMIO BAR */ @@ -133,22 +132,20 @@ static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) case NPU2_NTL0_BAR: case NPU2_NTL1_BAR: bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; - enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); break; case NPU2_GENID_BAR: bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; - enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); + bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); bar->size = 0x20000; break; default: bar->base = 0ul; - enabled = 0; + bar->enabled = false; bar->size = 0; break; } - - bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, bar->flags, enabled); } static void npu2_write_bar(struct npu2 *p, @@ -156,23 +153,23 @@ static void npu2_write_bar(struct npu2 *p, uint32_t gcid, uint32_t scom) { - uint64_t reg, val, enable = !!(bar->flags & NPU2_BAR_FLAG_ENABLED); + uint64_t reg, val; int block; switch (NPU2_REG(bar->reg)) { case NPU2_PHY_BAR: val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); break; case NPU2_NTL0_BAR: case NPU2_NTL1_BAR: val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); val = SETFIELD(NPU2_NTL_BAR_SIZE, val, 1); break; case NPU2_GENID_BAR: val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, enable); + val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); break; default: val = 0ul; @@ -211,10 +208,10 @@ static int64_t npu2_cfg_write_cmd(void *dev, * one GENID BAR, which is exposed via the first brick. */ enabled = !!(*data & PCI_CFG_CMD_MEM_EN); - ntl_npu_bar = &ndev->bars[0].npu2_bar; - genid_npu_bar = &ndev->bars[1].npu2_bar; + ntl_npu_bar = &ndev->bars[0]; + genid_npu_bar = &ndev->bars[1]; - ntl_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, ntl_npu_bar->flags, enabled); + ntl_npu_bar->enabled = enabled; npu2_write_bar(ndev->npu, ntl_npu_bar, 0, 0); /* @@ -223,16 +220,12 @@ static int64_t npu2_cfg_write_cmd(void *dev, * track the enables separately. */ if (NPU2DEV_BRICK(ndev)) - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED1, genid_npu_bar->flags, - enabled); + genid_npu_bar->enabled1 = enabled; else - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED0, genid_npu_bar->flags, - enabled); + genid_npu_bar->enabled0 = enabled; /* Enable the BAR if either device requests it enabled, otherwise disable it */ - genid_npu_bar->flags = SETFIELD(NPU2_BAR_FLAG_ENABLED, genid_npu_bar->flags, - !!(genid_npu_bar->flags & (NPU2_BAR_FLAG_ENABLED0 | - NPU2_BAR_FLAG_ENABLED1))); + genid_npu_bar->enabled = genid_npu_bar->enabled0 || genid_npu_bar->enabled1; npu2_write_bar(ndev->npu, genid_npu_bar, 0, 0); return OPAL_PARTIAL; @@ -243,20 +236,21 @@ static int64_t npu2_cfg_read_bar(struct npu2_dev *dev __unused, uint32_t offset, uint32_t size, uint32_t *data) { - struct npu2_pcie_bar *bar = (struct npu2_pcie_bar *) pcrf->data; + struct npu2_bar *bar = (struct npu2_bar *) pcrf->data; - if (!(bar->flags & NPU2_PCIE_BAR_FLAG_TRAPPED)) + if (!(bar->trapped)) return OPAL_PARTIAL; if ((size != 4) || (offset != pcrf->start && offset != pcrf->start + 4)) return OPAL_PARAMETER; - if (bar->flags & NPU2_PCIE_BAR_FLAG_SIZE_HI) - *data = bar->npu2_bar.size >> 32; + if (bar->size_hi) + *data = bar->size >> 32; else - *data = bar->npu2_bar.size; - bar->flags &= ~(NPU2_PCIE_BAR_FLAG_TRAPPED | NPU2_PCIE_BAR_FLAG_SIZE_HI); + *data = bar->size; + bar->trapped = false; + bar->size_hi = false; return OPAL_SUCCESS; } @@ -267,8 +261,8 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, uint32_t data) { struct pci_virt_device *pvd = dev->nvlink.pvd; - struct npu2_pcie_bar *bar = (struct npu2_pcie_bar *) pcrf->data; - struct npu2_bar old_bar, *npu2_bar = &bar->npu2_bar; + struct npu2_bar *bar = (struct npu2_bar *) pcrf->data; + struct npu2_bar old_bar; uint32_t pci_cmd; if ((size != 4) || @@ -277,36 +271,36 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, /* Return BAR size on next read */ if (data == 0xffffffff) { - bar->flags |= NPU2_PCIE_BAR_FLAG_TRAPPED; + bar->trapped = true; if (offset == pcrf->start + 4) - bar->flags |= NPU2_PCIE_BAR_FLAG_SIZE_HI; + bar->size_hi = true; return OPAL_SUCCESS; } if (offset == pcrf->start) { - npu2_bar->base &= 0xffffffff00000000; - npu2_bar->base |= (data & 0xfffffff0); + bar->base &= 0xffffffff00000000; + bar->base |= (data & 0xfffffff0); } else { - npu2_bar->base &= 0x00000000ffffffff; - npu2_bar->base |= ((uint64_t)data << 32); + bar->base &= 0x00000000ffffffff; + bar->base |= ((uint64_t)data << 32); PCI_VIRT_CFG_NORMAL_RD(pvd, PCI_CFG_CMD, 4, &pci_cmd); - if (NPU2_REG(npu2_bar->reg) == NPU2_GENID_BAR && NPU2DEV_BRICK(dev)) - npu2_bar->base -= 0x10000; + if (NPU2_REG(bar->reg) == NPU2_GENID_BAR && NPU2DEV_BRICK(dev)) + bar->base -= 0x10000; - old_bar.reg = npu2_bar->reg; + old_bar.reg = bar->reg; npu2_read_bar(dev->npu, &old_bar); /* Only allow changing the base address if the BAR is not enabled */ - if ((npu2_bar->flags & NPU2_BAR_FLAG_ENABLED) && - (npu2_bar->base != old_bar.base)) { - npu2_bar->base = old_bar.base; + if (bar->enabled && + (bar->base != old_bar.base)) { + bar->base = old_bar.base; return OPAL_HARDWARE; } - npu2_write_bar(dev->npu, &bar->npu2_bar, 0, 0); + npu2_write_bar(dev->npu, bar, 0, 0); } /* To update the config cache */ @@ -1452,13 +1446,13 @@ static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint /* NPU_REGS must be first in this list */ { .type = NPU_REGS, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_PHY, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_PHY, .index = 1, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .flags = NPU2_BAR_FLAG_ENABLED }, + .enabled = true }, { .type = NPU_NTL, .index = 0, .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL0_BAR) }, { .type = NPU_NTL, .index = 1, @@ -1715,7 +1709,7 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, static void npu2_populate_cfg(struct npu2_dev *dev) { struct pci_virt_device *pvd = dev->nvlink.pvd; - struct npu2_pcie_bar *bar; + struct npu2_bar *bar; uint32_t pos; /* 0x00 - Vendor/Device ID */ @@ -1737,9 +1731,9 @@ static void npu2_populate_cfg(struct npu2_dev *dev) /* 0x10/14 - BAR#0, NTL BAR */ bar = &dev->bars[0]; PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR0, 4, - (bar->npu2_bar.base & 0xfffffff0) | (bar->flags & 0xF), + (bar->base & 0xfffffff0) | bar->flags, 0x0000000f, 0x00000000); - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR1, 4, (bar->npu2_bar.base >> 32), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR1, 4, (bar->base >> 32), 0x00000000, 0x00000000); pci_virt_add_filter(pvd, PCI_CFG_BAR0, 8, PCI_REG_FLAG_READ | PCI_REG_FLAG_WRITE, @@ -1748,16 +1742,16 @@ static void npu2_populate_cfg(struct npu2_dev *dev) /* 0x18/1c - BAR#1, GENID BAR */ bar = &dev->bars[1]; if (NPU2DEV_BRICK(dev) == 0) - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->npu2_bar.base & 0xfffffff0) | - (bar->flags & 0xF), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->base & 0xfffffff0) | + bar->flags, 0x0000000f, 0x00000000); else /* Brick 1 gets the upper portion of the generation id register */ - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, ((bar->npu2_bar.base + 0x10000) & 0xfffffff0) | - (bar->flags & 0xF), + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, ((bar->base + 0x10000) & 0xfffffff0) | + bar->flags, 0x0000000f, 0x00000000); - PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR3, 4, (bar->npu2_bar.base >> 32), 0x00000000, + PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR3, 4, (bar->base >> 32), 0x00000000, 0x00000000); pci_virt_add_filter(pvd, PCI_CFG_BAR2, 8, PCI_REG_FLAG_READ | PCI_REG_FLAG_WRITE, @@ -1847,7 +1841,7 @@ static void npu2_populate_devices(struct npu2 *p, /* Populate BARs. BAR0/1 is the NTL bar. */ stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); - npu2_bar = &dev->bars[0].npu2_bar; + npu2_bar = &dev->bars[0]; npu2_bar->type = NPU_NTL; npu2_bar->index = dev->brick_index; npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? @@ -1857,7 +1851,7 @@ static void npu2_populate_devices(struct npu2 *p, dev->bars[0].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; /* BAR2/3 is the GENID bar. */ - npu2_bar = &dev->bars[1].npu2_bar; + npu2_bar = &dev->bars[1]; npu2_bar->type = NPU_GENID; npu2_bar->index = NPU2DEV_STACK(dev); npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); diff --git a/include/npu2.h b/include/npu2.h index 1de963d19928..da5a5597feb0 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -67,26 +67,21 @@ struct npu2_bar { enum phys_map_type type; int index; -#define NPU2_BAR_FLAG_ENABLED 0x0010 + uint32_t flags; /* NVLink: exported to PCI config space */ + uint64_t base; + uint64_t size; + uint64_t reg; + + bool enabled; /* Generation ID's are a single space in the hardware but we split * them in two for the emulated PCIe devices so we need to keep track * of which one has been enabled/disabled. */ -#define NPU2_BAR_FLAG_ENABLED0 0x0080 -#define NPU2_BAR_FLAG_ENABLED1 0x0100 - uint32_t flags; - uint64_t base; - uint64_t size; - uint64_t reg; -}; + bool enabled0; + bool enabled1; -/* Rpresents a BAR that is exposed via the PCIe emulated - * devices */ -struct npu2_pcie_bar { -#define NPU2_PCIE_BAR_FLAG_SIZE_HI 0x0020 -#define NPU2_PCIE_BAR_FLAG_TRAPPED 0x0040 - uint32_t flags; - struct npu2_bar npu2_bar; + bool size_hi; + bool trapped; }; enum npu2_dev_type { @@ -124,7 +119,7 @@ struct npu2_dev { uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; - struct npu2_pcie_bar bars[2]; + struct npu2_bar bars[2]; struct npu2 *npu; uint32_t bdfn; From patchwork Wed Dec 12 06:58:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011594 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F73K4QD0z9s5c for ; Wed, 12 Dec 2018 17:59:41 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F73K21x7zDqqD for ; Wed, 12 Dec 2018 17:59:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73B4Sf3zDqcG for ; Wed, 12 Dec 2018 17:59:34 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wTiD023018 for ; Wed, 12 Dec 2018 01:59:32 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pawgng390-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:31 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:29 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:27 -0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xQL961341876 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:26 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 99ED552052; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 009C452059; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 89DBAA026A; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:45 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-4275-0000-0000-000002EF53C8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-4276-0000-0000-000037FC7450 Message-Id: <542ba2cf95249cd4d20c798e79a095d3058d9a38.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 02/13] hw/npu2: Merge implementations of BAR accessor functions X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Move npu2_{get,read,write}_bar() to common code. Get rid of the OpenCAPI write_bar() function. Fix the OpenCAPI code to use the NVLink-style BAR accessor functions and struct npu2_bar. While we're there, fix a trivial bug in npu2_read_bar() when reading PHY BARs - the global MMIO BAR is stack 0, not stack 2. I don't think we ever read that BAR, so this has never been a problem. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-common.c | 77 ++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 109 ++++++++++++++++------------------------------ hw/npu2.c | 78 +--------------------------------- include/npu2.h | 4 ++- 4 files changed, 120 insertions(+), 148 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 6e6b12f0d1ae..3446acb45bea 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -22,6 +22,7 @@ #include #include #include +#include /* * We use the indirect method because it uses the same addresses as @@ -97,6 +98,82 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas (uint64_t)new_val << 32); } +void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) +{ + phys_map_get(gcid, bar->type, bar->index, &bar->base, &bar->size); +} + +void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) +{ + uint64_t reg, val; + + reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); + val = npu2_read(p, reg); + + switch (NPU2_REG(bar->reg)) { + case NPU2_PHY_BAR: + bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; + bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); + + if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_0) + /* This is the global MMIO BAR */ + bar->size = 0x1000000; + else + bar->size = 0x200000; + break; + case NPU2_NTL0_BAR: + case NPU2_NTL1_BAR: + bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; + bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); + bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); + break; + case NPU2_GENID_BAR: + bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; + bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); + bar->size = 0x20000; + break; + default: + bar->base = 0ul; + bar->enabled = false; + bar->size = 0; + break; + } +} + +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, + uint32_t scom) +{ + uint64_t reg, val; + int block; + + switch (NPU2_REG(bar->reg)) { + case NPU2_PHY_BAR: + val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); + val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); + break; + case NPU2_NTL0_BAR: + case NPU2_NTL1_BAR: + val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); + val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); + val = SETFIELD(NPU2_NTL_BAR_SIZE, val, ilog2(bar->size >> 16)); + break; + case NPU2_GENID_BAR: + val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); + val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); + break; + default: + val = 0ul; + } + + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + reg = NPU2_REG_OFFSET(0, block, bar->reg); + if (p) + npu2_write(p, reg, val); + else + npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); + } +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index b160a41741b9..88e4eb1974a2 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -730,63 +730,29 @@ static void address_translation_config(uint32_t gcid, uint32_t scom_base, } } -/* TODO: Merge this with NVLink implementation - we don't use the npu2_bar - * wrapper for the PHY BARs yet */ -static void write_bar(uint32_t gcid, uint32_t scom_base, uint64_t reg, - uint64_t addr, uint64_t size) -{ - uint64_t val; - int block; - switch (NPU2_REG(reg)) { - case NPU2_PHY_BAR: - val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, addr >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, 1); - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, addr >> 16); - val = SETFIELD(NPU2_NTL_BAR_SIZE, val, ilog2(size >> 16)); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, 1); - break; - case NPU2_GENID_BAR: - val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, addr >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, 1); - break; - default: - val = 0ul; - } - - for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { - npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(0, block, reg), - NPU2_MISC_DA_LEN_8B, val); - prlog(PR_DEBUG, "OCAPI: Setting BAR %llx to %llx\n", - NPU2_REG_OFFSET(0, block, reg), val); - } -} - static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, uint64_t reg[]) { - uint64_t addr, size; - - prlog(PR_DEBUG, "OCAPI: patching up PHY0 bar, %s\n", __func__); - phys_map_get(gcid, NPU_PHY, 0, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - addr, size); - prlog(PR_DEBUG, "OCAPI: patching up PHY1 bar, %s\n", __func__); - phys_map_get(gcid, NPU_PHY, 1, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - addr, size); - - prlog(PR_DEBUG, "OCAPI: setup global mmio, %s\n", __func__); - phys_map_get(gcid, NPU_REGS, 0, &addr, &size); - write_bar(gcid, scom_base, - NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - addr, size); - reg[0] = addr; - reg[1] = size; + struct npu2_bar *bar; + struct npu2_bar phy_bars[] = { + { .type = NPU_PHY, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 1, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_REGS, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), + .enabled = true }, + }; + + for (int i = 0; i < ARRAY_SIZE(phy_bars); i++) { + bar = &phy_bars[i]; + npu2_get_bar(gcid, bar); + npu2_write_bar(NULL, bar, gcid, scom_base); + } + reg[0] = phy_bars[2].base; + reg[1] = phy_bars[2].size; } /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ @@ -803,15 +769,18 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); phys_map_get(gcid, NPU_OCAPI_MMIO, dev->brick_index, &addr, &size); - - prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); - write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, - size); - dev->bars[0].base = addr; - dev->bars[0].size = size; - - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, addr >> 16); - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(size >> 16)); + dev->bars[0].type = NPU_OCAPI_MMIO; + dev->bars[0].index = dev->brick_index; + dev->bars[0].reg = NPU2_REG_OFFSET(stack, 0, offset); + dev->bars[0].enabled = true; + npu2_get_bar(gcid, &dev->bars[0]); + + prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", + dev->bars[0].base, dev->bars[0].size); + npu2_write_bar(NULL, &dev->bars[0], gcid, scom_base); + + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->bars[0].base >> 16); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->bars[0].size >> 16)); prlog(PR_DEBUG, "OCAPI: PA translation %llx\n", reg); npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, @@ -825,15 +794,15 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, { uint64_t stack = index_to_stack(dev->brick_index); int stack_num = stack - NPU2_STACK_STCK_0; - uint64_t addr, size; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); - phys_map_get(gcid, NPU_GENID, stack_num, &addr, &size); - prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", addr); - write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR), - addr, size); - dev->bars[1].base = addr; - dev->bars[1].size = size; + dev->bars[1].type = NPU_GENID; + dev->bars[1].index = stack_num; + dev->bars[1].reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); + dev->bars[1].enabled = true; + npu2_get_bar(gcid, &dev->bars[1]); + prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->bars[1].base); + npu2_write_bar(NULL, &dev->bars[1], gcid, scom_base); } static void otl_enabletx(uint32_t gcid, uint32_t scom_base, diff --git a/hw/npu2.c b/hw/npu2.c index ba5575cf3097..6b0880682427 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -106,84 +106,6 @@ static struct npu2_dev *npu2_bdf_to_dev(struct npu2 *p, return NULL; } -static inline void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar) -{ - phys_map_get(gcid, bar->type, bar->index, &bar->base, &bar->size); -} - -static void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) -{ - uint64_t reg, val; - - reg = NPU2_REG_OFFSET(0, NPU2_BLOCK_SM_0, bar->reg); - val = npu2_read(p, reg); - - switch (NPU2_REG(bar->reg)) { - case NPU2_PHY_BAR: - bar->base = GETFIELD(NPU2_PHY_BAR_ADDR, val) << 21; - bar->enabled = GETFIELD(NPU2_PHY_BAR_ENABLE, val); - - if (NPU2_REG_STACK(reg) == NPU2_STACK_STCK_2) - /* This is the global MMIO BAR */ - bar->size = 0x1000000; - else - bar->size = 0x200000; - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - bar->base = GETFIELD(NPU2_NTL_BAR_ADDR, val) << 16; - bar->enabled = GETFIELD(NPU2_NTL_BAR_ENABLE, val); - bar->size = 0x10000 << GETFIELD(NPU2_NTL_BAR_SIZE, val); - break; - case NPU2_GENID_BAR: - bar->base = GETFIELD(NPU2_GENID_BAR_ADDR, val) << 16; - bar->enabled = GETFIELD(NPU2_GENID_BAR_ENABLE, val); - bar->size = 0x20000; - break; - default: - bar->base = 0ul; - bar->enabled = false; - bar->size = 0; - break; - } -} - -static void npu2_write_bar(struct npu2 *p, - struct npu2_bar *bar, - uint32_t gcid, - uint32_t scom) -{ - uint64_t reg, val; - int block; - - switch (NPU2_REG(bar->reg)) { - case NPU2_PHY_BAR: - val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, bar->base >> 21); - val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, bar->enabled); - break; - case NPU2_NTL0_BAR: - case NPU2_NTL1_BAR: - val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, bar->enabled); - val = SETFIELD(NPU2_NTL_BAR_SIZE, val, 1); - break; - case NPU2_GENID_BAR: - val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, bar->base >> 16); - val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, bar->enabled); - break; - default: - val = 0ul; - } - - for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { - reg = NPU2_REG_OFFSET(0, block, bar->reg); - if (p) - npu2_write(p, reg, val); - else - npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); - } -} - /* Trap for PCI command (0x4) to enable or disable device's BARs */ static int64_t npu2_cfg_write_cmd(void *dev, struct pci_cfg_reg_filter *pcrf __unused, diff --git a/include/npu2.h b/include/npu2.h index da5a5597feb0..c5c5b43843c3 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -210,6 +210,10 @@ void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val); uint64_t npu2_read(struct npu2 *p, uint64_t reg); void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask); +void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar); +void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar); +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, + uint32_t scom); int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); From patchwork Wed Dec 12 06:58:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011606 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F77y0jhbz9s8J for ; Wed, 12 Dec 2018 18:03:42 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F77x6X35zDqnZ for ; Wed, 12 Dec 2018 18:03:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F76Q2jTtzDr4R for ; Wed, 12 Dec 2018 18:02:22 +1100 (AEDT) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC72EEq048780 for ; Wed, 12 Dec 2018 02:02:20 -0500 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pauk0vd88-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 02:02:17 -0500 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:30 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:27 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xQGG56361030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:26 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC7DAAE058; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 09678AE051; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:25 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 919B1A02C3; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:46 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0012-0000-0000-000002D7DCE7 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0013-0000-0000-0000210D5647 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 03/13] hw/npu2: Move PHY/NTL/GENID BAR assignment to common code X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Assignment of PHY/NTL/GENID BARs is currently duplicated between NVLink and OpenCAPI. This is going to cause us particular issues later on when we implement support for mixed-mode setups with NVLink and OpenCAPI on the same NPU. Centralise the assignment of PHY/NTL/GENID BARs in common code. Signed-off-by: Andrew Donnellan --- hw/npu2-common.c | 76 ++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 80 ++++---------------------------- hw/npu2.c | 116 +++++----------------------------------------- include/npu2.h | 4 +- 4 files changed, 104 insertions(+), 172 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 3446acb45bea..b140e9ffd064 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -174,6 +175,79 @@ void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, } } +static void assign_bars(struct npu2 *npu) +{ + uint32_t i; + struct npu2_bar *bar; + struct npu2_dev *dev; + struct npu2_bar phy_bars[] = { + /* NPU_REGS must be first in this list */ + { .type = NPU_REGS, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 0, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), + .enabled = true }, + { .type = NPU_PHY, .index = 1, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), + .enabled = true }, + }; + struct npu2_bar last_bar = + { .type = NPU_GENID, .index = 2, + .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_GENID_BAR) }; + + /* Set common PHY BARs */ + for (i = 0; i < ARRAY_SIZE(phy_bars); i++) { + bar = &phy_bars[i]; + npu2_get_bar(npu->chip_id, bar); + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + } + + /* Device BARs */ + for (i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type == NPU2_DEV_TYPE_UNKNOWN) + continue; + + /* NTL / OCAPI_MMIO BAR */ + bar = &dev->ntl_bar; + if (dev->type == NPU2_DEV_TYPE_NVLINK) + bar->type = NPU_NTL; + else if (dev->type == NPU2_DEV_TYPE_OPENCAPI) + bar->type = NPU_OCAPI_MMIO; + bar->index = dev->brick_index; + bar->reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev), + 0, NPU2DEV_BRICK(dev) == 0 ? + NPU2_NTL0_BAR : NPU2_NTL1_BAR); + bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + npu2_get_bar(npu->chip_id, bar); + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + + /* GENID BAR */ + bar = &dev->genid_bar; + bar->type = NPU_GENID; + bar->index = NPU2DEV_STACK(dev); + bar->reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev), + 0, NPU2_GENID_BAR); + bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; + npu2_get_bar(npu->chip_id, bar); + bar->size = 0x10000; + if (NPU2DEV_BRICK(dev)) + bar->base += 0x10000; + npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + }; + + /* Global MMIO BAR */ + npu->regs = (uint64_t *)phy_bars[0].base; + npu->regs_size = phy_bars[0].size; + + /* NTL and GENID BARs are exposed to kernel via the mm + * window */ + npu2_get_bar(npu->chip_id, &last_bar); + npu->mm_base = phy_bars[2].base + phy_bars[2].size; + npu->mm_size = last_bar.base + last_bar.size - npu->mm_base; +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; @@ -350,6 +424,8 @@ static void setup_devices(struct npu2 *npu) return; } + assign_bars(npu); + if (nvlink_detected) npu2_nvlink_init_npu(npu); else if (ocapi_detected) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 88e4eb1974a2..b374a1035ac9 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -730,57 +730,22 @@ static void address_translation_config(uint32_t gcid, uint32_t scom_base, } } -static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, - uint64_t reg[]) -{ - struct npu2_bar *bar; - struct npu2_bar phy_bars[] = { - { .type = NPU_PHY, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_REGS, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .enabled = true }, - }; - - for (int i = 0; i < ARRAY_SIZE(phy_bars); i++) { - bar = &phy_bars[i]; - npu2_get_bar(gcid, bar); - npu2_write_bar(NULL, bar, gcid, scom_base); - } - reg[0] = phy_bars[2].base; - reg[1] = phy_bars[2].size; -} - /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { uint64_t stack = index_to_stack(dev->brick_index); - uint64_t offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? - NPU2_NTL0_BAR : NPU2_NTL1_BAR; uint64_t pa_offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; - uint64_t addr, size, reg; + uint64_t reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - phys_map_get(gcid, NPU_OCAPI_MMIO, dev->brick_index, &addr, &size); - dev->bars[0].type = NPU_OCAPI_MMIO; - dev->bars[0].index = dev->brick_index; - dev->bars[0].reg = NPU2_REG_OFFSET(stack, 0, offset); - dev->bars[0].enabled = true; - npu2_get_bar(gcid, &dev->bars[0]); - - prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", - dev->bars[0].base, dev->bars[0].size); - npu2_write_bar(NULL, &dev->bars[0], gcid, scom_base); - - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->bars[0].base >> 16); - reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->bars[0].size >> 16)); + dev->ntl_bar.enabled = true; + npu2_write_bar(dev->npu, &dev->ntl_bar, gcid, scom_base); + + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->ntl_bar.base >> 16); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->ntl_bar.size >> 16)); prlog(PR_DEBUG, "OCAPI: PA translation %llx\n", reg); npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, @@ -788,23 +753,6 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, NPU2_MISC_DA_LEN_8B, reg); } -/* Procedure 13.1.3.9 - AFU Config BARs */ -static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, - struct npu2_dev *dev) -{ - uint64_t stack = index_to_stack(dev->brick_index); - int stack_num = stack - NPU2_STACK_STCK_0; - - prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); - dev->bars[1].type = NPU_GENID; - dev->bars[1].index = stack_num; - dev->bars[1].reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); - dev->bars[1].enabled = true; - npu2_get_bar(gcid, &dev->bars[1]); - prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", dev->bars[1].base); - npu2_write_bar(NULL, &dev->bars[1], gcid, scom_base); -} - static void otl_enabletx(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { @@ -1262,7 +1210,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].base + + genid_base = dev->genid_bar.base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; @@ -1320,7 +1268,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, if (rc) return rc; - genid_base = dev->bars[1].base + + genid_base = dev->genid_bar.base + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; @@ -1582,8 +1530,8 @@ static void setup_device(struct npu2_dev *dev) uint64_t mm_win[2]; /* Populate PHB device node */ - phys_map_get(dev->npu->chip_id, NPU_OCAPI_MMIO, dev->brick_index, &mm_win[0], - &mm_win[1]); + mm_win[0] = dev->ntl_bar.base; + mm_win[1] = dev->ntl_bar.size; prlog(PR_DEBUG, "OCAPI: Setting MMIO window to %016llx + %016llx\n", mm_win[0], mm_win[1]); dn_phb = dt_new_addr(dt_root, "pciex", mm_win[0]); @@ -1629,7 +1577,8 @@ static void setup_device(struct npu2_dev *dev) /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ setup_afu_mmio_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); /* Procedure 13.1.3.9 - AFU Config BARs */ - setup_afu_config_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); + dev->genid_bar.enabled = true; + npu2_write_bar(dev->npu, &dev->genid_bar, dev->npu->chip_id, dev->npu->xscom_base); set_fence_control(dev->npu->chip_id, dev->npu->xscom_base, dev->brick_index, 0b00); @@ -1670,7 +1619,6 @@ static void read_nvram_training_state(void) int npu2_opencapi_init_npu(struct npu2 *npu) { struct npu2_dev *dev; - uint64_t reg[2]; int rc; assert(platform.ocapi); @@ -1679,10 +1627,6 @@ int npu2_opencapi_init_npu(struct npu2 *npu) /* TODO: Test OpenCAPI with fast reboot and make it work */ disable_fast_reboot("OpenCAPI device enabled"); - setup_global_mmio_bar(npu->chip_id, npu->xscom_base, reg); - - npu->regs = (void *)reg[0]; - for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; if (dev->type != NPU2_DEV_TYPE_OPENCAPI) diff --git a/hw/npu2.c b/hw/npu2.c index 6b0880682427..6aa16a43f803 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -114,7 +114,6 @@ static int64_t npu2_cfg_write_cmd(void *dev, { struct pci_virt_device *pvd = dev; struct npu2_dev *ndev = pvd->data; - struct npu2_bar *ntl_npu_bar, *genid_npu_bar; bool enabled; if (!write) @@ -130,11 +129,9 @@ static int64_t npu2_cfg_write_cmd(void *dev, * one GENID BAR, which is exposed via the first brick. */ enabled = !!(*data & PCI_CFG_CMD_MEM_EN); - ntl_npu_bar = &ndev->bars[0]; - genid_npu_bar = &ndev->bars[1]; - ntl_npu_bar->enabled = enabled; - npu2_write_bar(ndev->npu, ntl_npu_bar, 0, 0); + ndev->ntl_bar.enabled = enabled; + npu2_write_bar(ndev->npu, &ndev->ntl_bar, 0, 0); /* * Enable/disable the GENID BAR. Two bricks share one GENID @@ -142,13 +139,14 @@ static int64_t npu2_cfg_write_cmd(void *dev, * track the enables separately. */ if (NPU2DEV_BRICK(ndev)) - genid_npu_bar->enabled1 = enabled; + ndev->genid_bar.enabled1 = enabled; else - genid_npu_bar->enabled0 = enabled; + ndev->genid_bar.enabled0 = enabled; /* Enable the BAR if either device requests it enabled, otherwise disable it */ - genid_npu_bar->enabled = genid_npu_bar->enabled0 || genid_npu_bar->enabled1; - npu2_write_bar(ndev->npu, genid_npu_bar, 0, 0); + ndev->genid_bar.enabled = ndev->genid_bar.enabled0 || + ndev->genid_bar.enabled1; + npu2_write_bar(ndev->npu, &ndev->genid_bar, 0, 0); return OPAL_PARTIAL; } @@ -1360,59 +1358,6 @@ static const struct phb_ops npu_ops = { .tce_kill = npu2_tce_kill, }; -static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint64_t mm_win[2]) -{ - uint32_t i; - struct npu2_bar *bar; - struct npu2_bar npu2_bars[] = { - /* NPU_REGS must be first in this list */ - { .type = NPU_REGS, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_PHY, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), - .enabled = true }, - { .type = NPU_NTL, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_NTL1_BAR) }, - { .type = NPU_NTL, .index = 2, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 3, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_NTL1_BAR) }, - { .type = NPU_NTL, .index = 4, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_NTL0_BAR) }, - { .type = NPU_NTL, .index = 5, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_NTL1_BAR) }, - { .type = NPU_GENID, .index = 0, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_GENID_BAR) }, - { .type = NPU_GENID, .index = 1, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_GENID_BAR) }, - { .type = NPU_GENID, .index = 2, - .reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_GENID_BAR) }, - }; - - for (i = 0; i < ARRAY_SIZE(npu2_bars); i++) { - bar = &npu2_bars[i]; - npu2_get_bar(gcid, bar); - npu2_write_bar(NULL, bar, gcid, scom); - } - - /* Global MMIO BAR */ - reg[0] = npu2_bars[0].base; - reg[1] = npu2_bars[0].size; - - /* NTL and GENID BARs are exposed to kernel via the mm - * window */ - mm_win[0] = npu2_bars[3].base; - mm_win[1] = npu2_bars[ARRAY_SIZE(npu2_bars) - 1].base + - npu2_bars[ARRAY_SIZE(npu2_bars) - 1].size - - mm_win[0]; -} - /* * Set up NPU for NVLink and create PCI root device node * accordingly. @@ -1474,19 +1419,9 @@ int npu2_nvlink_init_npu(struct npu2 *npu) xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); - /* Reassign the BARs */ - assign_mmio_bars(npu->chip_id, npu->xscom_base, reg, mm_win); - npu->regs = (void *)reg[0]; - npu->mm_base = mm_win[0]; - npu->mm_size = mm_win[1]; - - if (reg[0] && reg[1]) - prlog(PR_INFO, " Global MMIO BAR: %016llx (%lldMB)\n", - reg[0], reg[1] >> 20); - else - prlog(PR_ERR, " Global MMIO BAR: Disabled\n"); - /* Populate PCI root device node */ + reg[0] = (uint64_t)npu->regs; + reg[1] = npu->regs_size; np = dt_new_addr(dt_root, "pciex", reg[0]); assert(np); dt_add_property_strings(np, @@ -1501,6 +1436,8 @@ int npu2_nvlink_init_npu(struct npu2 *npu) dt_add_property_cells(np, "ibm,xscom-base", npu->xscom_base); dt_add_property_cells(np, "ibm,npcq", npu->dt_node->phandle); dt_add_property_cells(np, "ibm,links", npu->total_devices); + mm_win[0] = npu->mm_base; + mm_win[1] = npu->mm_size; dt_add_property(np, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(np, "ibm,phb-diag-data-size", 0); @@ -1651,7 +1588,7 @@ static void npu2_populate_cfg(struct npu2_dev *dev) PCI_VIRT_CFG_INIT_RO(pvd, PCI_CFG_CACHE_LINE_SIZE, 4, 0x00800000); /* 0x10/14 - BAR#0, NTL BAR */ - bar = &dev->bars[0]; + bar = &dev->ntl_bar; PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR0, 4, (bar->base & 0xfffffff0) | bar->flags, 0x0000000f, 0x00000000); @@ -1662,7 +1599,7 @@ static void npu2_populate_cfg(struct npu2_dev *dev) npu2_dev_cfg_bar, bar); /* 0x18/1c - BAR#1, GENID BAR */ - bar = &dev->bars[1]; + bar = &dev->genid_bar; if (NPU2DEV_BRICK(dev) == 0) PCI_VIRT_CFG_INIT(pvd, PCI_CFG_BAR2, 4, (bar->base & 0xfffffff0) | bar->flags, @@ -1726,7 +1663,6 @@ static void npu2_populate_devices(struct npu2 *p, struct npu2_dev *dev; struct dt_node *npu2_dn, *link; uint32_t npu_phandle, index = 0; - int stack; /* * Get the npu node which has the links which we expand here @@ -1741,7 +1677,6 @@ static void npu2_populate_devices(struct npu2 *p, p->phb_nvlink.scan_map = 0; dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { uint32_t group_id; - struct npu2_bar *npu2_bar; dev = &p->devices[index]; dev->type = NPU2_DEV_TYPE_NVLINK; @@ -1761,31 +1696,6 @@ static void npu2_populate_devices(struct npu2 *p, dev->pl_xscom_base = dt_prop_get_u64(link, "ibm,npu-phy"); dev->lane_mask = dt_prop_get_u32(link, "ibm,npu-lane-mask"); - /* Populate BARs. BAR0/1 is the NTL bar. */ - stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); - npu2_bar = &dev->bars[0]; - npu2_bar->type = NPU_NTL; - npu2_bar->index = dev->brick_index; - npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? - NPU2_NTL0_BAR : NPU2_NTL1_BAR); - npu2_get_bar(p->chip_id, npu2_bar); - - dev->bars[0].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; - - /* BAR2/3 is the GENID bar. */ - npu2_bar = &dev->bars[1]; - npu2_bar->type = NPU_GENID; - npu2_bar->index = NPU2DEV_STACK(dev); - npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR); - npu2_get_bar(p->chip_id, npu2_bar); - - /* The GENID is a single physical BAR that we split - * for each emulated device */ - npu2_bar->size = 0x10000; - if (NPU2DEV_BRICK(dev)) - npu2_bar->base += 0x10000; - dev->bars[1].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; - /* Initialize PCI virtual device */ dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); if (dev->nvlink.pvd) { diff --git a/include/npu2.h b/include/npu2.h index c5c5b43843c3..bf7fb6927dd4 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -119,7 +119,8 @@ struct npu2_dev { uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; - struct npu2_bar bars[2]; + struct npu2_bar ntl_bar; + struct npu2_bar genid_bar; struct npu2 *npu; uint32_t bdfn; @@ -150,6 +151,7 @@ struct npu2 { uint32_t chip_id; uint64_t xscom_base; void *regs; + uint64_t regs_size; uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; From patchwork Wed Dec 12 06:58:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011596 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F748399zz9s47 for ; Wed, 12 Dec 2018 18:00:24 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F7481HR5zDqvN for ; Wed, 12 Dec 2018 18:00:24 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73B4WmnzDqgl for ; Wed, 12 Dec 2018 17:59:34 +1100 (AEDT) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6xKFb041124 for ; Wed, 12 Dec 2018 01:59:32 -0500 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pau8nmyt7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:32 -0500 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:29 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:27 -0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xQYg8192484 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:26 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A203B4204B; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0898542042; Wed, 12 Dec 2018 06:59:26 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:25 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id A1281A030E; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:47 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0016-0000-0000-000002354A7F X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0017-0000-0000-0000328D7BCB Message-Id: <8e7850ca3aa8928a3e93164ed5a3147cbd79afe5.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 04/13] hw/npu2: Simplify npu2_write_bar() X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Now that we've moved most of the BAR assignment code into common code and we have an existing struct npu2 everywhere we need it, we don't need the gcid and scom parameters to npu2_write_bar() any more. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-common.c | 14 +++++--------- hw/npu2-opencapi.c | 4 ++-- hw/npu2.c | 6 +++--- include/npu2.h | 3 +-- 4 files changed, 11 insertions(+), 16 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index b140e9ffd064..6cbae9bffaf9 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -141,8 +141,7 @@ void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar) } } -void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, - uint32_t scom) +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar) { uint64_t reg, val; int block; @@ -168,10 +167,7 @@ void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { reg = NPU2_REG_OFFSET(0, block, bar->reg); - if (p) - npu2_write(p, reg, val); - else - npu2_scom_write(gcid, scom, reg, NPU2_MISC_DA_LEN_8B, val); + npu2_write(p, reg, val); } } @@ -200,7 +196,7 @@ static void assign_bars(struct npu2 *npu) for (i = 0; i < ARRAY_SIZE(phy_bars); i++) { bar = &phy_bars[i]; npu2_get_bar(npu->chip_id, bar); - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); } /* Device BARs */ @@ -221,7 +217,7 @@ static void assign_bars(struct npu2 *npu) NPU2_NTL0_BAR : NPU2_NTL1_BAR); bar->flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; npu2_get_bar(npu->chip_id, bar); - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); /* GENID BAR */ bar = &dev->genid_bar; @@ -234,7 +230,7 @@ static void assign_bars(struct npu2 *npu) bar->size = 0x10000; if (NPU2DEV_BRICK(dev)) bar->base += 0x10000; - npu2_write_bar(npu, bar, npu->chip_id, npu->xscom_base); + npu2_write_bar(npu, bar); }; /* Global MMIO BAR */ diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index b374a1035ac9..cc7c403351ce 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -742,7 +742,7 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); dev->ntl_bar.enabled = true; - npu2_write_bar(dev->npu, &dev->ntl_bar, gcid, scom_base); + npu2_write_bar(dev->npu, &dev->ntl_bar); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, dev->ntl_bar.base >> 16); reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(dev->ntl_bar.size >> 16)); @@ -1578,7 +1578,7 @@ static void setup_device(struct npu2_dev *dev) setup_afu_mmio_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); /* Procedure 13.1.3.9 - AFU Config BARs */ dev->genid_bar.enabled = true; - npu2_write_bar(dev->npu, &dev->genid_bar, dev->npu->chip_id, dev->npu->xscom_base); + npu2_write_bar(dev->npu, &dev->genid_bar); set_fence_control(dev->npu->chip_id, dev->npu->xscom_base, dev->brick_index, 0b00); diff --git a/hw/npu2.c b/hw/npu2.c index 6aa16a43f803..1e9fb581688f 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -131,7 +131,7 @@ static int64_t npu2_cfg_write_cmd(void *dev, enabled = !!(*data & PCI_CFG_CMD_MEM_EN); ndev->ntl_bar.enabled = enabled; - npu2_write_bar(ndev->npu, &ndev->ntl_bar, 0, 0); + npu2_write_bar(ndev->npu, &ndev->ntl_bar); /* * Enable/disable the GENID BAR. Two bricks share one GENID @@ -146,7 +146,7 @@ static int64_t npu2_cfg_write_cmd(void *dev, /* Enable the BAR if either device requests it enabled, otherwise disable it */ ndev->genid_bar.enabled = ndev->genid_bar.enabled0 || ndev->genid_bar.enabled1; - npu2_write_bar(ndev->npu, &ndev->genid_bar, 0, 0); + npu2_write_bar(ndev->npu, &ndev->genid_bar); return OPAL_PARTIAL; } @@ -220,7 +220,7 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, return OPAL_HARDWARE; } - npu2_write_bar(dev->npu, bar, 0, 0); + npu2_write_bar(dev->npu, bar); } /* To update the config cache */ diff --git a/include/npu2.h b/include/npu2.h index bf7fb6927dd4..64be9f4eb9dd 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -214,8 +214,7 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask); void npu2_get_bar(uint32_t gcid, struct npu2_bar *bar); void npu2_read_bar(struct npu2 *p, struct npu2_bar *bar); -void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar, uint32_t gcid, - uint32_t scom); +void npu2_write_bar(struct npu2 *p, struct npu2_bar *bar); int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); From patchwork Wed Dec 12 06:58:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011600 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F75T2nPgz9s47 for ; Wed, 12 Dec 2018 18:01:33 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F75T1K5vzDr5X for ; Wed, 12 Dec 2018 18:01:33 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73D5cGRzDqcG for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wRKP009936 for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pauvebq91-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:31 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSoR3342838 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2333F52054; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 7EAC35204F; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id A6588A034A; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:48 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0028-0000-0000-000003294E36 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0029-0000-0000-000023E576C5 Message-Id: <594c291fed20a99b05d443086746dc0fead9583e.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=863 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 05/13] hw/npu2: Don't repopulate NPU devices in NVLink init path X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" In 68415d5e38ef ("hw/npu2: Common NPU2 init routine between NVLink and OpenCAPI") we refactored a large chunk of the NPU init path into common code, including walking the device tree to setup npu2_devs based on the ibm,npu-link device tree nodes. We didn't actually remove the code that does that in npu2_populate_devices(), so currently we populate the devices in the common setup path, then repopulate them incorrectly in the NVLink setup path. Currently this is fine, because we don't support having both NVLink and OpenCAPI devices on the same NPU, but when we do, this will be a problem. Fix npu2_populate_devices() to only populate additional fields as required for NVLink devices. Rename it to npu2_configure_devices() to better reflect what it now does. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2.c | 43 +++++++++++-------------------------------- 1 file changed, 11 insertions(+), 32 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 1e9fb581688f..07213f9f75e1 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1644,12 +1644,12 @@ static void npu2_populate_cfg(struct npu2_dev *dev) PCI_VIRT_CFG_INIT_RO(pvd, pos + 1, 1, 0); } -static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group) +static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group, int size) { int i; int bdfn = (group << 3); - for (i = 0; i < p->total_devices; i++) { + for (i = 0; i < size; i++) { if ((p->devices[i].bdfn & 0xf8) == (bdfn & 0xf8)) bdfn++; } @@ -1657,45 +1657,25 @@ static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group) return bdfn; } -static void npu2_populate_devices(struct npu2 *p, - struct dt_node *dn) +static void npu2_configure_devices(struct npu2 *p) { struct npu2_dev *dev; - struct dt_node *npu2_dn, *link; - uint32_t npu_phandle, index = 0; - - /* - * Get the npu node which has the links which we expand here - * into pci like devices attached to our emulated phb. - */ - npu_phandle = dt_prop_get_u32(dn, "ibm,npcq"); - npu2_dn = dt_find_by_phandle(dt_root, npu_phandle); - assert(npu2_dn); + uint32_t index = 0; - /* Walk the link@x nodes to initialize devices */ - p->total_devices = 0; - p->phb_nvlink.scan_map = 0; - dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { + for (index = 0; index < p->total_devices; index++) { uint32_t group_id; dev = &p->devices[index]; - dev->type = NPU2_DEV_TYPE_NVLINK; - dev->npu = p; - dev->dt_node = link; - dev->link_index = dt_prop_get_u32(link, "ibm,npu-link-index"); - dev->brick_index = dev->link_index; + if (dev->type != NPU2_DEV_TYPE_NVLINK) + continue; - group_id = dt_prop_get_u32(link, "ibm,npu-group-id"); - dev->bdfn = npu_allocate_bdfn(p, group_id); + group_id = dt_prop_get_u32(dev->dt_node, "ibm,npu-group-id"); + dev->bdfn = npu_allocate_bdfn(p, group_id, index); /* This must be done after calling * npu_allocate_bdfn() */ - p->total_devices++; p->phb_nvlink.scan_map |= 0x1 << ((dev->bdfn & 0xf8) >> 3); - dev->pl_xscom_base = dt_prop_get_u64(link, "ibm,npu-phy"); - dev->lane_mask = dt_prop_get_u32(link, "ibm,npu-lane-mask"); - /* Initialize PCI virtual device */ dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); if (dev->nvlink.pvd) { @@ -1703,8 +1683,6 @@ static void npu2_populate_devices(struct npu2 *p, 0x1 << ((dev->nvlink.pvd->bdfn & 0xf8) >> 3); npu2_populate_cfg(dev); } - - index++; } } @@ -1899,13 +1877,14 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) npu->phb_nvlink.dt_node = dn; npu->phb_nvlink.ops = &npu_ops; npu->phb_nvlink.phb_type = phb_type_npu_v2; + npu->phb_nvlink.scan_map = 0; init_lock(&npu->lock); init_lock(&npu->phb_nvlink.lock); list_head_init(&npu->phb_nvlink.devices); list_head_init(&npu->phb_nvlink.virt_devices); npu2_setup_irqs(npu); - npu2_populate_devices(npu, dn); + npu2_configure_devices(npu); npu2_add_interrupt_map(npu, dn); npu2_add_phb_properties(npu); From patchwork Wed Dec 12 06:58:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011597 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F74X4Bc5z9s47 for ; Wed, 12 Dec 2018 18:00:44 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F74X2p6VzDqxg for ; Wed, 12 Dec 2018 18:00:44 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73C0fNRzDqnZ for ; Wed, 12 Dec 2018 17:59:34 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wQ4E104380 for ; Wed, 12 Dec 2018 01:59:33 -0500 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pavjvjgut-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:33 -0500 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:31 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:28 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xRJB57868500 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DD17E42045; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 89A7B4203F; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B6983A0388; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:49 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0012-0000-0000-000002D7DCE8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0013-0000-0000-0000210D5648 Message-Id: <57019d682bbb2e700baa7900f0d564c8409ec68c.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=758 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 06/13] hw/npu2: Rework npu2_add_interrupt_map() and skip non-NVLink devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Rework npu2_add_interrupt_map() so it only includes NVLink devices. Use the existing struct npu2_devs rather than accessing the device tree. Signed-off-by: Andrew Donnellan --- I don't really know how to test this. --- hw/npu2.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 07213f9f75e1..9e2c7d5fdda4 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1686,11 +1686,11 @@ static void npu2_configure_devices(struct npu2 *p) } } -static void npu2_add_interrupt_map(struct npu2 *p, - struct dt_node *dn) +static void npu2_add_interrupt_map(struct npu2 *p) { - struct dt_node *npu2_dn, *link, *phb_dn; - uint32_t npu2_phandle, index = 0, i; + struct dt_node *phb_dn; + struct npu2_dev *dev; + int index, i = 0, nv_devices = 0; uint32_t icsp = get_ics_phandle(); uint32_t *map; size_t map_size; @@ -1699,23 +1699,27 @@ static void npu2_add_interrupt_map(struct npu2 *p, assert(p->phb_nvlink.dt_node); phb_dn = p->phb_nvlink.dt_node; - npu2_phandle = dt_prop_get_u32(dn, "ibm,npcq"); - npu2_dn = dt_find_by_phandle(dt_root, npu2_phandle); - assert(npu2_dn); - map_size = 7 * sizeof(*map) * p->total_devices; + for (index = 0; index < p->total_devices; index++) { + if (p->devices[index].type == NPU2_DEV_TYPE_NVLINK) + nv_devices++; + } + + map_size = 7 * sizeof(*map) * nv_devices; map = malloc(map_size); - index = 0; - dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { - i = index * 7; - map[i + 0] = (p->devices[index].bdfn << 8); + + for (index = 0; index < p->total_devices; index++) { + dev = &p->devices[index]; + if (dev->type != NPU2_DEV_TYPE_NVLINK) + continue; + + map[i + 0] = (dev->bdfn << 8); map[i + 1] = 0; map[i + 2] = 0; - map[i + 3] = 1; /* INT A */ map[i + 4] = icsp; /* interrupt-parent */ - map[i + 5] = p->base_lsi + (index * 2) + 1; /* NDL No-Stall Event */ + map[i + 5] = p->base_lsi + (dev->brick_index * 2) + 1; /* NDL No-Stall Event */ map[i + 6] = 0; /* 0 = EDGE, 1 = LEVEL. */ - index++; + i += 7; } dt_add_property(phb_dn, "interrupt-map", map, map_size); free(map); @@ -1885,7 +1889,7 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) npu2_setup_irqs(npu); npu2_configure_devices(npu); - npu2_add_interrupt_map(npu, dn); + npu2_add_interrupt_map(npu); npu2_add_phb_properties(npu); slot = npu2_slot_create(&npu->phb_nvlink); From patchwork Wed Dec 12 06:58:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011604 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F7745kHjz9s5c for ; Wed, 12 Dec 2018 18:02:56 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F7743WgyzDqvL for ; Wed, 12 Dec 2018 18:02:56 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73F0bpXzDqnv for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wSOP022872 for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pawgng3ac-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:32 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSTn1114406 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 67BABAE053; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C2166AE04D; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id BD6FBA0389; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:50 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0028-0000-0000-000003294E37 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0029-0000-0000-000023E576C6 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 07/13] hw/npu2: Make IRQ setup code common X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Frederic Barrat The NPU IRQ setup code is currently duplicated between NVLink and OpenCAPI. Apart from the NPU2_MISC_IRQ_ENABLE2 register, the setup of IRQs is basically identical between NVLink and OpenCAPI. Move the NVLink IRQ setup code into the common path and get rid of all the OpenCAPI IRQ setup apart from the enable mask. Signed-off-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- Might need some testing --- hw/npu2-common.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 50 +-------------------- hw/npu2.c | 100 +----------------------------------------- include/npu2.h | 1 +- 4 files changed, 116 insertions(+), 147 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 6cbae9bffaf9..8edf761cf56b 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -24,6 +24,12 @@ #include #include #include +#include +#include + +#define NPU2_IRQ_BASE_SHIFT 13 +#define NPU2_N_DL_IRQS 35 +#define NPU2_N_DL_IRQS_ALIGN 64 /* * We use the indirect method because it uses the same addresses as @@ -244,6 +250,111 @@ static void assign_bars(struct npu2 *npu) npu->mm_size = last_bar.base + last_bar.size - npu->mm_base; } +static uint64_t npu2_ipi_attributes(struct irq_source *is __unused, uint32_t isn __unused) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + + if (idx == 18) + /* TCE Interrupt - used to detect a frozen PE */ + return IRQ_ATTR_TARGET_OPAL | IRQ_ATTR_TARGET_RARE | IRQ_ATTR_TYPE_MSI; + else + return IRQ_ATTR_TARGET_LINUX; +} + +static char *npu2_ipi_name(struct irq_source *is, uint32_t isn) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + const char *name; + + switch (idx) { + case 0: name = "NDL 0 Stall Event (brick 0)"; break; + case 1: name = "NDL 0 No-Stall Event (brick 0)"; break; + case 2: name = "NDL 1 Stall Event (brick 1)"; break; + case 3: name = "NDL 1 No-Stall Event (brick 1)"; break; + case 4: name = "NDL 2 Stall Event (brick 2)"; break; + case 5: name = "NDL 2 No-Stall Event (brick 2)"; break; + case 6: name = "NDL 5 Stall Event (brick 3)"; break; + case 7: name = "NDL 5 No-Stall Event (brick 3)"; break; + case 8: name = "NDL 4 Stall Event (brick 4)"; break; + case 9: name = "NDL 4 No-Stall Event (brick 4)"; break; + case 10: name = "NDL 3 Stall Event (brick 5)"; break; + case 11: name = "NDL 3 No-Stall Event (brick 5)"; break; + case 12: name = "NTL 0 Event"; break; + case 13: name = "NTL 1 Event"; break; + case 14: name = "NTL 2 Event"; break; + case 15: name = "NTL 3 Event"; break; + case 16: name = "NTL 4 Event"; break; + case 17: name = "NTL 5 Event"; break; + case 18: name = "TCE Event"; break; + case 19: name = "ATS Event"; break; + case 20: name = "CQ Event"; break; + case 21: name = "MISC Event"; break; + case 22: name = "NMMU Local Xstop"; break; + case 23: name = "Translate Fail (brick 2)"; break; + case 24: name = "Translate Fail (brick 3)"; break; + case 25: name = "Translate Fail (brick 4)"; break; + case 26: name = "Translate Fail (brick 5)"; break; + case 27: name = "OTL Event (brick 2)"; break; + case 28: name = "OTL Event (brick 3)"; break; + case 29: name = "OTL Event (brick 4)"; break; + case 30: name = "OTL Event (brick 5)"; break; + case 31: name = "XSL Event (brick 2)"; break; + case 32: name = "XSL Event (brick 3)"; break; + case 33: name = "XSL Event (brick 4)"; break; + case 34: name = "XSL Event (brick 5)"; break; + default: name = "Unknown"; + } + return strdup(name); +} + +static void npu2_err_interrupt(struct irq_source *is, uint32_t isn) +{ + struct npu2 *p = is->data; + uint32_t idx = isn - p->base_lsi; + + if (idx != 18) { + prerror("OPAL received unknown NPU2 interrupt %d\n", idx); + return; + } + + opal_update_pending_evt(OPAL_EVENT_PCI_ERROR, + OPAL_EVENT_PCI_ERROR); +} + +static const struct irq_source_ops npu2_ipi_ops = { + .interrupt = npu2_err_interrupt, + .attributes = npu2_ipi_attributes, + .name = npu2_ipi_name, +}; + +static void setup_irqs(struct npu2 *p) +{ + uint64_t reg, val; + void *tp; + + p->base_lsi = xive_alloc_ipi_irqs(p->chip_id, NPU2_N_DL_IRQS, NPU2_N_DL_IRQS_ALIGN); + if (p->base_lsi == XIVE_IRQ_ERROR) { + prlog(PR_ERR, "NPU: Failed to allocate interrupt sources\n"); + return; + } + xive_register_ipi_source(p->base_lsi, NPU2_N_DL_IRQS, p, &npu2_ipi_ops); + + /* Set IPI configuration */ + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_CFG); + val = npu2_read(p, reg); + val = SETFIELD(NPU2_MISC_CFG_IPI_PS, val, NPU2_MISC_CFG_IPI_PS_64K); + val = SETFIELD(NPU2_MISC_CFG_IPI_OS, val, NPU2_MISC_CFG_IPI_OS_AIX); + npu2_write(p, reg, val); + + /* Set IRQ base */ + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_IRQ_BASE); + tp = xive_get_trigger_port(p->base_lsi); + val = ((uint64_t)tp) << NPU2_IRQ_BASE_SHIFT; + npu2_write(p, reg, val); +} + static bool _i2c_presence_detect(struct npu2_dev *dev) { uint8_t state, data; @@ -421,6 +532,7 @@ static void setup_devices(struct npu2 *npu) } assign_bars(npu); + setup_irqs(npu); if (nvlink_detected) npu2_nvlink_init_npu(npu); diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index cc7c403351ce..dd40b24009f3 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -48,11 +48,9 @@ #include #include #include -#include #include #include -#define NPU_IRQ_LEVELS 35 #define NPU_IRQ_LEVELS_XSL 23 #define MAX_PE_HANDLE ((1 << 15) - 1) #define TL_MAX_TEMPLATE 63 @@ -1383,7 +1381,7 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, * Pass the hw irq number for the translation fault irq * irq levels 23 -> 26 are for translation faults, 1 per brick */ - irq = dev->npu->irq_base + NPU_IRQ_LEVELS_XSL; + irq = dev->npu->base_lsi + NPU_IRQ_LEVELS_XSL; if (stacku == NPU2_STACK_STCK_2U) irq += 2; if (block == NPU2_BLOCK_OTL1) @@ -1456,43 +1454,9 @@ static void mask_nvlink_fir(struct npu2 *p) NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); } -static int setup_irq(struct npu2 *p) +static int enable_xsl_irq(struct npu2 *p) { - uint64_t reg, mmio_addr; - uint32_t base; - - base = xive_alloc_ipi_irqs(p->chip_id, NPU_IRQ_LEVELS, 64); - if (base == XIVE_IRQ_ERROR) { - /** - * @fwts-label OCAPIIRQAllocationFailed - * @fwts-advice OpenCAPI IRQ setup failed. This is probably - * a firmware bug. OpenCAPI functionality will be broken. - */ - prlog(PR_ERR, "OCAPI: Couldn't allocate interrupts for NPU\n"); - return -1; - } - p->irq_base = base; - - xive_register_ipi_source(base, NPU_IRQ_LEVELS, NULL, NULL); - mmio_addr = (uint64_t ) xive_get_trigger_port(base); - prlog(PR_DEBUG, "OCAPI: NPU base irq %d @%llx\n", base, mmio_addr); - reg = (mmio_addr & NPU2_MISC_IRQ_BASE_MASK) << 13; - npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_BASE, - NPU2_MISC_DA_LEN_8B, reg); - /* - * setup page size = 64k - * - * OS type is set to AIX: opal also runs with 2 pages per interrupt, - * so to cover the max offset for 35 levels of interrupt, we need - * bits 41 to 46, which is what the AIX setting does. There's no - * other meaning for that AIX setting. - */ - reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_CFG, - NPU2_MISC_DA_LEN_8B); - reg |= NPU2_MISC_CFG_IPI_PS; - reg &= ~NPU2_MISC_CFG_IPI_OS; - npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_CFG, - NPU2_MISC_DA_LEN_8B, reg); + uint64_t reg; /* enable translation interrupts for all bricks */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE2, @@ -1619,7 +1583,6 @@ static void read_nvram_training_state(void) int npu2_opencapi_init_npu(struct npu2 *npu) { struct npu2_dev *dev; - int rc; assert(platform.ocapi); read_nvram_training_state(); @@ -1645,10 +1608,7 @@ int npu2_opencapi_init_npu(struct npu2 *npu) address_translation_config(npu->chip_id, npu->xscom_base, dev->brick_index); } - /* Procedure 13.1.3.10 - Interrupt Configuration */ - rc = setup_irq(npu); - if (rc) - goto failed; + enable_xsl_irq(npu); for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; @@ -1658,8 +1618,6 @@ int npu2_opencapi_init_npu(struct npu2 *npu) } return 0; -failed: - return -1; } static const struct phb_ops npu2_opencapi_ops = { diff --git a/hw/npu2.c b/hw/npu2.c index 9e2c7d5fdda4..8a4611424609 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -20,7 +20,6 @@ #include #include #include -#include #include #include #include @@ -36,14 +35,9 @@ #include #include #include -#include #include #include -#define NPU2_IRQ_BASE_SHIFT 13 -#define NPU2_N_DL_IRQS 23 -#define NPU2_N_DL_IRQS_ALIGN 64 - #define VENDOR_CAP_START 0x80 #define VENDOR_CAP_END 0x90 #define VENDOR_CAP_LEN 0x10 @@ -1780,99 +1774,6 @@ static void npu2_add_phb_properties(struct npu2 *p) hi32(mm_size), lo32(mm_size)); } -static uint64_t npu2_ipi_attributes(struct irq_source *is __unused, uint32_t isn __unused) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - - if (idx == 18) - /* TCE Interrupt - used to detect a frozen PE */ - return IRQ_ATTR_TARGET_OPAL | IRQ_ATTR_TARGET_RARE | IRQ_ATTR_TYPE_MSI; - else - return IRQ_ATTR_TARGET_LINUX; -} - -static char *npu2_ipi_name(struct irq_source *is, uint32_t isn) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - const char *name; - - switch (idx) { - case 0: name = "NDL 0 Stall Event (brick 0)"; break; - case 1: name = "NDL 0 No-Stall Event (brick 0)"; break; - case 2: name = "NDL 1 Stall Event (brick 1)"; break; - case 3: name = "NDL 1 No-Stall Event (brick 1)"; break; - case 4: name = "NDL 2 Stall Event (brick 2)"; break; - case 5: name = "NDL 2 No-Stall Event (brick 2)"; break; - case 6: name = "NDL 5 Stall Event (brick 3)"; break; - case 7: name = "NDL 5 No-Stall Event (brick 3)"; break; - case 8: name = "NDL 4 Stall Event (brick 4)"; break; - case 9: name = "NDL 4 No-Stall Event (brick 4)"; break; - case 10: name = "NDL 3 Stall Event (brick 5)"; break; - case 11: name = "NDL 3 No-Stall Event (brick 5)"; break; - case 12: name = "NTL 0 Event"; break; - case 13: name = "NTL 1 Event"; break; - case 14: name = "NTL 2 Event"; break; - case 15: name = "NTL 3 Event"; break; - case 16: name = "NTL 4 Event"; break; - case 17: name = "NTL 5 Event"; break; - case 18: name = "TCE Event"; break; - case 19: name = "ATS Event"; break; - case 20: name = "CQ Event"; break; - case 21: name = "MISC Event"; break; - case 22: name = "NMMU Local Xstop"; break; - default: name = "Unknown"; - } - return strdup(name); -} - -static void npu2_err_interrupt(struct irq_source *is, uint32_t isn) -{ - struct npu2 *p = is->data; - uint32_t idx = isn - p->base_lsi; - - if (idx != 18) { - prerror("OPAL received unknown NPU2 interrupt %d\n", idx); - return; - } - - opal_update_pending_evt(OPAL_EVENT_PCI_ERROR, - OPAL_EVENT_PCI_ERROR); -} - -static const struct irq_source_ops npu2_ipi_ops = { - .interrupt = npu2_err_interrupt, - .attributes = npu2_ipi_attributes, - .name = npu2_ipi_name, -}; - -static void npu2_setup_irqs(struct npu2 *p) -{ - uint64_t reg, val; - void *tp; - - p->base_lsi = xive_alloc_ipi_irqs(p->chip_id, NPU2_N_DL_IRQS, NPU2_N_DL_IRQS_ALIGN); - if (p->base_lsi == XIVE_IRQ_ERROR) { - prlog(PR_ERR, "NPU: Failed to allocate interrupt sources, IRQs for NDL No-stall events will not be available.\n"); - return; - } - xive_register_ipi_source(p->base_lsi, NPU2_N_DL_IRQS, p, &npu2_ipi_ops ); - - /* Set IPI configuration */ - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_CFG); - val = npu2_read(p, reg); - val = SETFIELD(NPU2_MISC_CFG_IPI_PS, val, NPU2_MISC_CFG_IPI_PS_64K); - val = SETFIELD(NPU2_MISC_CFG_IPI_OS, val, NPU2_MISC_CFG_IPI_OS_AIX); - npu2_write(p, reg, val); - - /* Set IRQ base */ - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_IRQ_BASE); - tp = xive_get_trigger_port(p->base_lsi); - val = ((uint64_t)tp) << NPU2_IRQ_BASE_SHIFT; - npu2_write(p, reg, val); -} - void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) { struct pci_slot *slot; @@ -1887,7 +1788,6 @@ void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) list_head_init(&npu->phb_nvlink.devices); list_head_init(&npu->phb_nvlink.virt_devices); - npu2_setup_irqs(npu); npu2_configure_devices(npu); npu2_add_interrupt_map(npu); npu2_add_phb_properties(npu); diff --git a/include/npu2.h b/include/npu2.h index 64be9f4eb9dd..9eb436db524a 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -155,7 +155,6 @@ struct npu2 { uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; - uint32_t irq_base; uint32_t total_devices; struct npu2_dev *devices; enum phys_map_type gpu_map_type; From patchwork Wed Dec 12 06:58:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011601 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F75n6Mycz9s5c for ; Wed, 12 Dec 2018 18:01:49 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F75n511qzDr4F for ; Wed, 12 Dec 2018 18:01:49 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73D5cGpzDqgl for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wTw0023035 for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pawgng3ae-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:32 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSGP8782186 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6B4A14C058; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C48CA4C04E; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id CF421A038A; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:51 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0008-0000-0000-000002A02346 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0009-0000-0000-0000220A9E41 Message-Id: <98baba04c43bce19f397e9d720ed39a0b10c4aa2.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 08/13] platforms/astbmc/witherspoon: Rework NPU presence detection X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Rework NPU presence detection in preparation for supporting both NVLink and OpenCAPI devices operating simultaneously on the same NPU. If an OpenCAPI card is connected to GPU#0, and an NVLink GPU is connected to GPU#1, the GPU will only receive 2 links rather than the usual 3. The reason for this is that without the OpenCAPI card, the GPU would be use links 3-5, connected to NPU bricks 3-5, which needs both stacks 1 and 2 to be in NVLink mode. However, with an OpenCAPI card in the GPU#0 slot that uses links 0-1, we need to use NPU bricks 2-3, which means stack 1 must be set in OpenCAPI mode. As such, the GPU will be restricted to using links 4 and 5. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- platforms/astbmc/witherspoon.c | 62 ++++++++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 15 deletions(-) diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index fe138991696f..c41f0c5b1971 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -233,6 +233,8 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) int rc; bool gpu0_present, gpu1_present; + enum npu2_dev_type gpu0_type = NPU2_DEV_TYPE_UNKNOWN; + enum npu2_dev_type gpu1_type = NPU2_DEV_TYPE_UNKNOWN; if (witherspoon_type != WITHERSPOON_TYPE_REDBUD) { prlog(PR_DEBUG, "PLAT: Setting all NPU links to NVLink, OpenCAPI only supported on Redbud\n"); @@ -298,19 +300,11 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) if (state & (1 << 0)) { prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is OpenCAPI\n", chip->id); - /* - * On witherspoon, bricks 2 and 3 are connected to - * the lanes matching links 0 and 1 in OpenCAPI mode. - */ - set_link_details(npu, 1, 3, NPU2_DEV_TYPE_OPENCAPI); - /* We current don't support using the second link */ - set_link_details(npu, 0, 2, NPU2_DEV_TYPE_UNKNOWN); + gpu0_type = NPU2_DEV_TYPE_OPENCAPI; } else { prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is NVLink\n", chip->id); - set_link_details(npu, 0, 0, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 1, 1, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 2, 2, NPU2_DEV_TYPE_NVLINK); + gpu0_type = NPU2_DEV_TYPE_NVLINK; } } @@ -318,16 +312,54 @@ static void witherspoon_npu2_device_detect(struct npu2 *npu) if (state & (1 << 1)) { prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is OpenCAPI\n", chip->id); - set_link_details(npu, 4, 4, NPU2_DEV_TYPE_OPENCAPI); - /* We current don't support using the second link */ - set_link_details(npu, 5, 5, NPU2_DEV_TYPE_UNKNOWN); + gpu1_type = NPU2_DEV_TYPE_OPENCAPI; } else { prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is NVLink\n", chip->id); + gpu1_type = NPU2_DEV_TYPE_NVLINK; + } + } + + if (gpu0_type == NPU2_DEV_TYPE_OPENCAPI) { + set_link_details(npu, 1, 3, NPU2_DEV_TYPE_OPENCAPI); + /* We currently don't support using the second link */ + set_link_details(npu, 0, 2, NPU2_DEV_TYPE_UNKNOWN); + } + + if (gpu0_type == NPU2_DEV_TYPE_NVLINK) { + set_link_details(npu, 0, 0, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 1, 1, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 2, 2, NPU2_DEV_TYPE_NVLINK); + } + + if (gpu1_type == NPU2_DEV_TYPE_OPENCAPI) { + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_OPENCAPI); + /* We currently don't support using the second link */ + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_UNKNOWN); + } + + /* + * If an OpenCAPI card is connected to GPU#0, and an NVLink GPU is + * connected to GPU#1, the GPU will only receive 2 links rather than the + * usual 3. + * + * The reason for this is that without the OpenCAPI card, the GPU would + * be use links 3-5, connected to NPU bricks 3-5, which needs both + * stacks 1 and 2 to be in NVLink mode. + * + * However, with an OpenCAPI card in the GPU#0 slot that uses links 0-1, + * we need to use NPU bricks 2-3, which means stack 1 must be set in + * OpenCAPI mode. As such, the GPU will be restricted to using links 4 + * and 5. + */ + if (gpu1_type == NPU2_DEV_TYPE_NVLINK) { + if (gpu0_type == NPU2_DEV_TYPE_OPENCAPI) { + prlog(PR_WARNING, "PLAT: Chip %d GPU#1 will operate at reduced performance due to presence of OpenCAPI device. For optimal performance, swap device locations\n", chip->id); + } else { set_link_details(npu, 3, 3, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 4, 4, NPU2_DEV_TYPE_NVLINK); - set_link_details(npu, 5, 5, NPU2_DEV_TYPE_NVLINK); } + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_NVLINK); } return; From patchwork Wed Dec 12 06:58:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011605 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F77Q1xMWz9s47 for ; Wed, 12 Dec 2018 18:03:14 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F77Q0QZ4zDr4t for ; Wed, 12 Dec 2018 18:03:14 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73H1BYkzDqcG for ; Wed, 12 Dec 2018 17:59:39 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wX0Z104611 for ; Wed, 12 Dec 2018 01:59:37 -0500 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pavjvjgve-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:36 -0500 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:32 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSwX56951018 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 64AAC52052; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id BF76152050; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id DEC54A038B; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:52 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0020-0000-0000-000002F5CB10 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0021-0000-0000-00002145F54F Message-Id: <6142a5f24a1277d8e70b1ce54bee07c6468c9a50.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 09/13] hw/npu2: Combined NPU brick configuration procedure X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" To support the case where we have both OpenCAPI and NVLink devices connected to the same NPU, we need to be able to configure each NPU stack/brick separately depending on the device type it is connected to. Refactor the existing code so that we can set the transport muxes and other relevant NPU registers on a per-brick/stack basis. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- alistair: I've managed to do it before 2020! --- hw/npu2-common.c | 126 +++++++++++++++++++++++++++++++++++++++++++++- hw/npu2-opencapi.c | 109 +--------------------------------------- hw/npu2.c | 121 +++++++++++++++++++++++-------------------- include/npu2-regs.h | 4 +- include/npu2.h | 1 +- 5 files changed, 196 insertions(+), 165 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 8edf761cf56b..9c383ef7f35f 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -409,6 +409,130 @@ void npu2_i2c_presence_detect(struct npu2 *npu) } } +/* Procedure 13.1.3.1, OpenCAPI NPU Workbook - Select OCAPI vs NVLink */ +static void set_brick_config(struct npu2 *npu) +{ + /* Step 1 - Set Transport MUX controls to select correct OTL or NTL */ + uint64_t reg; + uint64_t ndlmux_brk0to2, ocmux_brk0to1, ocmux_brk4to5; + uint64_t phy_config_scom; + struct npu2_dev *dev; + + prlog(PR_DEBUG, "NPU: %s: Setting transport mux controls\n", __func__); + + /* + * Disable NVLink link layers on PHYs being used for OpenCAPI. + * Experience suggests that this needs to be done early + * (before setting the transport muxes) otherwise we get link + * internal errors. + */ + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + switch (dev->brick_index) { + case 2: + case 3: + phy_config_scom = OBUS_LL0_IOOL_PHY_CONFIG; + break; + case 4: + case 5: + phy_config_scom = OBUS_LL3_IOOL_PHY_CONFIG; + break; + default: + assert(false); + } + /* Disable NV-Link link layers */ + xscom_read(npu->chip_id, phy_config_scom, ®); + reg &= ~OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED; + xscom_write(npu->chip_id, phy_config_scom, reg); + } + + /* Optical IO Transport Mux Config for Bricks 0-2 and 4-5 */ + reg = npu2_scom_read(npu->chip_id, npu->xscom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B); + ndlmux_brk0to2 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); + ocmux_brk0to1 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); + ocmux_brk4to5 = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type == NPU2_DEV_TYPE_UNKNOWN) + continue; + + switch (dev->brick_index) { + case 0: /* NTL0.0 */ + assert(dev->type == NPU2_DEV_TYPE_NVLINK); + ndlmux_brk0to2 |= 0b100; + break; + case 1: /* NTL0.1 */ + assert(dev->type == NPU2_DEV_TYPE_NVLINK); + ndlmux_brk0to2 |= 0b010; + break; + case 2: /* NTL1.0 / OTL1.0 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ndlmux_brk0to2 &= ~0b100; + ocmux_brk0to1 |= 0b10; + } else { + ndlmux_brk0to2 |= 0b001; + } + break; + case 3: /* NTL1.1 / OTL1.1 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ndlmux_brk0to2 &= ~0b010; + ocmux_brk0to1 |= 0b01; + } + break; + case 4: /* NTL2.0 / OTL2.0 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ocmux_brk4to5 |= 0b10; + } else { + ocmux_brk4to5 &= ~0b10; + } + break; + case 5: /* NTL2.1 / OTL2.1 */ + if (dev->type == NPU2_DEV_TYPE_OPENCAPI) { + ocmux_brk4to5 |= 0b01; + } else { + ocmux_brk4to5 &= ~0b01; + } + break; + default: + assert(false); + } + } + + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, ndlmux_brk0to2); + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, ocmux_brk0to1); + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, ocmux_brk4to5); + npu2_scom_write(npu->chip_id, npu->xscom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B, reg); + + /* + * PowerBus Optical Miscellaneous Config Register + */ + xscom_read(npu->chip_id, PU_IOE_PB_MISC_CFG, ®); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + switch (dev->brick_index) { + case 3: + if (dev->type == NPU2_DEV_TYPE_NVLINK) + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_03_NPU_NOT_PB, reg, 1); + break; + case 4: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB, reg, 1); + break; + case 5: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB, reg, 1); + break; + default: + break; + } + } + xscom_write(npu->chip_id, PU_IOE_PB_MISC_CFG, reg); +} + static struct npu2 *setup_npu(struct dt_node *dn) { struct npu2 *npu; @@ -469,6 +593,7 @@ static struct npu2 *setup_npu(struct dt_node *dn) dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); /* May be overridden by platform presence detection */ dev->brick_index = dev->link_index; + dev->group_id = dt_prop_get_u32(np, "ibm,npu-group-id"); /* Will be overridden by presence detection */ dev->type = NPU2_DEV_TYPE_UNKNOWN; dev->npu = npu; @@ -573,6 +698,7 @@ void probe_npu2(void) if (!npu) continue; platform.npu2_device_detect(npu); + set_brick_config(npu); setup_devices(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index dd40b24009f3..8ecd323416f4 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -196,112 +196,6 @@ static uint64_t get_odl_endpoint_info(uint32_t gcid, uint64_t index) return reg; } -static void disable_nvlink(uint32_t gcid, int index) -{ - uint64_t phy_config_scom, reg; - - switch (index) { - case 2: - case 3: - phy_config_scom = OBUS_LL0_IOOL_PHY_CONFIG; - break; - case 4: - case 5: - phy_config_scom = OBUS_LL3_IOOL_PHY_CONFIG; - break; - default: - assert(false); - } - /* Disable NV-Link link layers */ - xscom_read(gcid, phy_config_scom, ®); - reg &= ~OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED; - reg &= ~OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED; - reg &= ~OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED; - xscom_write(gcid, phy_config_scom, reg); -} - -/* Procedure 13.1.3.1 - select OCAPI vs NVLink for bricks 2-3/4-5 */ - -static void set_transport_mux_controls(uint32_t gcid, uint32_t scom_base, - int index, enum npu2_dev_type type) -{ - /* Step 1 - Set Transport MUX controls to select correct OTL or NTL */ - uint64_t reg; - uint64_t field; - - /* TODO: Rework this to select for NVLink too */ - assert(type == NPU2_DEV_TYPE_OPENCAPI); - - prlog(PR_DEBUG, "OCAPI: %s: Setting transport mux controls\n", __func__); - - /* Optical IO Transport Mux Config for Bricks 0-2 and 4-5 */ - reg = npu2_scom_read(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, - NPU2_MISC_DA_LEN_8B); - switch (index) { - case 0: - case 1: - /* not valid for OpenCAPI */ - assert(false); - break; - case 2: /* OTL1.0 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); - field &= ~0b100; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, - field); - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); - field |= 0b10; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, - field); - break; - case 3: /* OTL1.1 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); - field &= ~0b010; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, - field); - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); - field |= 0b01; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, - field); - break; - case 4: /* OTL2.0 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); - field |= 0b10; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, - field); - break; - case 5: /* OTL2.1 */ - field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); - field |= 0b01; - reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, - field); - break; - default: - assert(false); - } - npu2_scom_write(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, - NPU2_MISC_DA_LEN_8B, reg); - - /* - * PowerBus Optical Miscellaneous Config Register - select - * OpenCAPI for b4/5 and A-Link for b3 - */ - xscom_read(gcid, PU_IOE_PB_MISC_CFG, ®); - switch (index) { - case 0: - case 1: - case 2: - case 3: - break; - case 4: - reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB, reg, 1); - break; - case 5: - reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB, reg, 1); - break; - } - xscom_write(gcid, PU_IOE_PB_MISC_CFG, reg); -} - static void enable_odl_phy_mux(uint32_t gcid, int index) { uint64_t reg; @@ -579,9 +473,6 @@ static void brick_config(uint32_t gcid, uint32_t scom_base, int index) * We assume at this point that the PowerBus Hotplug Mode Control * register is correctly set by Hostboot */ - disable_nvlink(gcid, index); - set_transport_mux_controls(gcid, scom_base, index, - NPU2_DEV_TYPE_OPENCAPI); enable_odl_phy_mux(gcid, index); disable_alink_fp(gcid); enable_xsl_clocks(gcid, scom_base, index); diff --git a/hw/npu2.c b/hw/npu2.c index 8a4611424609..9e60b4e16f29 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1352,6 +1352,62 @@ static const struct phb_ops npu_ops = { .tce_kill = npu2_tce_kill, }; +static void enable_nvlink(struct npu2_dev *dev) +{ + struct npu2 *npu = dev->npu; + int stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); + int block; + uint64_t addr, val; + + /* CQ_SM Misc Config #0 - enable NVLink mode */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + addr = NPU2_REG_OFFSET(stack, block, NPU2_CQ_SM_MISC_CFG0); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_SM_MISC_CFG0_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + } + + /* CQ_CTL Misc Config #0 - enable NVLink mode */ + addr = NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, NPU2_CQ_CTL_MISC_CFG); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_CTL_MISC_CFG_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* CQ_DAT Misc Config #1 - enable NVLink mode */ + addr = NPU2_REG_OFFSET(stack, NPU2_BLOCK_DAT, NPU2_CQ_DAT_MISC_CFG); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_CQ_DAT_MISC_CFG_CONFIG_NVLINK_MODE; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* NTL Misc Config 2 - enable NTL brick and checks */ + addr = NPU2_NTL_MISC_CFG2(dev); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val |= NPU2_NTL_MISC_CFG2_BRICK_ENABLE; + val |= NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA; + val |= NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA; + val |= NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + + /* High Water Marks */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + addr = NPU2_REG_OFFSET(stack, block, NPU2_HIGH_WATER_MARKS); + val = npu2_scom_read(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B); + val &= ~NPU2_HIGH_WATER_MARKS_PWR0; + val |= PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); + npu2_scom_write(npu->chip_id, npu->xscom_base, addr, + NPU2_MISC_DA_LEN_8B, val); + } +} + /* * Set up NPU for NVLink and create PCI root device node * accordingly. @@ -1359,59 +1415,12 @@ static const struct phb_ops npu_ops = { int npu2_nvlink_init_npu(struct npu2 *npu) { struct dt_node *np; - uint64_t reg[2], mm_win[2], val; + uint64_t reg[2], mm_win[2]; - /* TODO: Clean this up with register names, etc. when we get - * time. This just turns NVLink mode on in each brick and should - * get replaced with a patch from ajd once we've worked out how - * things are going to work there. - * - * Obviously if the year is now 2020 that didn't happen and you - * should fix this :-) */ - xscom_write_mask(npu->chip_id, 0x5011000, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011030, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011060, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011090, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011200, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011230, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011260, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011290, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011400, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011430, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011460, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(npu->chip_id, 0x5011490, PPC_BIT(58), PPC_BIT(58)); - - xscom_write_mask(npu->chip_id, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(npu->chip_id, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(npu->chip_id, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(npu->chip_id, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); - - val = NPU2_NTL_MISC_CFG2_BRICK_ENABLE | - NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA | - NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA | - NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; - xscom_write_mask(npu->chip_id, 0x5011110, val, val); - xscom_write_mask(npu->chip_id, 0x5011130, val, val); - xscom_write_mask(npu->chip_id, 0x5011310, val, val); - xscom_write_mask(npu->chip_id, 0x5011330, val, val); - xscom_write_mask(npu->chip_id, 0x5011510, val, val); - xscom_write_mask(npu->chip_id, 0x5011530, val, val); - - val = PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); - xscom_write_mask(npu->chip_id, 0x5011009, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011039, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011069, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011099, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011209, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011239, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011269, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011299, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011409, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011439, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); - xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); + for (int i = 0; i < npu->total_devices; i++) { + if (npu->devices[i].type == NPU2_DEV_TYPE_NVLINK) + enable_nvlink(&npu->devices[i]); + } /* Populate PCI root device node */ reg[0] = (uint64_t)npu->regs; @@ -1644,6 +1653,9 @@ static uint32_t npu_allocate_bdfn(struct npu2 *p, uint32_t group, int size) int bdfn = (group << 3); for (i = 0; i < size; i++) { + if (p->devices[i].type != NPU2_DEV_TYPE_NVLINK) + continue; + if ((p->devices[i].bdfn & 0xf8) == (bdfn & 0xf8)) bdfn++; } @@ -1657,14 +1669,11 @@ static void npu2_configure_devices(struct npu2 *p) uint32_t index = 0; for (index = 0; index < p->total_devices; index++) { - uint32_t group_id; - dev = &p->devices[index]; if (dev->type != NPU2_DEV_TYPE_NVLINK) continue; - group_id = dt_prop_get_u32(dev->dt_node, "ibm,npu-group-id"); - dev->bdfn = npu_allocate_bdfn(p, group_id, index); + dev->bdfn = npu_allocate_bdfn(p, dev->group_id, index); /* This must be done after calling * npu_allocate_bdfn() */ diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 8273b2be26a2..a8f59dcdbe2f 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -119,6 +119,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_SM_MISC_CFG0 0x000 #define NPU2_CQ_SM_MISC_CFG0_CONFIG_ENABLE_PBUS PPC_BIT(38) #define NPU2_CQ_SM_MISC_CFG0_CONFIG_OCAPI_MODE PPC_BIT(57) +#define NPU2_CQ_SM_MISC_CFG0_CONFIG_NVLINK_MODE PPC_BIT(58) #define NPU2_CQ_SM_MISC_CFG1 0x008 #define NPU2_CQ_SM_MISC_CFG2 0x148 #define NPU2_PB_EPSILON 0x010 @@ -152,6 +153,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_LOW_WATER_MARKS 0x040 #define NPU2_LOW_WATER_MARKS_ENABLE_MACHINE_ALLOC PPC_BIT(51) #define NPU2_HIGH_WATER_MARKS 0x048 +#define NPU2_HIGH_WATER_MARKS_PWR0 PPC_BITMASK(6, 11); #define NPU2_RELAXED_ORDERING_CFG(n) (0x050 + (n)*8) #define NPU2_RELAXED_ORDERING_SOURCE(n) (PPC_BITMASK(0,31) >> ((n)*32)) #define NPU2_RELAXED_ORDERING_SOURCE_ENA PPC_BITMASK32(0,3) @@ -207,6 +209,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* CTL block registers */ #define NPU2_CQ_CTL_MISC_CFG 0x000 #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(52) +#define NPU2_CQ_CTL_MISC_CFG_CONFIG_NVLINK_MODE PPC_BIT(53) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL0_ENABLE PPC_BIT(55) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL1_ENABLE PPC_BIT(56) #define NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG 0x0B0 @@ -261,6 +264,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* DAT block registers */ #define NPU2_CQ_DAT_MISC_CFG 0x008 #define NPU2_CQ_DAT_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(40) +#define NPU2_CQ_DAT_MISC_CFG_CONFIG_NVLINK_MODE PPC_BIT(41) #define NPU2_CQ_DAT_ECC_CFG 0x010 #define NPU2_CQ_DAT_SCRATCH0 0x018 #define NPU2_CQ_DAT_ECC_STATUS 0x020 diff --git a/include/npu2.h b/include/npu2.h index 9eb436db524a..8f4747006980 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -117,6 +117,7 @@ struct npu2_dev { enum npu2_dev_type type; uint32_t link_index; uint32_t brick_index; + uint32_t group_id; uint64_t pl_xscom_base; struct dt_node *dt_node; struct npu2_bar ntl_bar; From patchwork Wed Dec 12 06:58:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011599 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F75B3JKNz9s47 for ; Wed, 12 Dec 2018 18:01:18 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F75B1vQnzDr13 for ; Wed, 12 Dec 2018 18:01:18 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73D5pSHzDqnZ for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wN93104232 for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pavjvjgvf-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:32 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSM6852416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 714D4A405B; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CB3ACA4040; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id EE0BDA0399; Wed, 12 Dec 2018 17:59:24 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:53 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-4275-0000-0000-000002EF53CB X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-4276-0000-0000-000037FC7452 Message-Id: <7094690ae0a8a43bc100f16dd4d4e6b9f1c0fc2a.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=261 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 10/13] hw/npu2-opencapi: FIR masking for mixed setups X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" When setting up an NPU with an OpenCAPI device, we need to mask the FIR bits for NDL Stall/NoStall, which are used for a different purpose on OpenCAPI vs NVLink. Currently, we just mask the bits for all links/DLs. When we support mixed setups of OpenCAPI + NVLink on the same NPU, we don't want to mask all the bits. Only mask the FIR bits for the specific links which are OpenCAPI. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-opencapi.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 8ecd323416f4..d79727ad4199 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1303,7 +1303,8 @@ static void npu2_opencapi_final_fixup(struct phb *phb) static void mask_nvlink_fir(struct npu2 *p) { - uint64_t reg; + uint64_t reg, mask = 0ull; + int link_num; /* * From section 13.1.3.10 of the NPU workbook: "the NV-Link @@ -1312,35 +1313,46 @@ static void mask_nvlink_fir(struct npu2 *p) * OpenCAPI. Therefore, the corresponding bits in NPU FIR * Register 1 must be masked and configured to NOT cause the * NPU to go into Freeze or Fence mode or send an Interrupt." - * - * FIXME: will need to revisit when mixing nvlink with - * opencapi. Assumes an opencapi-only setup on both PHYs for - * now. */ + for (int i = 0; i < p-> total_devices; i++) { + struct npu2_dev *dev = &p->devices[i]; + /* Only mask OpenCAPI links */ + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + + if (dev->brick_index == 2 || dev->brick_index == 3) { + link_num = dev->brick_index - 2; + } else { + link_num = dev->brick_index; + } + mask = SETFIELD(PPC_BITMASK(link_num * 2, + link_num * 2 + 1), + mask, 0b11); + } /* Mask FIRs */ xscom_read(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, ®); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0xFFF); + reg |= mask; xscom_write(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, reg); /* freeze disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* fence disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); /* irq disable */ reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B); - reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + reg &= ~mask; npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); } From patchwork Wed Dec 12 06:58:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011602 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F7674xpNz9s47 for ; Wed, 12 Dec 2018 18:02:07 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F7671DQ2zDqwD for ; Wed, 12 Dec 2018 18:02:07 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73D5cJ7zDql9 for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wRVI009894 for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pauvebq99-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:34 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:32 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSOA57868508 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6CAD0A4065; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C950DA405C; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 09636A039A; Wed, 12 Dec 2018 17:59:25 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:54 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-4275-0000-0000-000002EF53CA X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-4276-0000-0000-000037FC7451 Message-Id: <87ffa1801b51c2f13e39128c57b50596be50215d.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 11/13] hw/npu2: Fix OpenCAPI PE assignment X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" When we support mixing NVLink and OpenCAPI devices on the same NPU, we're going to have to share the same range of 16 PE numbers between NVLink and OpenCAPI PHBs. For OpenCAPI devices, PE assignment is only significant for determining which System Interrupt Log register is used for a particular brick - unlike NVLink, it doesn't play any role in determining how links are fenced. Split the PE range into a lower half which is used for NVLink, and an upper half that is used for OpenCAPI, with a fixed PE number assigned per brick. As the PE assignment for OpenCAPI devices is fixed, set the PE once during device init and then ignore calls to the set_pe() operation. Suggested-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- hw/npu2-opencapi.c | 75 +++++++++++++++++++++-------------------------- include/npu2.h | 21 +++++++++++-- 2 files changed, 52 insertions(+), 44 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index d79727ad4199..a57f556b102d 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -482,6 +482,20 @@ static void brick_config(uint32_t gcid, uint32_t scom_base, int index) enable_pb_snooping(gcid, scom_base, index); } +/* Procedure 13.1.3.4 - Brick to PE Mapping */ +static void pe_config(struct npu2_dev *dev) +{ + /* We currently use a fixed PE assignment per brick */ + uint64_t val, reg; + val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, NPU2_OCAPI_PE(dev)); + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, 0); + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, + NPU2_MISC_BRICK0_BDF2PE_MAP0 + + (dev->brick_index * 0x18)); + npu2_write(dev->npu, reg, val); +} + /* Procedure 13.1.3.5 - TL Configuration */ static void tl_config(uint32_t gcid, uint32_t scom_base, uint64_t index) { @@ -1214,48 +1228,18 @@ static int64_t npu2_opencapi_ioda_reset(struct phb __unused *phb, return OPAL_SUCCESS; } -static int64_t npu2_opencapi_set_pe(struct phb *phb, - uint64_t pe_num, - uint64_t bdfn, - uint8_t bcompare, - uint8_t dcompare, - uint8_t fcompare, - uint8_t action) +static int64_t npu2_opencapi_set_pe(struct phb __unused *phb, + uint64_t __unused pe_num, + uint64_t __unused bdfn, + uint8_t __unused bcompare, + uint8_t __unused dcompare, + uint8_t __unused fcompare, + uint8_t __unused action) { - struct npu2 *p; - struct npu2_dev *dev; - uint64_t reg, val, pe_bdfn; - - /* Sanity check */ - if (action != OPAL_MAP_PE && action != OPAL_UNMAP_PE) - return OPAL_PARAMETER; - if (pe_num >= NPU2_MAX_PE_NUM) - return OPAL_PARAMETER; - if (bdfn >> 8) - return OPAL_PARAMETER; - if (bcompare != OpalPciBusAll || - dcompare != OPAL_COMPARE_RID_DEVICE_NUMBER || - fcompare != OPAL_COMPARE_RID_FUNCTION_NUMBER) - return OPAL_UNSUPPORTED; - - /* Get the NPU2 device */ - dev = phb_to_npu2_dev_ocapi(phb); - if (!dev) - return OPAL_PARAMETER; - - p = dev->npu; - - pe_bdfn = dev->bdfn; - - val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; - val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); - val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, pe_bdfn); - reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + - (dev->brick_index * 0x18)); - p->bdf2pe_cache[dev->brick_index] = val; - npu2_write(p, reg, val); - + /* + * Ignored on OpenCAPI - we use fixed PE assignments. May need + * addressing when we support dual-link devices. + */ return OPAL_SUCCESS; } @@ -1424,7 +1408,13 @@ static void setup_device(struct npu2_dev *dev) dt_add_property_cells(dn_phb, "ibm,links", 1); dt_add_property(dn_phb, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(dn_phb, "ibm,phb-diag-data-size", 0); + + /* + * We ignore whatever PE numbers Linux tries to set, so we just + * advertise enough that Linux won't complain + */ dt_add_property_cells(dn_phb, "ibm,opal-num-pes", NPU2_MAX_PE_NUM); + dt_add_property_cells(dn_phb, "ibm,opal-reserved-pe", NPU2_RESERVED_PE_NUM); dt_add_property_cells(dn_phb, "ranges", 0x02000000, hi32(mm_win[0]), lo32(mm_win[0]), @@ -1504,6 +1494,9 @@ int npu2_opencapi_init_npu(struct npu2 *npu) /* Procedure 13.1.3.1 - Select OCAPI vs NVLink */ brick_config(npu->chip_id, npu->xscom_base, dev->brick_index); + /* Procedure 13.1.3.4 - Brick to PE Mapping */ + pe_config(dev); + /* Procedure 13.1.3.5 - Transaction Layer Configuration */ tl_config(npu->chip_id, npu->xscom_base, dev->brick_index); diff --git a/include/npu2.h b/include/npu2.h index 8f4747006980..c7b20f19fde1 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -45,9 +45,24 @@ dev->npu->chip_id, dev->brick_index, ## a) -/* Number of PEs supported */ -#define NPU2_MAX_PE_NUM 16 -#define NPU2_RESERVED_PE_NUM 15 +/* + * Number of PEs supported + * + * The NPU supports PE numbers from 0-15. At present, we only assign a maximum + * of 1 PE per brick. + * + * NVLink devices are currently exposed to Linux underneath a single virtual + * PHB. Therefore, we give NVLink half the available PEs, which is enough for + * 6 bricks plus 1 reserved PE. + * + * For OpenCAPI, the BDF-to-PE registers are used exclusively for mapping + * bricks to System Interrupt Log registers (the BDF component of those + * registers is ignored). Currently, we allocate a fixed PE based on the brick + * index in the upper half of the PE namespace. + */ +#define NPU2_MAX_PE_NUM 8 +#define NPU2_RESERVED_PE_NUM 7 +#define NPU2_OCAPI_PE(ndev) ((ndev)->brick_index + NPU2_MAX_PE_NUM) #define NPU2_LINKS_PER_CHIP 6 From patchwork Wed Dec 12 06:58:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011598 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F74r5CcKz9s47 for ; Wed, 12 Dec 2018 18:01:00 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F74r3mt0zDr3F for ; Wed, 12 Dec 2018 18:01:00 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73C49xnzDqcG for ; Wed, 12 Dec 2018 17:59:35 +1100 (AEDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6wbWl058130 for ; Wed, 12 Dec 2018 01:59:33 -0500 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0b-001b2d01.pphosted.com with ESMTP id 2paucecv2x-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:32 -0500 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:31 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xSuD58589260 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2BF19AE055; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CD151AE053; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 1BA6AA039C; Wed, 12 Dec 2018 17:59:25 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:55 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-0016-0000-0000-000002354A80 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-0017-0000-0000-0000328D7BCC Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=826 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 12/13] hw/npu2: Always check device type when looping through NPU devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Fix all the remaining places in the NVLink code where we loop through all the devices on an NPU and don't check whether it's an NVLink device - triggering NTL resets, setting up relaxed ordering, and setting up LPAR mapping. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 9e60b4e16f29..12e370968914 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1165,7 +1165,7 @@ static int64_t npu2_hreset(struct pci_slot *slot __unused) for (i = 0; i < p->total_devices; i++) { ndev = &p->devices[i]; - if (ndev) { + if (ndev && ndev->type == NPU2_DEV_TYPE_NVLINK) { NPU2DEVINF(ndev, "Resetting device\n"); reset_ntl(ndev); } @@ -1189,7 +1189,7 @@ static int64_t npu2_creset(struct pci_slot *slot) for (i = 0; i < p->total_devices; i++) { ndev = &p->devices[i]; - if (ndev) { + if (ndev && ndev->type == NPU2_DEV_TYPE_NVLINK) { NPU2DEVINF(ndev, "Resetting device\n"); reset_ntl(ndev); } @@ -2016,7 +2016,8 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, /* Need to find an NVLink to send the ATSDs for this device over */ for (i = 0; i < p->total_devices; i++) { - if (p->devices[i].nvlink.gpu_bdfn == bdf) { + if (p->devices[i].type == NPU2_DEV_TYPE_NVLINK && + p->devices[i].nvlink.gpu_bdfn == bdf) { ndev = &p->devices[i]; break; } @@ -2200,6 +2201,9 @@ static int npu2_set_relaxed_ordering(uint32_t gcid, int pec, bool enable) npu = phb_to_npu2_nvlink(phb); for (int i = 0; i < npu->total_devices; i++) { ndev = &npu->devices[i]; + if (ndev->type != NPU2_DEV_TYPE_NVLINK) + continue; + if (enable) rc = npu2_enable_relaxed_ordering(ndev, gcid, pec); else From patchwork Wed Dec 12 06:58:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1011603 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43F76d5jt8z9s47 for ; Wed, 12 Dec 2018 18:02:33 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 43F76d4PrnzDqrX for ; Wed, 12 Dec 2018 18:02:33 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 43F73F0vrDzDqpp for ; Wed, 12 Dec 2018 17:59:36 +1100 (AEDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wBC6xQKm140969 for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2pasww7qas-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Dec 2018 01:59:35 -0500 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 12 Dec 2018 06:59:31 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 12 Dec 2018 06:59:29 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id wBC6xST455312574 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 12 Dec 2018 06:59:28 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78B144204F; Wed, 12 Dec 2018 06:59:28 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D3D9542042; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 12 Dec 2018 06:59:27 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 2AFC9A03BC; Wed, 12 Dec 2018 17:59:25 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 12 Dec 2018 17:58:56 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18121206-4275-0000-0000-000002EF53C9 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18121206-4276-0000-0000-000037FC7453 Message-Id: <5c2eff06db01029e6f284f4586405bb5b18b60a0.1544597914.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-12-12_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1812120061 Subject: [Skiboot] [PATCH 13/13] hw/npu2-common: Allow mixed mode NPU setups X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Now that we have all the support in place for NPUs with both NVLink and OpenCAPI devices, get rid of the error that aborts NPU init when a mixed setup is detected. While we're there, rename setup_devices() to more accurately reflect what it does, and move the calls to the NVLink/OpenCAPI setup code out of there into the main probe function. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-common.c | 28 ++++++---------------------- hw/npu2-opencapi.c | 5 ----- 2 files changed, 6 insertions(+), 27 deletions(-) diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 9c383ef7f35f..978ed4561ebc 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -618,26 +618,19 @@ failed: return NULL; } -static void setup_devices(struct npu2 *npu) +static void add_link_type_properties(struct npu2 *npu) { - bool nvlink_detected = false, ocapi_detected = false; struct npu2_dev *dev; - /* - * TODO: In future, we'll do brick configuration here to support mixed - * setups. - */ for (int i = 0; i < npu->total_devices; i++) { dev = &npu->devices[i]; switch (dev->type) { case NPU2_DEV_TYPE_NVLINK: - nvlink_detected = true; dt_add_property_strings(dev->dt_node, "ibm,npu-link-type", "nvlink"); break; case NPU2_DEV_TYPE_OPENCAPI: - ocapi_detected = true; dt_add_property_strings(dev->dt_node, "ibm,npu-link-type", "opencapi"); @@ -650,19 +643,6 @@ static void setup_devices(struct npu2 *npu) "unknown"); } } - - if (nvlink_detected && ocapi_detected) { - prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported, aborting NPU init\n"); - return; - } - - assign_bars(npu); - setup_irqs(npu); - - if (nvlink_detected) - npu2_nvlink_init_npu(npu); - else if (ocapi_detected) - npu2_opencapi_init_npu(npu); } void probe_npu2(void) @@ -698,7 +678,11 @@ void probe_npu2(void) if (!npu) continue; platform.npu2_device_detect(npu); + add_link_type_properties(npu); set_brick_config(npu); - setup_devices(npu); + assign_bars(npu); + setup_irqs(npu); + npu2_nvlink_init_npu(npu); + npu2_opencapi_init_npu(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index a57f556b102d..71661648d8e0 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -19,15 +19,10 @@ * * This file provides support for OpenCAPI as implemented on POWER9. * - * At present, we initialise the NPU separately from the NVLink code in npu2.c. - * As such, we don't currently support mixed NVLink and OpenCAPI configurations - * on the same NPU for machines such as Witherspoon. - * * Procedure references in this file are to the POWER9 OpenCAPI NPU Workbook * (IBM internal document). * * TODO: - * - Support for mixed NVLink and OpenCAPI on the same NPU * - Support for link ganging (one AFU using multiple links) * - Link reset and error handling * - Presence detection