From patchwork Mon Nov 13 11:06:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Neuling X-Patchwork-Id: 837415 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb7C43fw2z9sRV for ; Mon, 13 Nov 2017 22:07:28 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb7C42f03zDqkJ for ; Mon, 13 Nov 2017 22:07:28 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb7BV73PQzDqjw for ; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: from localhost.localdomain (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 3yb7BV0dfpz9sRV; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: by localhost.localdomain (Postfix, from userid 1000) id E4713EEA9EC; Mon, 13 Nov 2017 22:06:57 +1100 (AEDT) From: Michael Neuling To: stewart@linux.vnet.ibm.com Date: Mon, 13 Nov 2017 22:06:40 +1100 Message-Id: <20171113110644.15478-1-mikey@neuling.org> X-Mailer: git-send-email 2.14.1 Subject: [Skiboot] [PATCH 1/5] npu2: Create npu2_write_mcd() X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: skiboot@lists.ozlabs.org, mikey@neuling.org, Reza Arbab , alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" This code is replicated, so let's put it in a function. Also add some cleanups. No functional change. Signed-off-by: Michael Neuling Reviewed-by: Balbir Singh --- hw/npu2.c | 24 ++++++++++++++++-------- include/npu2-regs.h | 3 +++ 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index d215b4ce97..77b8e8c133 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -834,6 +834,20 @@ static int64_t npu2_ioda_reset(struct phb *phb, bool purge) return OPAL_SUCCESS; } +static void npu2_write_mcd(struct npu2 *p, uint64_t pcb_addr, uint64_t addr, + uint64_t size) +{ + uint64_t val; + + NPU2DBG(p, "Setting MCD addr:%llx\n", pcb_addr); + assert(is_pow2(size)); + + val = MCD_BANK_CN_VALID; + val = SETFIELD(MCD_BANK_CN_SIZE, val, (size >> 25) - 1); + val = SETFIELD(MCD_BANK_CN_ADDR, val, addr >> 25); + xscom_write(p->chip_id, pcb_addr, val); +} + static void npu2_hw_init(struct npu2 *p) { int i; @@ -873,10 +887,7 @@ static void npu2_hw_init(struct npu2 *p) /* Allocate the biggest chunk first as we assume gpu_max_addr has the * highest alignment. */ addr = gpu_max_addr - size; - val = PPC_BIT(0); - val = SETFIELD(PPC_BITMASK(13, 29), val, (size >> 25) - 1); - val = SETFIELD(PPC_BITMASK(33, 63), val, addr >> 25); - xscom_write(p->chip_id, MCD0_BANK0_CN3, val); + npu2_write_mcd(p, MCD0_BANK0_CN3, addr, size); total_size -= size; if (total_size) { /* total_size was not a power of two, but the remainder should @@ -885,10 +896,7 @@ static void npu2_hw_init(struct npu2 *p) size = 1ull << ilog2(total_size); addr -= size; assert(addr <= gpu_min_addr); - val = PPC_BIT(0); - val = SETFIELD(PPC_BITMASK(13, 29), val, (size >> 25) - 1); - val = SETFIELD(PPC_BITMASK(33, 63), val, addr >> 25); - xscom_write(p->chip_id, MCD1_BANK0_CN3, val); + npu2_write_mcd(p, MCD1_BANK0_CN3, addr, size); } } diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 307e93bd4c..c77a43a4c8 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -28,6 +28,9 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); * code */ #define MCD0_BANK0_CN3 0x301100d #define MCD1_BANK0_CN3 0x301140d +#define MCD_BANK_CN_VALID PPC_BIT(0) +#define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) +#define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) #define NPU2_REG_OFFSET(stack, block, offset) \ (((stack) << 20) | ((block) << 16) | (offset)) From patchwork Mon Nov 13 11:06:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Neuling X-Patchwork-Id: 837417 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb7CY02lbz9sRn for ; Mon, 13 Nov 2017 22:07:53 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb7CX31fGzDqjt for ; Mon, 13 Nov 2017 22:07:52 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from ozlabs.org (bilbo.ozlabs.org [103.22.144.67]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb7BV6L7VzDqhL for ; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: from localhost.localdomain (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 3yb7BV0kxxz9sRn; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: by localhost.localdomain (Postfix, from userid 1000) id E8CA7EEA9ED; Mon, 13 Nov 2017 22:06:57 +1100 (AEDT) From: Michael Neuling To: stewart@linux.vnet.ibm.com Date: Mon, 13 Nov 2017 22:06:41 +1100 Message-Id: <20171113110644.15478-2-mikey@neuling.org> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171113110644.15478-1-mikey@neuling.org> References: <20171113110644.15478-1-mikey@neuling.org> Subject: [Skiboot] [PATCH 2/5] npu2: Refactor BAR setting code X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: skiboot@lists.ozlabs.org, mikey@neuling.org, Reza Arbab , alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" This refactors the BAR setting code to make it clearer and handle a larger range of BAR addresses. No functional change. Signed-off-by: Michael Neuling Reviewed-by: Balbir Singh --- hw/npu2.c | 11 ++++++----- include/npu2-regs.h | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 77b8e8c133..9f2aaad45e 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -638,11 +638,12 @@ static int npu2_assign_gmb(struct npu2_dev *ndev) npu2_get_gpu_base(ndev, &base, &size); - /* Base address is in GB */ - base >>= 30; - val = SETFIELD(NPU2_MEM_BAR_SEL_MEM, 0ULL, 4); - val = SETFIELD(NPU2_MEM_BAR_NODE_ADDR, val, base); - val = SETFIELD(NPU2_MEM_BAR_GROUP | NPU2_MEM_BAR_CHIP, val, p->chip_id); + NPU2DBG(p, "Setting BAR region dt:%llx\n", base); + val = SETFIELD(NPU2_MEM_BAR_EN, 0ULL, 1); + val = SETFIELD(NPU2_MEM_BAR_SEL_MEM, val, base >> (63-14)); + val = SETFIELD(NPU2_MEM_BAR_GROUP, val, base >> (63-18)); + val = SETFIELD(NPU2_MEM_BAR_CHIP, val, base >> (63-21)); + val = SETFIELD(NPU2_MEM_BAR_NODE_ADDR, val, base >> (63-33)); val = SETFIELD(NPU2_MEM_BAR_POISON, val, 1); val = SETFIELD(NPU2_MEM_BAR_GRANULE, val, 0); diff --git a/include/npu2-regs.h b/include/npu2-regs.h index c77a43a4c8..cef9dbf7f8 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -105,7 +105,8 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); #define NPU2_TIMER_CFG 0x018 #define NPU2_GPU0_MEM_BAR 0x020 #define NPU2_GPU1_MEM_BAR 0x028 -#define NPU2_MEM_BAR_SEL_MEM PPC_BITMASK(0,2) +#define NPU2_MEM_BAR_EN PPC_BIT(0) +#define NPU2_MEM_BAR_SEL_MEM PPC_BITMASK(1,2) #define NPU2_MEM_BAR_GROUP PPC_BITMASK(3,6) #define NPU2_MEM_BAR_CHIP PPC_BITMASK(7,9) #define NPU2_MEM_BAR_NODE_ADDR PPC_BITMASK(10,21) From patchwork Mon Nov 13 11:06:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Neuling X-Patchwork-Id: 837419 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb7Cv3WSBz9sRV for ; Mon, 13 Nov 2017 22:08:11 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb7Cv2kVWzDqjr for ; Mon, 13 Nov 2017 22:08:11 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb7BW1cXVzDqjs for ; Mon, 13 Nov 2017 22:06:59 +1100 (AEDT) Received: from localhost.localdomain (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 3yb7BV0rF8z9sxR; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: by localhost.localdomain (Postfix, from userid 1000) id EB32EEEA9EE; Mon, 13 Nov 2017 22:06:57 +1100 (AEDT) From: Michael Neuling To: stewart@linux.vnet.ibm.com Date: Mon, 13 Nov 2017 22:06:42 +1100 Message-Id: <20171113110644.15478-3-mikey@neuling.org> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171113110644.15478-1-mikey@neuling.org> References: <20171113110644.15478-1-mikey@neuling.org> Subject: [Skiboot] [PATCH 3/5] npu2: MCD refactor X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: skiboot@lists.ozlabs.org, mikey@neuling.org, Reza Arbab , alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Pull out MCD writing code into npu2_mcd_init() No functional change. Signed-off-by: Michael Neuling Acked-by: Balbir Singh --- hw/npu2.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 9f2aaad45e..79ebaff75f 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -849,21 +849,10 @@ static void npu2_write_mcd(struct npu2 *p, uint64_t pcb_addr, uint64_t addr, xscom_write(p->chip_id, pcb_addr, val); } -static void npu2_hw_init(struct npu2 *p) +static void npu2_mcd_init(struct npu2 *p) { int i; - uint64_t val, size, addr, gpu_min_addr, gpu_max_addr, total_size; - - npu2_ioda_reset(&p->phb, false); - - /* Enable XTS retry mode */ - val = npu2_read(p, NPU2_XTS_CFG); - npu2_write(p, NPU2_XTS_CFG, val | NPU2_XTS_CFG_MMIOSD | NPU2_XTS_CFG_TRY_ATR_RO); - - if (!is_p9dd1()) { - val = npu2_read(p, NPU2_XTS_CFG2); - npu2_write(p, NPU2_XTS_CFG2, val | NPU2_XTS_CFG2_NO_FLUSH_ENA); - } + uint64_t size, addr, gpu_min_addr, gpu_max_addr, total_size; /* Init memory cache directory (MCD) registers. */ phys_map_get(p->chip_id, GPU_MEM, NPU2_LINKS_PER_CHIP - 1, @@ -901,6 +890,24 @@ static void npu2_hw_init(struct npu2 *p) } } +static void npu2_hw_init(struct npu2 *p) +{ + uint64_t val; + + npu2_ioda_reset(&p->phb, false); + + /* Enable XTS retry mode */ + val = npu2_read(p, NPU2_XTS_CFG); + npu2_write(p, NPU2_XTS_CFG, val | NPU2_XTS_CFG_MMIOSD | NPU2_XTS_CFG_TRY_ATR_RO); + + if (!is_p9dd1()) { + val = npu2_read(p, NPU2_XTS_CFG2); + npu2_write(p, NPU2_XTS_CFG2, val | NPU2_XTS_CFG2_NO_FLUSH_ENA); + } + + npu2_mcd_init(p); +} + static int64_t npu2_map_pe_dma_window_real(struct phb *phb, uint64_t pe_num, uint16_t window_id, From patchwork Mon Nov 13 11:06:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Neuling X-Patchwork-Id: 837418 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb7Ck1763z9sRV for ; Mon, 13 Nov 2017 22:08:02 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb7Ck0HTbzDqj8 for ; Mon, 13 Nov 2017 22:08:02 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from ozlabs.org (ozlabs.org [IPv6:2401:3900:2:1::2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb7BW1KF8zDqb5 for ; Mon, 13 Nov 2017 22:06:59 +1100 (AEDT) Received: from localhost.localdomain (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 3yb7BV0y1rz9t2W; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: by localhost.localdomain (Postfix, from userid 1000) id F1AEEEEA9EF; Mon, 13 Nov 2017 22:06:57 +1100 (AEDT) From: Michael Neuling To: stewart@linux.vnet.ibm.com Date: Mon, 13 Nov 2017 22:06:43 +1100 Message-Id: <20171113110644.15478-4-mikey@neuling.org> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171113110644.15478-1-mikey@neuling.org> References: <20171113110644.15478-1-mikey@neuling.org> Subject: [Skiboot] [PATCH 4/5] phys-map: Rename GPU_MEM to GPU_MEM_4T_DOWN X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: skiboot@lists.ozlabs.org, mikey@neuling.org, Reza Arbab , alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" This map is soon to be replaced, but we are going to keep it around for a little while so that we support older hostboot firmware. Rename it for now. Signed-off-by: Michael Neuling Reviewed-by: Balbir Singh --- hw/npu2.c | 8 ++++---- hw/phys-map.c | 14 +++++++------- include/phys-map.h | 2 +- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 79ebaff75f..082591a638 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -596,7 +596,7 @@ static void npu2_get_gpu_base(struct npu2_dev *ndev, uint64_t *addr, uint64_t *s int group; group = (ndev->bdfn >> 3) & 0x1f; - phys_map_get(ndev->npu->chip_id, GPU_MEM, group, addr, size); + phys_map_get(ndev->npu->chip_id, GPU_MEM_4T_DOWN, group, addr, size); } static void npu2_dn_fixup_gmb(struct dt_node *pd_dn, struct npu2_dev *ndev) @@ -855,16 +855,16 @@ static void npu2_mcd_init(struct npu2 *p) uint64_t size, addr, gpu_min_addr, gpu_max_addr, total_size; /* Init memory cache directory (MCD) registers. */ - phys_map_get(p->chip_id, GPU_MEM, NPU2_LINKS_PER_CHIP - 1, + phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, NPU2_LINKS_PER_CHIP - 1, &gpu_min_addr, NULL); - phys_map_get(p->chip_id, GPU_MEM, 0, &gpu_max_addr, &size); + phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, 0, &gpu_max_addr, &size); gpu_max_addr += size; /* We assume GPU memory is contiguous from the first possible GPU to the * last and that the size is the same so best to check that. */ for (i = 0; i < NPU2_LINKS_PER_CHIP; i++) { uint64_t tmp; - phys_map_get(p->chip_id, GPU_MEM, i, &addr, &tmp); + phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, i, &addr, &tmp); assert((addr >= gpu_min_addr) && (addr + tmp <= gpu_max_addr)); assert(tmp == size); } diff --git a/hw/phys-map.c b/hw/phys-map.c index 8cccbf07b7..02bc33b8f8 100644 --- a/hw/phys-map.c +++ b/hw/phys-map.c @@ -37,15 +37,15 @@ static const struct phys_map_info *phys_map; static const struct phys_map_entry phys_map_table_nimbus[] = { - /* System memory u pto 4TB minus GPU memory */ + /* System memory upto 4TB minus GPU memory */ { SYSTEM_MEM, 0, 0x0000000000000000ull, 0x0000034000000000ull }, /* GPU memory from 4TB - 128GB*GPU */ - { GPU_MEM, 5, 0x0000034000000000ull, 0x0000002000000000ull }, - { GPU_MEM, 4, 0x0000036000000000ull, 0x0000002000000000ull }, - { GPU_MEM, 3, 0x0000038000000000ull, 0x0000002000000000ull }, - { GPU_MEM, 2, 0x000003a000000000ull, 0x0000002000000000ull }, - { GPU_MEM, 1, 0x000003c000000000ull, 0x0000002000000000ull }, - { GPU_MEM, 0, 0x000003e000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 5, 0x0000034000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 4, 0x0000036000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 3, 0x0000038000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 2, 0x000003a000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 1, 0x000003c000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_DOWN, 0, 0x000003e000000000ull, 0x0000002000000000ull }, /* 0 TB offset @ MMIO 0x0006000000000000ull */ { PHB4_64BIT_MMIO, 0, 0x0006000000000000ull, 0x0000004000000000ull }, diff --git a/include/phys-map.h b/include/phys-map.h index 3621bfd20a..c9b2389615 100644 --- a/include/phys-map.h +++ b/include/phys-map.h @@ -26,7 +26,7 @@ enum phys_map_type { NULL_MAP, SYSTEM_MEM, - GPU_MEM, + GPU_MEM_4T_DOWN, PHB4_64BIT_MMIO, PHB4_32BIT_MMIO, PHB4_XIVE_ESB, From patchwork Mon Nov 13 11:06:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Neuling X-Patchwork-Id: 837416 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yb7CK6zyQz9sRV for ; Mon, 13 Nov 2017 22:07:41 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yb7CK5dypzDqjw for ; Mon, 13 Nov 2017 22:07:41 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from ozlabs.org (bilbo.ozlabs.org [103.22.144.67]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yb7BV6Q1hzDqjL for ; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: from localhost.localdomain (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 3yb7BV1C9jz9t2c; Mon, 13 Nov 2017 22:06:58 +1100 (AEDT) Received: by localhost.localdomain (Postfix, from userid 1000) id F408DEEA9F0; Mon, 13 Nov 2017 22:06:57 +1100 (AEDT) From: Michael Neuling To: stewart@linux.vnet.ibm.com Date: Mon, 13 Nov 2017 22:06:44 +1100 Message-Id: <20171113110644.15478-5-mikey@neuling.org> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171113110644.15478-1-mikey@neuling.org> References: <20171113110644.15478-1-mikey@neuling.org> Subject: [Skiboot] [PATCH 5/5] npu2: Move to new GPU memory map X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: skiboot@lists.ozlabs.org, mikey@neuling.org, Reza Arbab , alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" There are three different ways we configure the MCD and memory map. 1) Old way (current way) Skiboot configures the MCD and puts GPUs at 4TB and below 2) New way with MCD Hostboot configures the MCD and skiboot puts GPU at 4TB and above 3) New way without MCD No one configures the MCD and skiboot puts GPU at 4TB and below The patch keeps option 1 and adds options 2 and 3. The different configurations are detected using certain scoms (see patch). Option 1 will go away eventually as it's a configuration that can cause xstops or data integrity problems. We are keeping it around to support existing hostboot. Option 2 supports only 4 GPUs and 512GB of memory per socket. Option 3 supports 6 GPUs and 4TB of memory but may have some performance impact. Signed-off-by: Michael Neuling --- hw/npu2.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++----- hw/phys-map.c | 5 +++++ include/npu2-regs.h | 5 +++++ include/npu2.h | 1 + include/phys-map.h | 1 + 5 files changed, 63 insertions(+), 5 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index 082591a638..9752dec5d7 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -593,10 +593,11 @@ static struct dt_node *npu2_create_memory_dn(uint64_t addr, uint64_t size) * on bdfn. */ static void npu2_get_gpu_base(struct npu2_dev *ndev, uint64_t *addr, uint64_t *size) { + struct npu2 *p = ndev->npu; int group; group = (ndev->bdfn >> 3) & 0x1f; - phys_map_get(ndev->npu->chip_id, GPU_MEM_4T_DOWN, group, addr, size); + phys_map_get(ndev->npu->chip_id, p->gpu_map_type, group, addr, size); } static void npu2_dn_fixup_gmb(struct dt_node *pd_dn, struct npu2_dev *ndev) @@ -855,16 +856,16 @@ static void npu2_mcd_init(struct npu2 *p) uint64_t size, addr, gpu_min_addr, gpu_max_addr, total_size; /* Init memory cache directory (MCD) registers. */ - phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, NPU2_LINKS_PER_CHIP - 1, + phys_map_get(p->chip_id, p->gpu_map_type, NPU2_LINKS_PER_CHIP - 1, &gpu_min_addr, NULL); - phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, 0, &gpu_max_addr, &size); + phys_map_get(p->chip_id, p->gpu_map_type, 0, &gpu_max_addr, &size); gpu_max_addr += size; /* We assume GPU memory is contiguous from the first possible GPU to the * last and that the size is the same so best to check that. */ for (i = 0; i < NPU2_LINKS_PER_CHIP; i++) { uint64_t tmp; - phys_map_get(p->chip_id, GPU_MEM_4T_DOWN, i, &addr, &tmp); + phys_map_get(p->chip_id, p->gpu_map_type, i, &addr, &tmp); assert((addr >= gpu_min_addr) && (addr + tmp <= gpu_max_addr)); assert(tmp == size); } @@ -905,7 +906,52 @@ static void npu2_hw_init(struct npu2 *p) npu2_write(p, NPU2_XTS_CFG2, val | NPU2_XTS_CFG2_NO_FLUSH_ENA); } - npu2_mcd_init(p); + /* + * There are three different ways we configure the MCD and memory map. + * 1) Old way + * Skiboot configures the MCD and puts GPUs at 4TB and below + * 2) New way with MCD + * Hostboot configures the MCD and skiboot puts GPU at 4TB and above + * 3) New way without MCD + * No one configures the MCD and skiboot puts GPU at 4TB and below + * + * 1) Will go away evenutally as it's a configuration that can + * cause an xstop or data integrity problems. We are keeping + * it around to support existing hostboot. Print error + * message if used. + * 2) Is for smaller memory configurations and will be used + * initially for GPUs on Witherspoon. Supports only to + * 512GB of memory and 4 GPUs per socket. + * 3) Is for fully populated configurations of 4TB of memory + * and 6GPUs per socket. May have performance impacts. + * + * The different configurations can be detected via the following scoms: + * 1) 0x5011c0c bit 2 = 1, 0x5011c0a bits 42:48 = 0 + * 2) 0x5011c0c bit 2 = 1, 0x5011c0a bits 42:48 = 3 + * 3) 0x5011c0c bit 2 = 0, 0x5011c0a bits 42:48 = 0 + */ + + /* Get 0x05011c0c bit 2 = 1 */ + xscom_read(p->chip_id, PB_CENT_HP_MODE_CURR, &val); + if ((val & PB_CFG_CHG_RATE_GP_MASTER) != 0) { + /* Get 0x05011c0a bits 42:48 */ + xscom_read(p->chip_id, PB_CENT_MODE, &val); + if (GETFIELD(PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT, val) == 0) { + /* 1) */ + NPU2DBG(p, "Using old memory map + MCD enabled in skiboot\n"); + NPU2ERR(p, "!!! Old firmware detected. Update hostboot for new MCD mapping !!!\n"); + p->gpu_map_type = GPU_MEM_4T_DOWN; + npu2_mcd_init(p); + } else { + /* 2) */ + NPU2DBG(p, "Using small memory map + MCD enabled\n"); + p->gpu_map_type = GPU_MEM_4T_UP; + } + } else { + /* 3) */ + NPU2DBG(p, "Using large memory map + MCD disabled\n"); + p->gpu_map_type = GPU_MEM_4T_DOWN; + } } static int64_t npu2_map_pe_dma_window_real(struct phb *phb, diff --git a/hw/phys-map.c b/hw/phys-map.c index 02bc33b8f8..a2b5dbd262 100644 --- a/hw/phys-map.c +++ b/hw/phys-map.c @@ -46,6 +46,11 @@ static const struct phys_map_entry phys_map_table_nimbus[] = { { GPU_MEM_4T_DOWN, 2, 0x000003a000000000ull, 0x0000002000000000ull }, { GPU_MEM_4T_DOWN, 1, 0x000003c000000000ull, 0x0000002000000000ull }, { GPU_MEM_4T_DOWN, 0, 0x000003e000000000ull, 0x0000002000000000ull }, + /* GPU memory from 4TB + 128GB*GPU. 4 GPUs only */ + { GPU_MEM_4T_UP, 0, 0x0000040000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_UP, 1, 0x0000042000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_UP, 2, 0x0000044000000000ull, 0x0000002000000000ull }, + { GPU_MEM_4T_UP, 3, 0x0000046000000000ull, 0x0000002000000000ull }, /* 0 TB offset @ MMIO 0x0006000000000000ull */ { PHB4_64BIT_MMIO, 0, 0x0006000000000000ull, 0x0000004000000000ull }, diff --git a/include/npu2-regs.h b/include/npu2-regs.h index cef9dbf7f8..2764f9291e 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -31,6 +31,11 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); #define MCD_BANK_CN_VALID PPC_BIT(0) #define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) #define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) +#define PB_CENT_HP_MODE_CURR 0x5011c0c +#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) +#define PB_CENT_MODE 0x5011c0a +#define PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT PPC_BITMASK(42,48) + #define NPU2_REG_OFFSET(stack, block, offset) \ (((stack) << 20) | ((block) << 16) | (offset)) diff --git a/include/npu2.h b/include/npu2.h index f11a13a0b2..925eae0a64 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -128,6 +128,7 @@ struct npu2 { uint32_t base_lsi; uint32_t total_devices; struct npu2_dev *devices; + enum phys_map_type gpu_map_type; /* IODA cache */ uint64_t lxive_cache[8]; diff --git a/include/phys-map.h b/include/phys-map.h index c9b2389615..73adda079e 100644 --- a/include/phys-map.h +++ b/include/phys-map.h @@ -27,6 +27,7 @@ enum phys_map_type { NULL_MAP, SYSTEM_MEM, GPU_MEM_4T_DOWN, + GPU_MEM_4T_UP, PHB4_64BIT_MMIO, PHB4_32BIT_MMIO, PHB4_XIVE_ESB,