From patchwork Fri Jan 24 04:57:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 1228678 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 483n1x2jltz9sRY for ; Fri, 24 Jan 2020 15:57:25 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 483n1x10mMzDqZq for ; Fri, 24 Jan 2020 15:57:25 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=ajd@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 483n1r2PWMzDqZR for ; Fri, 24 Jan 2020 15:57:19 +1100 (AEDT) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 00O4v5X1065984 for ; Thu, 23 Jan 2020 23:57:17 -0500 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2xqmjsyvcr-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 23 Jan 2020 23:57:16 -0500 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 24 Jan 2020 04:57:14 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 24 Jan 2020 04:57:13 -0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 00O4vC2B36569214 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jan 2020 04:57:12 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 535EBA405B; Fri, 24 Jan 2020 04:57:12 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC46FA405F; Fri, 24 Jan 2020 04:57:11 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 24 Jan 2020 04:57:11 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id C9343A01CA; Fri, 24 Jan 2020 15:57:07 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Fri, 24 Jan 2020 15:57:03 +1100 X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 20012404-0028-0000-0000-000003D3E8AD X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 20012404-0029-0000-0000-000024982606 Message-Id: <20200124045703.8801-1-ajd@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.572 definitions=2020-01-23_13:2020-01-23, 2020-01-23 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 bulkscore=0 phishscore=0 adultscore=0 spamscore=0 mlxlogscore=999 suspectscore=1 mlxscore=0 impostorscore=0 lowpriorityscore=0 clxscore=1015 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1910280000 definitions=main-2001240039 Subject: [Skiboot] [PATCH v2] hw/npu2-opencapi: Support multiple LPC devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: clombard@linux.ibm.com, alastair@d-silva.org Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Currently, we only have a single range for LPC memory per chip, and we only allow a single device to use that range. With upcoming Hostboot/SBE changes, we'll use the chip address extension mask to give us multiple ranges by using the masked bits of the group ID. Each device can now allocate a whole 4TB non-mirrored region. We still don't do >4TB ranges. If the extension mask is not set correctly, we'll fall back to only permitting one device and printing an error suggesting a firmware upgrade. Signed-off-by: Andrew Donnellan --- I've been able to test this under limited configurations, so far so good. v1->v2: - fix excessively large range size which I'd left in there for some reason fix size Signed-off-by: Andrew Donnellan --- hw/npu2-opencapi.c | 43 +++++++++++++++++++++++++++++-------------- hw/phys-map.c | 15 +++++++++------ include/npu2.h | 2 ++ 3 files changed, 40 insertions(+), 20 deletions(-) diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 19589c92d477..37d05340e00b 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -2179,25 +2179,38 @@ static void set_mem_bar(struct npu2_dev *dev, uint64_t base, uint64_t size) static int64_t alloc_mem_bar(struct npu2_dev *dev, uint64_t size, uint64_t *bar) { - uint64_t phys_map_base, phys_map_size; + uint64_t phys_map_base, phys_map_size, val; int rc = OPAL_SUCCESS; lock(&dev->npu->lock); - /* - * Right now, we support 1 allocation per chip, of up to 4TB. - * - * In future, we will use chip address extension to support - * >4TB ranges, and we will implement a more sophisticated - * allocator to allow an allocation for every link on a chip. - */ - - if (dev->npu->lpc_mem_allocated) { + if (dev->lpc_mem_base) { + OCAPIERR(dev, "LPC allocation failed - BAR already in use\n"); rc = OPAL_RESOURCE; goto out; } - phys_map_get(dev->npu->chip_id, OCAPI_MEM, 0, &phys_map_base, &phys_map_size); + xscom_read(dev->npu->chip_id, PB_CENT_MODE, &val); + if (GETFIELD(PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT, val) == 0b1100100) { + phys_map_get(dev->npu->chip_id, OCAPI_MEM, + dev->brick_index - 2, &phys_map_base, + &phys_map_size); + } else { + bool in_use = false; + for (int i = 0; i < dev->npu->total_devices; i++) { + if (dev->npu->devices[i].lpc_mem_base) + in_use = true; + } + + if (in_use) { + OCAPIERR(dev, "LPC allocation failed - single device per chip limit, FW upgrade required (pb_cent_mode=0x%016llx)\n", val); + rc = OPAL_RESOURCE; + goto out; + } + + phys_map_get(dev->npu->chip_id, OCAPI_MEM, 0, &phys_map_base, + &phys_map_size); + } if (size > phys_map_size) { /** @@ -2223,7 +2236,8 @@ static int64_t alloc_mem_bar(struct npu2_dev *dev, uint64_t size, uint64_t *bar) set_mem_bar(dev, phys_map_base, size); *bar = phys_map_base; - dev->npu->lpc_mem_allocated = dev; + dev->lpc_mem_base = phys_map_base; + dev->lpc_mem_size = size; out: unlock(&dev->npu->lock); @@ -2236,13 +2250,14 @@ static int64_t release_mem_bar(struct npu2_dev *dev) lock(&dev->npu->lock); - if (dev->npu->lpc_mem_allocated != dev) { + if (!dev->lpc_mem_base) { rc = OPAL_PARAMETER; goto out; } set_mem_bar(dev, 0, 0); - dev->npu->lpc_mem_allocated = NULL; + dev->lpc_mem_base = 0; + dev->lpc_mem_size = 0; out: unlock(&dev->npu->lock); diff --git a/hw/phys-map.c b/hw/phys-map.c index 9917da7cdf94..fd79b3c123a4 100644 --- a/hw/phys-map.c +++ b/hw/phys-map.c @@ -44,15 +44,18 @@ static const struct phys_map_entry phys_map_table_nimbus[] = { { GPU_MEM_4T_UP, 3, 0x0000046000000000ull, 0x0000002000000000ull }, /* - * OpenCAPI LPC Memory - single 4TB range per chip, fills - * whole second non-mirrored region. + * OpenCAPI LPC Memory * - * Longer term, we're going to use chip address extension to - * enable >4TB to be allocated per chip. At that point, we - * may have to find another way of assigning these ranges - * outside of phys-map. + * With chip address extension enabled, we allocate 4TB ranges + * (in the second non-mirrored region) for each OpenCAPI link + * by varying the upper 2 bits of the group ID. + * + * We don't currently support >4TB ranges. */ { OCAPI_MEM, 0, 0x0002000000000000ull, 0x0000040000000000ull }, + { OCAPI_MEM, 1, 0x0002200000000000ull, 0x0000040000000000ull }, + { OCAPI_MEM, 2, 0x0002400000000000ull, 0x0000040000000000ull }, + { OCAPI_MEM, 3, 0x0002600000000000ull, 0x0000040000000000ull }, /* 0 TB offset @ MMIO 0x0006000000000000ull */ { PHB4_64BIT_MMIO, 0, 0x0006000000000000ull, 0x0000004000000000ull }, diff --git a/include/npu2.h b/include/npu2.h index d2a3430e3e3a..c72e6c8e8f11 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -150,6 +150,8 @@ struct npu2_dev { uint64_t linux_pe; unsigned long train_start; unsigned long train_timeout; + uint64_t lpc_mem_base; + uint64_t lpc_mem_size; }; struct npu2 {