From patchwork Wed Feb 7 06:50:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870248 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsT12Y3pz9ryk for ; Wed, 7 Feb 2018 17:52:21 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsT11SN7zF10c for ; Wed, 7 Feb 2018 17:52:21 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRl1HjkzF0Sb for ; Wed, 7 Feb 2018 17:51:14 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176oFvE002059 for ; Wed, 7 Feb 2018 01:51:13 -0500 Received: from e06smtp14.uk.ibm.com (e06smtp14.uk.ibm.com [195.75.94.110]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyt87vx6f-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:12 -0500 Received: from localhost by e06smtp14.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:10 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp14.uk.ibm.com (192.168.101.144) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:08 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176p8gm49152114; Wed, 7 Feb 2018 06:51:08 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 296B54C052; Wed, 7 Feb 2018 06:44:59 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 81D8C4C058; Wed, 7 Feb 2018 06:44:58 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:58 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 43182A025B; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:47 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0016-0000-0000-00000520334C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0017-0000-0000-0000285CE7C2 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 01/10] npu2: Split out common helper functions into separate file X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Split out common helper functions for NPU register access into a separate file, as these will be used extensively by both NVLink and OpenCAPI code. Signed-off-by: Andrew Donnellan Acked-by: Reza Arbab --- v1->v2: - Add copyright header (Fred) --- hw/Makefile.inc | 2 +- hw/npu2-common.c | 114 +++++++++++++++++++++++++++++++++++++++++++++- hw/npu2.c | 92 +------------------------------------ include/npu2-regs.h | 5 ++- include/npu2.h | 2 +- 5 files changed, 122 insertions(+), 93 deletions(-) create mode 100644 hw/npu2-common.c diff --git a/hw/Makefile.inc b/hw/Makefile.inc index 04cacd12c408..cf8649d00270 100644 --- a/hw/Makefile.inc +++ b/hw/Makefile.inc @@ -7,7 +7,7 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o -HW_OBJS += phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o +HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o HW=hw/built-in.o # FIXME hack this for now diff --git a/hw/npu2-common.c b/hw/npu2-common.c new file mode 100644 index 000000000000..0c22d6188819 --- /dev/null +++ b/hw/npu2-common.c @@ -0,0 +1,114 @@ +/* Copyright 2013-2018 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include +#include + +bool is_p9dd1(void) +{ + struct proc_chip *chip = next_chip(NULL); + + return chip && + (chip->type == PROC_CHIP_P9_NIMBUS || + chip->type == PROC_CHIP_P9_CUMULUS) && + (chip->ec_level & 0xf0) == 0x10; +} + +/* + * We use the indirect method because it uses the same addresses as + * the MMIO offsets (NPU RING) + */ +static void npu2_scom_set_addr(uint64_t gcid, uint64_t scom_base, + uint64_t addr, uint64_t size) +{ + uint64_t isa = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_ADDR : + NPU2_MISC_SCOM_IND_SCOM_ADDR; + + addr = SETFIELD(NPU2_MISC_DA_ADDR, 0ull, addr); + addr = SETFIELD(NPU2_MISC_DA_LEN, addr, size); + xscom_write(gcid, scom_base + isa, addr); +} + +void npu2_scom_write(uint64_t gcid, uint64_t scom_base, + uint64_t reg, uint64_t size, + uint64_t val) +{ + uint64_t isd = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_DATA : + NPU2_MISC_SCOM_IND_SCOM_DATA; + + npu2_scom_set_addr(gcid, scom_base, reg, size); + xscom_write(gcid, scom_base + isd, val); +} + +uint64_t npu2_scom_read(uint64_t gcid, uint64_t scom_base, + uint64_t reg, uint64_t size) +{ + uint64_t val; + uint64_t isd = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_DATA : + NPU2_MISC_SCOM_IND_SCOM_DATA; + + npu2_scom_set_addr(gcid, scom_base, reg, size); + xscom_read(gcid, scom_base + isd, &val); + + return val; +} + +void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val) +{ + npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, + (uint64_t)val << 32); +} + +uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg) +{ + return npu2_scom_read(p->chip_id, p->xscom_base, reg, + NPU2_MISC_DA_LEN_4B) >> 32; +} + +void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val) +{ + npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B, val); +} + +uint64_t npu2_read(struct npu2 *p, uint64_t reg) +{ + return npu2_scom_read(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B); +} + +void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask) +{ + uint64_t new_val; + + new_val = npu2_read(p, reg); + new_val &= ~mask; + new_val |= val & mask; + npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B, new_val); +} + +void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask) +{ + uint32_t new_val; + + new_val = npu2_read_4b(p, reg); + new_val &= ~mask; + new_val |= val & mask; + npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, + (uint64_t)new_val << 32); +} diff --git a/hw/npu2.c b/hw/npu2.c index c88394b678d6..41c30f404593 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -64,98 +64,6 @@ * configure one particular BAR. */ -static bool is_p9dd1(void) -{ - struct proc_chip *chip = next_chip(NULL); - - return chip && - (chip->type == PROC_CHIP_P9_NIMBUS || - chip->type == PROC_CHIP_P9_CUMULUS) && - (chip->ec_level & 0xf0) == 0x10; -} - -/* - * We use the indirect method because it uses the same addresses as - * the MMIO offsets (NPU RING) - */ -static void npu2_scom_set_addr(uint64_t gcid, uint64_t scom_base, - uint64_t addr, uint64_t size) -{ - uint64_t isa = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_ADDR : - NPU2_MISC_SCOM_IND_SCOM_ADDR; - - addr = SETFIELD(NPU2_MISC_DA_ADDR, 0ull, addr); - addr = SETFIELD(NPU2_MISC_DA_LEN, addr, size); - xscom_write(gcid, scom_base + isa, addr); -} - -static void npu2_scom_write(uint64_t gcid, uint64_t scom_base, - uint64_t reg, uint64_t size, - uint64_t val) -{ - uint64_t isd = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_DATA : - NPU2_MISC_SCOM_IND_SCOM_DATA; - - npu2_scom_set_addr(gcid, scom_base, reg, size); - xscom_write(gcid, scom_base + isd, val); -} - -static uint64_t npu2_scom_read(uint64_t gcid, uint64_t scom_base, - uint64_t reg, uint64_t size) -{ - uint64_t val; - uint64_t isd = is_p9dd1() ? NPU2_DD1_MISC_SCOM_IND_SCOM_DATA : - NPU2_MISC_SCOM_IND_SCOM_DATA; - - npu2_scom_set_addr(gcid, scom_base, reg, size); - xscom_read(gcid, scom_base + isd, &val); - - return val; -} - -void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val) -{ - npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, - (uint64_t)val << 32); -} - -uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg) -{ - return npu2_scom_read(p->chip_id, p->xscom_base, reg, - NPU2_MISC_DA_LEN_4B) >> 32; -} - -void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val) -{ - npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B, val); -} - -uint64_t npu2_read(struct npu2 *p, uint64_t reg) -{ - return npu2_scom_read(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B); -} - -void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask) -{ - uint64_t new_val; - - new_val = npu2_read(p, reg); - new_val &= ~mask; - new_val |= val & mask; - npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_8B, new_val); -} - -void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mask) -{ - uint32_t new_val; - - new_val = npu2_read_4b(p, reg); - new_val &= ~mask; - new_val |= val & mask; - npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, - (uint64_t)new_val << 32); -} - /* Set a specific flag in the vendor config space */ void npu2_set_link_flag(struct npu2_dev *ndev, uint8_t flag) { diff --git a/include/npu2-regs.h b/include/npu2-regs.h index e739ac5091a6..b190d0f0767a 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -23,6 +23,11 @@ void npu2_write4(struct npu2 *p, uint64_t reg, uint64_t val); uint64_t npu2_read(struct npu2 *p, uint64_t reg); void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val); void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); +uint64_t npu2_scom_read(uint64_t gcid, uint64_t scom_base, + uint64_t reg, uint64_t size); +void npu2_scom_write(uint64_t gcid, uint64_t scom_base, + uint64_t reg, uint64_t size, + uint64_t val); /* These aren't really NPU specific registers but we initialise them in NPU * code */ diff --git a/include/npu2.h b/include/npu2.h index dae152a6b034..d0c9ac502a70 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -163,4 +163,6 @@ void npu2_dev_procedure_reset(struct npu2_dev *dev); void npu2_set_link_flag(struct npu2_dev *ndev, uint8_t flag); void npu2_clear_link_flag(struct npu2_dev *ndev, uint8_t flag); extern int nv_zcal_nominal; +bool is_p9dd1(void); + #endif /* __NPU2_H */ From patchwork Wed Feb 7 06:50:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870249 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsTK43Nqz9ryk for ; Wed, 7 Feb 2018 17:52:37 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsTK17vGzF12b for ; Wed, 7 Feb 2018 17:52:37 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRm0kwyzF0TH for ; Wed, 7 Feb 2018 17:51:15 +1100 (AEDT) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176oXJq020235 for ; Wed, 7 Feb 2018 01:51:14 -0500 Received: from e06smtp10.uk.ibm.com (e06smtp10.uk.ibm.com [195.75.94.106]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyqhgtaq2-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:13 -0500 Received: from localhost by e06smtp10.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:11 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp10.uk.ibm.com (192.168.101.140) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:08 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176p8AF43057244; Wed, 7 Feb 2018 06:51:08 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 25AB8A4040; Wed, 7 Feb 2018 06:44:34 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78FE8A4051; Wed, 7 Feb 2018 06:44:33 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:33 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 54F4DA0273; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:48 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0040-0000-0000-0000040D2CB0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0041-0000-0000-00002610DCF2 Message-Id: <97e4b75f54a5831b1d806ba5ae1decdcaf82c0a9.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 02/10] npu2: Rework NPU data structures for OpenCAPI X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Unlike NVLink, OpenCAPI registers a separate PHB for each device, in order to allow us to force Linux to use the correct MMIO windows for each NPU link. This requires some reworking of NPU data structures to account for the fact that a PHB could correspond to either an NPU (NVLink) or a single link (OpenCAPI). At some later point, we may want to rework the NVLink code to present a separate PHB per device in order to simplify this. For now, we split NVLink-specific device data into a separate struct in order to make it clear which fields are NVLink-only. Additionally, add helper functions to correctly translate between OpenCAPI/NVLink PHBs and the underlying structures, and various fields for OpenCAPI data that we're going to need later on. Signed-off-by: Andrew Donnellan Acked-by: Reza Arbab --- v1->v2: - rebase on master - add fields for i2c port v3->v4: - keep npu2_set_pe() as an NVLink only function (Alistair) --- Alistair expressed some concern about this, after some discussion we've decided that we'll do some further refactoring on top of this, but as a separate series. --- hw/npu2-hw-procedures.c | 6 +- hw/npu2.c | 128 +++++++++++++++++++++-------------------- include/npu2.h | 76 ++++++++++++++++++------ include/pci.h | 1 +- 4 files changed, 128 insertions(+), 83 deletions(-) diff --git a/hw/npu2-hw-procedures.c b/hw/npu2-hw-procedures.c index b21c399d138d..d66632fcbdb9 100644 --- a/hw/npu2-hw-procedures.c +++ b/hw/npu2-hw-procedures.c @@ -790,7 +790,7 @@ static int64_t npu_dev_procedure_read(struct npu2_dev *dev, uint32_t offset, if (size != 4) { /* Short config reads are not supported */ - prlog(PR_ERR, "NPU%d: Short read of procedure register\n", dev->npu->phb.opal_id); + prlog(PR_ERR, "NPU%d: Short read of procedure register\n", npu2_dev_to_phb(dev)->opal_id); return OPAL_PARAMETER; } @@ -812,7 +812,7 @@ static int64_t npu_dev_procedure_read(struct npu2_dev *dev, uint32_t offset, default: prlog(PR_ERR, "NPU%d: Invalid vendor specific offset 0x%08x\n", - dev->npu->phb.opal_id, offset); + npu2_dev_to_phb(dev)->opal_id, offset); rc = OPAL_PARAMETER; } @@ -828,7 +828,7 @@ static int64_t npu_dev_procedure_write(struct npu2_dev *dev, uint32_t offset, if (size != 4) { /* Short config writes are not supported */ prlog(PR_ERR, "NPU%d: Short read of procedure register\n", - dev->npu->phb.opal_id); + npu2_dev_to_phb(dev)->opal_id); return OPAL_PARAMETER; } diff --git a/hw/npu2.c b/hw/npu2.c index 41c30f404593..3162bb613e7a 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -67,16 +67,16 @@ /* Set a specific flag in the vendor config space */ void npu2_set_link_flag(struct npu2_dev *ndev, uint8_t flag) { - ndev->link_flags |= flag; - PCI_VIRT_CFG_INIT_RO(ndev->pvd, VENDOR_CAP_START + - VENDOR_CAP_PCI_DEV_OFFSET, 1, ndev->link_flags); + ndev->nvlink.link_flags |= flag; + PCI_VIRT_CFG_INIT_RO(ndev->nvlink.pvd, VENDOR_CAP_START + + VENDOR_CAP_PCI_DEV_OFFSET, 1, ndev->nvlink.link_flags); } void npu2_clear_link_flag(struct npu2_dev *ndev, uint8_t flag) { - ndev->link_flags &= ~flag; - PCI_VIRT_CFG_INIT_RO(ndev->pvd, VENDOR_CAP_START + - VENDOR_CAP_PCI_DEV_OFFSET, 1, ndev->link_flags); + ndev->nvlink.link_flags &= ~flag; + PCI_VIRT_CFG_INIT_RO(ndev->nvlink.pvd, VENDOR_CAP_START + + VENDOR_CAP_PCI_DEV_OFFSET, 1, ndev->nvlink.link_flags); } static inline void npu2_ioda_sel(struct npu2 *p, uint32_t table, @@ -97,7 +97,7 @@ static struct npu2_dev *npu2_bdf_to_dev(struct npu2 *p, if (bdfn & ~0xff) return NULL; - pvd = pci_virt_find_device(&p->phb, bdfn); + pvd = pci_virt_find_device(&p->phb_nvlink, bdfn); if (pvd) return pvd->data; @@ -270,7 +270,7 @@ static int64_t npu2_cfg_write_bar(struct npu2_dev *dev, uint32_t offset, uint32_t size, uint32_t data) { - struct pci_virt_device *pvd = dev->pvd; + struct pci_virt_device *pvd = dev->nvlink.pvd; struct npu2_pcie_bar *bar = (struct npu2_pcie_bar *) pcrf->data; struct npu2_bar old_bar, *npu2_bar = &bar->npu2_bar; uint32_t pci_cmd; @@ -343,7 +343,7 @@ static int64_t npu2_dev_cfg_exp_devcap(void *dev, if ((size != 2) || (offset & 1)) { /* Short config writes are not supported */ prlog(PR_ERR, "NPU%d: Unsupported write to pcie control register\n", - ndev->phb->opal_id); + ndev->nvlink.phb->opal_id); return OPAL_PARAMETER; } @@ -407,9 +407,9 @@ static int __npu2_dev_bind_pci_dev(struct phb *phb __unused, pcislot = (char *)dt_prop_get(pci_dt_node, "ibm,slot-label"); prlog(PR_DEBUG, "NPU2: comparing GPU '%s' and NPU2 '%s'\n", - pcislot, dev->slot_label); + pcislot, dev->nvlink.slot_label); - if (streq(pcislot, dev->slot_label)) + if (streq(pcislot, dev->nvlink.slot_label)) return 1; return 0; @@ -420,20 +420,20 @@ static void npu2_dev_bind_pci_dev(struct npu2_dev *dev) struct phb *phb; uint32_t i; - if (dev->pd) + if (dev->nvlink.pd) return; for (i = 0; i < 64; i++) { - if (dev->npu->phb.opal_id == i) + if (dev->npu->phb_nvlink.opal_id == i) continue; phb = pci_get_phb(i); if (!phb) continue; - dev->pd = pci_walk_dev(phb, NULL, __npu2_dev_bind_pci_dev, dev); - if (dev->pd) { - dev->phb = phb; + dev->nvlink.pd = pci_walk_dev(phb, NULL, __npu2_dev_bind_pci_dev, dev); + if (dev->nvlink.pd) { + dev->nvlink.phb = phb; /* Found the device, set the bit in config space */ npu2_set_link_flag(dev, NPU2_DEV_PCI_LINKED); return; @@ -441,7 +441,7 @@ static void npu2_dev_bind_pci_dev(struct npu2_dev *dev) } prlog(PR_INFO, "%s: No PCI device for NPU2 device %04x:00:%02x.0 to bind to. If you expect a GPU to be there, this is a problem.\n", - __func__, dev->npu->phb.opal_id, dev->index); + __func__, dev->npu->phb_nvlink.opal_id, dev->index); } static struct lock pci_npu_phandle_lock = LOCK_UNLOCKED; @@ -628,14 +628,14 @@ static int npu2_dn_fixup(struct phb *phb, struct pci_device *pd, void *data __unused) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); struct npu2_dev *dev; uint32_t speed; const char *label; dev = npu2_bdf_to_dev(p, pd->bdfn); assert(dev); - if (dev->phb || dev->pd) + if (dev->nvlink.phb || dev->nvlink.pd) return 0; npu2_assign_gmb(dev); @@ -675,7 +675,7 @@ static int npu2_dn_fixup(struct phb *phb, return 0; } - dev->slot_label = label; + dev->nvlink.slot_label = label; /* * Bind the emulated PCI device with the real one, which can't @@ -684,14 +684,14 @@ static int npu2_dn_fixup(struct phb *phb, * for it */ npu2_dev_bind_pci_dev(dev); - if (dev->phb && dev->pd && dev->pd->dn) { - if (dt_find_property(dev->pd->dn, "ibm,npu")) - npu2_append_phandle(dev->pd->dn, pd->dn->phandle); + if (dev->nvlink.phb && dev->nvlink.pd && dev->nvlink.pd->dn) { + if (dt_find_property(dev->nvlink.pd->dn, "ibm,npu")) + npu2_append_phandle(dev->nvlink.pd->dn, pd->dn->phandle); else - dt_add_property_cells(dev->pd->dn, "ibm,npu", pd->dn->phandle); + dt_add_property_cells(dev->nvlink.pd->dn, "ibm,npu", pd->dn->phandle); - dt_add_property_cells(pd->dn, "ibm,gpu", dev->pd->dn->phandle); - dev->gpu_bdfn = dev->pd->bdfn; + dt_add_property_cells(pd->dn, "ibm,gpu", dev->nvlink.pd->dn->phandle); + dev->nvlink.gpu_bdfn = dev->nvlink.pd->bdfn; } return 0; @@ -739,7 +739,7 @@ static void npu2_init_ioda_cache(struct npu2 *p) static int64_t npu2_ioda_reset(struct phb *phb, bool purge) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); uint32_t i; if (purge) { @@ -814,7 +814,7 @@ static void npu2_hw_init(struct npu2 *p) { uint64_t val; - npu2_ioda_reset(&p->phb, false); + npu2_ioda_reset(&p->phb_nvlink, false); /* Enable XTS retry mode */ val = npu2_read(p, NPU2_XTS_CFG); @@ -881,7 +881,7 @@ static int64_t npu2_map_pe_dma_window_real(struct phb *phb, uint64_t pci_start_addr, uint64_t pci_mem_size) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); uint64_t end; uint64_t tve; @@ -939,7 +939,7 @@ static int64_t npu2_map_pe_dma_window(struct phb *phb, uint64_t tce_table_size, uint64_t tce_page_size) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); uint64_t tts_encoded; uint64_t data64 = 0; @@ -1008,7 +1008,7 @@ static int64_t npu2_set_pe(struct phb *phb, uint8_t fcompare, uint8_t action) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p; struct npu2_dev *dev; uint64_t reg, val; @@ -1023,15 +1023,20 @@ static int64_t npu2_set_pe(struct phb *phb, dcompare != OPAL_COMPARE_RID_DEVICE_NUMBER || fcompare != OPAL_COMPARE_RID_FUNCTION_NUMBER) return OPAL_UNSUPPORTED; + if (phb->phb_type != phb_type_npu_v2) + return OPAL_PARAMETER; + + p = phb_to_npu2_nvlink(phb); + if (!p) + return OPAL_PARAMETER; - /* Get the NPU2 device */ dev = npu2_bdf_to_dev(p, bdfn); if (!dev) return OPAL_PARAMETER; val = NPU2_CQ_BRICK_BDF2PE_MAP_ENABLE; val = SETFIELD(NPU2_CQ_BRICK_BDF2PE_MAP_PE, val, pe_num); - val = SETFIELD(NPU2_CQ_BRICK_BDF2PE_MAP_BDF, val, dev->gpu_bdfn); + val = SETFIELD(NPU2_CQ_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); if (!NPU2DEV_BRICK(dev)) reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->index/2, @@ -1043,7 +1048,7 @@ static int64_t npu2_set_pe(struct phb *phb, npu2_write(p, reg, val); val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); - val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, dev->gpu_bdfn); + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); p->bdf2pe_cache[dev->index] = val; @@ -1126,7 +1131,7 @@ static int64_t npu2_eeh_next_error(struct phb *phb, uint16_t *pci_error_type, uint16_t *severity) { - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); int i; uint64_t result = 0; @@ -1154,7 +1159,7 @@ static int64_t npu2_tce_kill(struct phb *phb, uint32_t kill_type, uint64_t pe_number, uint32_t tce_size, uint64_t dma_addr, uint32_t npages) { - struct npu2 *npu = phb_to_npu2(phb); + struct npu2 *npu = phb_to_npu2_nvlink(phb); uint32_t tce_page_size; uint64_t val; @@ -1389,7 +1394,7 @@ static uint32_t npu2_populate_pcie_cap(struct npu2_dev *dev, uint32_t start, uint32_t prev_cap) { - struct pci_virt_device *pvd = dev->pvd; + struct pci_virt_device *pvd = dev->nvlink.pvd; uint32_t val; /* Add capability list */ @@ -1471,12 +1476,12 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, uint32_t start, uint32_t prev_cap) { - struct pci_virt_device *pvd = dev->pvd; + struct pci_virt_device *pvd = dev->nvlink.pvd; /* Capbility list */ PCI_VIRT_CFG_INIT_RO(pvd, prev_cap, 1, start); PCI_VIRT_CFG_INIT_RO(pvd, start, 1, PCI_CFG_CAP_ID_VENDOR); - dev->vendor_cap = start; + dev->nvlink.vendor_cap = start; /* Length and version */ PCI_VIRT_CFG_INIT_RO(pvd, start + 2, 1, VENDOR_CAP_LEN); @@ -1503,7 +1508,7 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, static void npu2_populate_cfg(struct npu2_dev *dev) { - struct pci_virt_device *pvd = dev->pvd; + struct pci_virt_device *pvd = dev->nvlink.pvd; struct npu2_pcie_bar *bar; uint32_t pos; @@ -1611,12 +1616,13 @@ static void npu2_populate_devices(struct npu2 *p, /* Walk the link@x nodes to initialize devices */ p->total_devices = 0; - p->phb.scan_map = 0; + p->phb_nvlink.scan_map = 0; dt_for_each_compatible(npu2_dn, link, "ibm,npu-link") { uint32_t group_id; struct npu2_bar *npu2_bar; dev = &p->devices[index]; + dev->type = NPU2_DEV_TYPE_NVLINK; dev->npu = p; dev->dt_node = link; dev->index = dt_prop_get_u32(link, "ibm,npu-link-index"); @@ -1627,7 +1633,7 @@ static void npu2_populate_devices(struct npu2 *p, /* This must be done after calling * npu_allocate_bdfn() */ p->total_devices++; - p->phb.scan_map |= 0x1 << ((dev->bdfn & 0xf8) >> 3); + p->phb_nvlink.scan_map |= 0x1 << ((dev->bdfn & 0xf8) >> 3); dev->pl_xscom_base = dt_prop_get_u64(link, "ibm,npu-phy"); dev->lane_mask = dt_prop_get_u32(link, "ibm,npu-lane-mask"); @@ -1658,10 +1664,10 @@ static void npu2_populate_devices(struct npu2 *p, dev->bars[1].flags = PCI_CFG_BAR_TYPE_MEM | PCI_CFG_BAR_MEM64; /* Initialize PCI virtual device */ - dev->pvd = pci_virt_add_device(&p->phb, dev->bdfn, 0x100, dev); - if (dev->pvd) { - p->phb.scan_map |= - 0x1 << ((dev->pvd->bdfn & 0xf8) >> 3); + dev->nvlink.pvd = pci_virt_add_device(&p->phb_nvlink, dev->bdfn, 0x100, dev); + if (dev->nvlink.pvd) { + p->phb_nvlink.scan_map |= + 0x1 << ((dev->nvlink.pvd->bdfn & 0xf8) >> 3); npu2_populate_cfg(dev); } @@ -1679,8 +1685,8 @@ static void npu2_add_interrupt_map(struct npu2 *p, size_t map_size; uint32_t mask[] = {0xff00, 0x0, 0x0, 0x7}; - assert(p->phb.dt_node); - phb_dn = p->phb.dt_node; + assert(p->phb_nvlink.dt_node); + phb_dn = p->phb_nvlink.dt_node; npu2_phandle = dt_prop_get_u32(dn, "ibm,npcq"); npu2_dn = dt_find_by_phandle(dt_root, npu2_phandle); @@ -1707,7 +1713,7 @@ static void npu2_add_interrupt_map(struct npu2 *p, static void npu2_add_phb_properties(struct npu2 *p) { - struct dt_node *np = p->phb.dt_node; + struct dt_node *np = p->phb_nvlink.dt_node; uint32_t icsp = get_ics_phandle(); uint64_t mm_base, mm_size, mmio_atsd; @@ -1871,20 +1877,20 @@ static void npu2_create_phb(struct dt_node *dn) p->devices = pmem + sizeof(struct npu2); /* Generic PHB */ - p->phb.dt_node = dn; - p->phb.ops = &npu_ops; - p->phb.phb_type = phb_type_npu_v2; + p->phb_nvlink.dt_node = dn; + p->phb_nvlink.ops = &npu_ops; + p->phb_nvlink.phb_type = phb_type_npu_v2; init_lock(&p->lock); - init_lock(&p->phb.lock); - list_head_init(&p->phb.devices); - list_head_init(&p->phb.virt_devices); + init_lock(&p->phb_nvlink.lock); + list_head_init(&p->phb_nvlink.devices); + list_head_init(&p->phb_nvlink.virt_devices); npu2_setup_irqs(p); npu2_populate_devices(p, dn); npu2_add_interrupt_map(p, dn); npu2_add_phb_properties(p); - slot = npu2_slot_create(&p->phb); + slot = npu2_slot_create(&p->phb_nvlink); if (!slot) { /** @@ -1895,7 +1901,7 @@ static void npu2_create_phb(struct dt_node *dn) prlog(PR_ERR, "NPU2: Cannot create PHB slot\n"); } - pci_register_phb(&p->phb, OPAL_DYNAMIC_PHB_ID); + pci_register_phb(&p->phb_nvlink, OPAL_DYNAMIC_PHB_ID); npu2_init_ioda_cache(p); npu2_hw_init(p); @@ -1955,7 +1961,7 @@ static int64_t opal_npu_init_context(uint64_t phb_id, int pasid, uint64_t msr, uint64_t bdf) { struct phb *phb = pci_get_phb(phb_id); - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); uint64_t xts_bdf, xts_bdf_pid = 0; int id, lparshort; @@ -2033,7 +2039,7 @@ opal_call(OPAL_NPU_INIT_CONTEXT, opal_npu_init_context, 4); static int opal_npu_destroy_context(uint64_t phb_id, uint64_t pid, uint64_t bdf) { struct phb *phb = pci_get_phb(phb_id); - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); uint64_t xts_bdf, xts_bdf_pid; uint64_t lparshort; int id, rc = 0; @@ -2081,7 +2087,7 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, uint64_t lpcr) { struct phb *phb = pci_get_phb(phb_id); - struct npu2 *p = phb_to_npu2(phb); + struct npu2 *p = phb_to_npu2_nvlink(phb); struct npu2_dev *ndev = NULL; uint64_t xts_bdf_lpar, rc = OPAL_SUCCESS; int i; @@ -2128,7 +2134,7 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, /* Need to find an NVLink to send the ATSDs for this device over */ for (i = 0; i < p->total_devices; i++) { - if (p->devices[i].gpu_bdfn == bdf) { + if (p->devices[i].nvlink.gpu_bdfn == bdf) { ndev = &p->devices[i]; break; } diff --git a/include/npu2.h b/include/npu2.h index d0c9ac502a70..94e537b4d6ec 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -21,14 +21,14 @@ /* Debugging options */ #define NPU2DBG(p, fmt, a...) prlog(PR_DEBUG, "NPU%d: " fmt, \ - (p)->phb.opal_id, ##a) + (p)->phb_nvlink.opal_id, ##a) #define NPU2INF(p, fmt, a...) prlog(PR_INFO, "NPU%d: " fmt, \ - (p)->phb.opal_id, ##a) + (p)->phb_nvlink.opal_id, ##a) #define NPU2ERR(p, fmt, a...) prlog(PR_ERR, "NPU%d: " fmt, \ - (p)->phb.opal_id, ##a) + (p)->phb_nvlink.opal_id, ##a) #define NPU2DEVLOG(l, p, fmt, a...) prlog(l, "NPU%d:%d:%d.%d " fmt, \ - (p)->npu->phb.opal_id, \ + (p)->npu->phb_nvlink.opal_id, \ ((p)->bdfn >> 8) & 0xff, \ ((p)->bdfn >> 3) & 0x1f, \ (p)->bdfn & 0x7, ##a) @@ -80,18 +80,18 @@ struct npu2_pcie_bar { struct npu2_bar npu2_bar; }; +enum npu2_dev_type { + NPU2_DEV_TYPE_NVLINK, + NPU2_DEV_TYPE_OPENCAPI, +}; + struct npu2; -struct npu2_dev { - uint32_t index; - uint64_t pl_xscom_base; - struct dt_node *dt_node; - struct npu2_pcie_bar bars[2]; - struct npu2 *npu; - /* Device and function numbers are allocated based on GPU - * association. Links to connected to the same GPU will be - * exposed as different functions of the same bus/device. */ - uint32_t bdfn; +struct npu2_dev_nvlink { + /* For NVLink, device and function numbers are allocated based + * on GPU association. Links to connected to the same GPU will + * be exposed as different functions of the same + * bus/device. */ uint32_t gpu_bdfn; /* PCI virtual device and the associated GPU device */ @@ -104,6 +104,20 @@ struct npu2_dev { /* Vendor specific capability */ uint32_t vendor_cap; + /* Used to associate the NPU device with GPU PCI devices */ + const char *slot_label; +}; + +struct npu2_dev { + enum npu2_dev_type type; + uint32_t index; + uint64_t pl_xscom_base; + struct dt_node *dt_node; + struct npu2_pcie_bar bars[2]; + struct npu2 *npu; + + uint32_t bdfn; + /* Which PHY lanes this device is associated with */ uint32_t lane_mask; @@ -114,8 +128,12 @@ struct npu2_dev { unsigned long procedure_tb; uint32_t procedure_status; - /* Used to associate the NPU device with GPU PCI devices */ - const char *slot_label; + /* NVLink */ + struct npu2_dev_nvlink nvlink; + + /* OpenCAPI */ + struct phb phb_ocapi; + uint64_t i2c_port_id_ocapi; }; struct npu2 { @@ -142,12 +160,32 @@ struct npu2 { * tables. */ struct lock lock; - struct phb phb; + /* NVLink */ + struct phb phb_nvlink; }; -static inline struct npu2 *phb_to_npu2(struct phb *phb) +static inline struct npu2 *phb_to_npu2_nvlink(struct phb *phb) +{ + assert(phb->phb_type == phb_type_npu_v2); + return container_of(phb, struct npu2, phb_nvlink); +} + +static inline struct npu2_dev *phb_to_npu2_dev_ocapi(struct phb *phb) +{ + assert(phb->phb_type == phb_type_npu_v2_opencapi); + return container_of(phb, struct npu2_dev, phb_ocapi); +} + +static inline struct phb *npu2_dev_to_phb(struct npu2_dev *ndev) { - return container_of(phb, struct npu2, phb); + switch (ndev->type) { + case NPU2_DEV_TYPE_NVLINK: + return &ndev->npu->phb_nvlink; + case NPU2_DEV_TYPE_OPENCAPI: + return &ndev->phb_ocapi; + default: + assert(false); + } } void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val); diff --git a/include/pci.h b/include/pci.h index c085b6b867bd..695f8f7cf60e 100644 --- a/include/pci.h +++ b/include/pci.h @@ -344,6 +344,7 @@ enum phb_type { phb_type_pcie_v3, phb_type_pcie_v4, phb_type_npu_v2, + phb_type_npu_v2_opencapi, }; struct phb { From patchwork Wed Feb 7 06:50:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870245 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsRs3HkXz9s71 for ; Wed, 7 Feb 2018 17:51:21 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsRs1xGPzF0sn for ; Wed, 7 Feb 2018 17:51:21 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRk3wWCzF0Sb for ; Wed, 7 Feb 2018 17:51:14 +1100 (AEDT) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176n1WN053740 for ; Wed, 7 Feb 2018 01:51:12 -0500 Received: from e06smtp14.uk.ibm.com (e06smtp14.uk.ibm.com [195.75.94.110]) by mx0b-001b2d01.pphosted.com with ESMTP id 2fytf9v2q2-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:11 -0500 Received: from localhost by e06smtp14.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:10 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp14.uk.ibm.com (192.168.101.144) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:08 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176p8IH7536812; Wed, 7 Feb 2018 06:51:08 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ECA314C04A; Wed, 7 Feb 2018 06:44:58 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A2884C046; Wed, 7 Feb 2018 06:44:58 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:58 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 674FDA0278; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:49 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0016-0000-0000-00000520334B X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0017-0000-0000-0000285CE7C1 Message-Id: <55f914f24a8aff663853c3126a161bca8ce84074.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 03/10] platform: Add fields for OpenCAPI platform data X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Add a platform_ocapi struct to store platform-specific values for resetting OpenCAPI devices via I2C and for setting up the ODL PHY. A later patch will add this to the relevant platforms. Signed-off-by: Andrew Donnellan --- v1->v2: - remove odl01 reset data (Fred) - remove i2c 1.8v hack (Fred) --- include/platform.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/platform.h b/include/platform.h index f3af390705f7..1ba0e28ac3d0 100644 --- a/include/platform.h +++ b/include/platform.h @@ -44,6 +44,17 @@ struct bmc_platform { uint32_t ipmi_oem_pnor_access_status; }; +/* OpenCAPI platform-specific I2C information */ +struct platform_ocapi { + uint8_t i2c_engine; /* I2C engine number */ + uint8_t i2c_port; /* I2C port number */ + uint32_t i2c_offset[3]; /* Offsets on I2C device */ + uint8_t i2c_odl0_data[3]; /* Data to reset ODL0 */ + uint8_t i2c_odl1_data[3]; /* Data to reset ODL1 */ + bool odl_phy_swap; /* Swap ODL1 to use brick 2 rather than + * brick 1 lanes */ +}; + /* * Each platform can provide a set of hooks * that can affect the generic code @@ -58,6 +69,9 @@ struct platform { */ const struct bmc_platform *bmc; + /* OpenCAPI platform-specific I2C information */ + const struct platform_ocapi *ocapi; + /* * Probe platform, return true on a match, called before * any allocation has been performed outside of the heap From patchwork Wed Feb 7 06:50:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870246 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsSH5WK2z9s71 for ; Wed, 7 Feb 2018 17:51:43 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsSH3tvhzF0ss for ; Wed, 7 Feb 2018 17:51:43 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRk4wMgzF0TH for ; Wed, 7 Feb 2018 17:51:14 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176n4Qj102067 for ; Wed, 7 Feb 2018 01:51:12 -0500 Received: from e06smtp13.uk.ibm.com (e06smtp13.uk.ibm.com [195.75.94.109]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyv8nrnbe-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:11 -0500 Received: from localhost by e06smtp13.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:10 -0000 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp13.uk.ibm.com (192.168.101.143) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:08 -0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176p8W562980098; Wed, 7 Feb 2018 06:51:08 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 361D711C054; Wed, 7 Feb 2018 06:44:31 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9174111C050; Wed, 7 Feb 2018 06:44:30 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:30 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 6B9ADA027C; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:50 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0012-0000-0000-000005AC2CC0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0013-0000-0000-00001927E2AF Message-Id: <09df0b89fd401f3ba07f3319b0836ed38cbf76f6.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 04/10] p9-adu: Implement update sequence for PowerBus Hotplug Mode Control X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" The PowerBus Hotplug Mode Control register is used to control various things about the PowerBus. Due to hardware timing constraints, updating this register requires more than just a simple SCOM write. There are two sets of pb_hp_mode registers, current (B) and next (A). Updating is done by a SCOM write to the next registers, followed by a SWITCH_AB pulse from the Alter/Display Unit, which triggers a hardware sequence to swap between the current and next registers. The sequence used in this code was derived from the hardware procedures used by Hostboot. This will be used in a subsequent patch as part of the OpenCAPI/NVLink init sequence. Signed-off-by: Andrew Donnellan --- v1->v2: - assert if we don't find master chip (Fred) - make sure we don't accidentally send a SWITCH_CD, per hostboot (Fred) --- hw/Makefile.inc | 2 +- hw/npu2.c | 1 +- hw/p9-adu.c | 184 +++++++++++++++++++++++++++++++++++++++++++++- include/npu2-regs.h | 2 +- include/p9-adu.h | 59 ++++++++++++++- 5 files changed, 245 insertions(+), 3 deletions(-) create mode 100644 hw/p9-adu.c create mode 100644 include/p9-adu.h diff --git a/hw/Makefile.inc b/hw/Makefile.inc index cf8649d00270..27d8a382d784 100644 --- a/hw/Makefile.inc +++ b/hw/Makefile.inc @@ -7,7 +7,7 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o -HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o +HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o p9-adu.o HW=hw/built-in.o # FIXME hack this for now diff --git a/hw/npu2.c b/hw/npu2.c index 3162bb613e7a..b6eff9cb40c8 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -37,6 +37,7 @@ #include #include #include +#include #define NPU2_IRQ_BASE_SHIFT 13 #define NPU2_N_DL_IRQS 23 diff --git a/hw/p9-adu.c b/hw/p9-adu.c new file mode 100644 index 000000000000..d2117d184551 --- /dev/null +++ b/hw/p9-adu.c @@ -0,0 +1,184 @@ +/* Copyright 2013-2018 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Support code for POWER9 Fabric Alter/Display Unit (ADU) hotplug functionality + * + * The ADU acts as a bridge between the PowerBus and the Pervasive Interconnect + * Bus (PIB). + * + * Among other things, the ADU is used to support PowerBus Hotplug. Skiboot + * needs to set the PowerBus Hotplug Mode Control register as part of OpenCAPI + * and NVLink initialisation. + */ + +#include +#include +#include +#include +#include + +/* + * Lock or unlock ADU + */ +static int p9_adu_manage_lock(bool lock) +{ + int rc; + uint64_t val = 0; + struct proc_chip *chip = NULL; + if (lock) { + val |= PU_ALTD_CMD_REG_FBC_LOCKED; + val |= PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM; + val |= PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS; + } + + while ((chip = next_chip(chip))) { + rc = xscom_write(chip->id, PU_ALTD_CMD_REG, val); + if (rc != OPAL_SUCCESS) { + /* TODO: Lock picking support */ + prlog(PR_ERR, + "ADU: Error %d writing command (chip %d)\n", + rc, chip->id); + return OPAL_HARDWARE; + } + } + return OPAL_SUCCESS; +} + +static uint32_t find_master_chip(void) +{ + uint64_t reg; + struct proc_chip *chip = NULL; + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PB_CENT_HP_MODE_CURR, ®); + if (reg & PB_CFG_MASTER_CHIP) + return chip->id; + } + assert(false); +} + +/* + * Trigger a SWITCH_AB pulse to switch the current PowerBus Hotplug Mode Control + * register set. + * + * Overview of sequence: + * + * - acquire lock and reset all ADUs + * - configure all ADUs for AB switch + * - configure one ADU (on the fabric master chip) to issue a + * quiesce/switch/reinit + * - check status + * - clear switch selectors + * - reset all ADUs + * - unlock + */ +static void p9_adu_switch_ab(void) +{ + uint32_t gcid = find_master_chip(); + struct proc_chip *chip = NULL; + uint64_t reg; + uint64_t val; + int rc = OPAL_SUCCESS; + + /* + * There's a performance issue on P9DD1 that requires a workaround: + * see IBM defect HW397129. However, this code isn't expected to be + * used on DD1 machines. + */ + + rc = p9_adu_manage_lock(true); + if (rc) + goto err; + + /* Set PB_SWITCH_AB on all ADUs */ + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PU_SND_MODE_REG, ®); + reg |= PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_CD; + rc = xscom_write(chip->id, PU_SND_MODE_REG, reg); + if (rc) + goto err_switch; + } + + /* Set address 0 */ + rc = xscom_write(gcid, PU_ALTD_ADDR_REG, 0); + if (rc) + goto err_switch; + + /* Configure ADU to issue quiesce + switch + reinit */ + val = PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE; + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT, + val, QUIESCE_SWITCH_WAIT_COUNT); + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT; + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT, + val, INIT_SWITCH_WAIT_COUNT); + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH; /* see HW397129, DD2 */ + rc = xscom_write(gcid, PU_ALTD_OPTION_REG, val); + if (rc) + goto err_switch; + + /* Set up command */ + val = PU_ALTD_CMD_REG_FBC_LOCKED; + val |= PU_ALTD_CMD_REG_FBC_ALTD_START_OP; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_SCOPE, val, + PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM); + val |= PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY; + val |= PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TTYPE, val, + PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER); + val |= PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TSIZE, val, + PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1); + xscom_write(gcid, PU_ALTD_CMD_REG, val); + + /* + * TODO: check ADU status is consistent, see + * p9_build_smp_adu_check_status() in hostboot + */ + +err_switch: + /* Reset switch controls */ + chip = NULL; + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PU_SND_MODE_REG, ®); + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_CD; + xscom_write(chip->id, PU_SND_MODE_REG, reg); + } + + /* Reset ADUs */ + p9_adu_manage_lock(true); + +err: + /* Unlock */ + p9_adu_manage_lock(false); +} + +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val) +{ + /* Write next value */ + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); + + /* Send switch pulse */ + p9_adu_switch_ab(); + + /* Now that switch is complete, overwrite old value */ + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); +} diff --git a/include/npu2-regs.h b/include/npu2-regs.h index b190d0f0767a..b6dae7bb13fd 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -36,8 +36,6 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define MCD_BANK_CN_VALID PPC_BIT(0) #define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) #define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) -#define PB_CENT_HP_MODE_CURR 0x5011c0c -#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) #define PB_CENT_MODE 0x5011c0a #define PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT PPC_BITMASK(42,48) diff --git a/include/p9-adu.h b/include/p9-adu.h new file mode 100644 index 000000000000..a418a23cb293 --- /dev/null +++ b/include/p9-adu.h @@ -0,0 +1,59 @@ +/* Copyright 2013-2018 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val); + +#define PU_ALTD_ADDR_REG 0x0090000 + +#define PU_ALTD_CMD_REG 0x0090001 +#define PU_ALTD_CMD_REG_FBC_ALTD_START_OP PPC_BIT(2) +#define PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS PPC_BIT(3) +#define PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM PPC_BIT(4) +#define PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE PPC_BIT(6) +#define PU_ALTD_CMD_REG_FBC_LOCKED PPC_BIT(11) +#define PU_ALTD_CMD_REG_FBC_LOCK_ID PPC_BITMASK(12, 15) +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE PPC_BITMASK(16, 18) +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM 0b101 +#define PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY PPC_BIT(20) +#define PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE PPC_BIT(24) +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE PPC_BITMASK(25, 31) +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER 0b0110001 +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE PPC_BITMASK(32, 39) +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1 0b00000010 + +#define PU_ALTD_OPTION_REG 0x0090002 +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE PPC_BIT(23) +#define PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT PPC_BITMASK(28, 47) +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT PPC_BIT(51) +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH PPC_BIT(52) +#define PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT PPC_BITMASK(54, 63) + +#define QUIESCE_SWITCH_WAIT_COUNT 128 +#define INIT_SWITCH_WAIT_COUNT 128 + +#define PU_SND_MODE_REG 0x0090021 +#define PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB PPC_BIT(30) +#define PU_SND_MODE_REG_ENABLE_PB_SWITCH_CD PPC_BIT(31) + +/* Hotplug mode registers */ +#define PB_WEST_HP_MODE_NEXT 0x501180B +#define PB_CENT_HP_MODE_NEXT 0x5011C0B +#define PB_EAST_HP_MODE_NEXT 0x501200B +#define PB_WEST_HP_MODE_CURR 0x501180C +#define PB_CENT_HP_MODE_CURR 0x5011C0C +#define PB_EAST_HP_MODE_CURR 0x501200C +#define PB_CFG_MASTER_CHIP PPC_BIT(0) +#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) From patchwork Wed Feb 7 06:50:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870255 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsVs0gfZz9ryk for ; Wed, 7 Feb 2018 17:53:57 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsVr6mfxzF13j for ; Wed, 7 Feb 2018 17:53:56 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRp48rrzF0Sb for ; Wed, 7 Feb 2018 17:51:18 +1100 (AEDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176o5rc133725 for ; Wed, 7 Feb 2018 01:51:16 -0500 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyumsa05r-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:15 -0500 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:13 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:11 -0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAHS26542122; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7A6B611C050; Wed, 7 Feb 2018 06:44:33 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5159D11C05C; Wed, 7 Feb 2018 06:44:32 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:32 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 7242FA03A0; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:51 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0020-0000-0000-000003F31B68 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0021-0000-0000-0000428594BF Message-Id: <62638d9cb3f9d53e156fdd86abbbd286dd2d4797.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 05/10] npu2-opencapi: Configure NPU for OpenCAPI X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Scan the device tree for NPUs with OpenCAPI links and configure the NPU per the initialisation sequence in the NPU OpenCAPI workbook. Training of individual links and setup of per-AFU/link configuration will be in a later patch. Signed-off-by: Andrew Donnellan Signed-off-by: Frederic Barrat Reviewed-by: Frederic Barrat --- v1->v2: - fix coding style (Fred) v2->v3: - Mask FIRs for repurposed nvlink signals (Fred) - Fix IRQ register mnemonic (Fred) v3->v4: - Fix comment about workbook inits (Fred) - Redo the way we abort NVLink init when OpenCAPI configured (Alistair) v4->v5: - Fix the --- I accidentally deleted in the patch email... --- core/init.c | 2 +- hw/Makefile.inc | 3 +- hw/npu2-opencapi.c | 838 +++++++++++++++++++++++++++++++++++++++++++++- hw/npu2.c | 9 +- include/npu2-regs.h | 90 +++++- include/npu2.h | 2 +- include/skiboot.h | 1 +- 7 files changed, 944 insertions(+), 1 deletion(-) create mode 100644 hw/npu2-opencapi.c diff --git a/core/init.c b/core/init.c index ec9f32981cd4..fcf7a3fdccd8 100644 --- a/core/init.c +++ b/core/init.c @@ -1042,6 +1042,8 @@ void __noreturn __nomcount main_cpu_entry(const void *fdt) /* Probe NPUs */ probe_npu(); probe_npu2(); + /* TODO: Eventually, we'll do NVLink and OpenCAPI together */ + probe_npu2_opencapi(); /* Initialize PCI */ pci_init_slots(); diff --git a/hw/Makefile.inc b/hw/Makefile.inc index 27d8a382d784..cfc86468c396 100644 --- a/hw/Makefile.inc +++ b/hw/Makefile.inc @@ -7,7 +7,8 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o -HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o p9-adu.o +HW_OBJS += npu2-common.o npu2-opencapi.o phys-map.o sbe-p9.o capp.o occ-sensor.o +HW_OBJS += vas.o p9-adu.o HW=hw/built-in.o # FIXME hack this for now diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c new file mode 100644 index 000000000000..6ff3d3317ee0 --- /dev/null +++ b/hw/npu2-opencapi.c @@ -0,0 +1,838 @@ +/* Copyright 2013-2018 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Support for OpenCAPI on POWER9 NPUs + * + * This file provides support for OpenCAPI as implemented on POWER9. + * + * At present, we initialise the NPU separately from the NVLink code in npu2.c. + * As such, we don't currently support mixed NVLink and OpenCAPI configurations + * on the same NPU for machines such as Witherspoon. + * + * Procedure references in this file are to the POWER9 OpenCAPI NPU Workbook + * (IBM internal document). + * + * TODO: + * - Support for mixed NVLink and OpenCAPI on the same NPU + * - Support for link ganging (one AFU using multiple links) + * - Link reset and error handling + * - Presence detection + * - Consume HDAT NPU information + * - LPC Memory support + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NPU_IRQ_LEVELS 35 + +static inline uint64_t index_to_stack(uint64_t index) { + switch (index) { + case 2: + case 3: + return NPU2_STACK_STCK_1; + break; + case 4: + case 5: + return NPU2_STACK_STCK_2; + break; + default: + assert(false); + } +} + +static inline uint64_t index_to_stacku(uint64_t index) { + switch (index) { + case 2: + case 3: + return NPU2_STACK_STCK_1U; + break; + case 4: + case 5: + return NPU2_STACK_STCK_2U; + break; + default: + assert(false); + } +} + +static inline uint64_t index_to_block(uint64_t index) { + switch (index) { + case 2: + case 4: + return NPU2_BLOCK_OTL0; + break; + case 3: + case 5: + return NPU2_BLOCK_OTL1; + break; + default: + assert(false); + } +} + +/* Procedure 13.1.3.1 - select OCAPI vs NVLink for bricks 2-3/4-5 */ + +static void set_transport_mux_controls(uint32_t gcid, uint32_t scom_base, + int index, enum npu2_dev_type type) +{ + /* Step 1 - Set Transport MUX controls to select correct OTL or NTL */ + uint64_t reg; + uint64_t field; + + /* TODO: Rework this to select for NVLink too */ + assert(type == NPU2_DEV_TYPE_OPENCAPI); + + prlog(PR_DEBUG, "OCAPI: %s: Setting transport mux controls\n", __func__); + + /* Optical IO Transport Mux Config for Bricks 0-2 and 4-5 */ + reg = npu2_scom_read(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B); + switch (index) { + case 0: + case 1: + /* not valid for OpenCAPI */ + assert(false); + break; + case 2: /* OTL1.0 */ + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); + field &= ~0b100; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, + field); + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); + field |= 0b10; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, + field); + break; + case 3: /* OTL1.1 */ + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg); + field &= ~0b010; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2, reg, + field); + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg); + field |= 0b01; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1, reg, + field); + break; + case 4: /* OTL2.0 */ + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); + field |= 0b10; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, + field); + break; + case 5: /* OTL2.1 */ + field = GETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg); + field |= 0b01; + reg = SETFIELD(NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5, reg, + field); + break; + default: + assert(false); + } + npu2_scom_write(gcid, scom_base, NPU2_MISC_OPTICAL_IO_CFG0, + NPU2_MISC_DA_LEN_8B, reg); + + /* + * PowerBus Optical Miscellaneous Config Register - select + * OpenCAPI for b4/5 and A-Link for b3 + */ + xscom_read(gcid, PU_IOE_PB_MISC_CFG, ®); + switch (index) { + case 0: + case 1: + case 2: + case 3: + break; + case 4: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB, reg, 1); + break; + case 5: + reg = SETFIELD(PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB, reg, 1); + break; + } + xscom_write(gcid, PU_IOE_PB_MISC_CFG, reg); +} + +static void enable_odl_phy_mux(uint32_t gcid, int index) +{ + uint64_t reg; + uint64_t phy_config_scom; + prlog(PR_DEBUG, "OCAPI: %s: Enabling ODL to PHY MUXes\n", __func__); + /* Step 2 - Enable MUXes for ODL to PHY connection */ + switch (index) { + case 2: + case 3: + phy_config_scom = OBUS_LL0_IOOL_PHY_CONFIG; + break; + case 4: + case 5: + phy_config_scom = OBUS_LL3_IOOL_PHY_CONFIG; + break; + default: + assert(false); + } + + /* PowerBus OLL PHY Training Config Register */ + xscom_read(gcid, phy_config_scom, ®); + + /* Enable ODLs to use shared PHYs */ + reg |= OBUS_IOOL_PHY_CONFIG_ODL0_ENABLED; + reg |= OBUS_IOOL_PHY_CONFIG_ODL1_ENABLED; + + /* + * Swap ODL1 to use brick 2 lanes instead of brick 1 lanes if using a + * 22-pin cable for OpenCAPI connection. + */ + if (platform.ocapi->odl_phy_swap) + reg |= OBUS_IOOL_PHY_CONFIG_ODL_PHY_SWAP; + else + reg &= ~OBUS_IOOL_PHY_CONFIG_ODL_PHY_SWAP; + + /* Disable A-Link link layers */ + reg &= ~OBUS_IOOL_PHY_CONFIG_LINK0_OLL_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_LINK1_OLL_ENABLED; + + /* Disable NV-Link link layers */ + reg &= ~OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED; + reg &= ~OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED; + xscom_write(gcid, phy_config_scom, reg); +} + +static void disable_alink_fp(uint32_t gcid) +{ + uint64_t reg = 0; + + prlog(PR_DEBUG, "OCAPI: %s: Disabling A-Link framer/parsers\n", __func__); + /* Step 3 - Disable A-Link framers/parsers */ + /* TODO: Confirm if needed on OPAL system */ + + reg |= PU_IOE_PB_FP_CFG_FP0_FMR_DISABLE; + reg |= PU_IOE_PB_FP_CFG_FP0_PRS_DISABLE; + reg |= PU_IOE_PB_FP_CFG_FP1_FMR_DISABLE; + reg |= PU_IOE_PB_FP_CFG_FP1_PRS_DISABLE; + xscom_write(gcid, PU_IOE_PB_FP01_CFG, reg); + xscom_write(gcid, PU_IOE_PB_FP23_CFG, reg); + xscom_write(gcid, PU_IOE_PB_FP45_CFG, reg); + xscom_write(gcid, PU_IOE_PB_FP67_CFG, reg); +} + +static void set_pb_hp_opencapi(uint32_t gcid, int index) +{ + /* Step 4 - Set PowerBus HotPlug Mode Registers */ + uint64_t reg; + + prlog(PR_DEBUG, "OCAPI: %s: Setting PowerBus Hotplug Mode registers\n", __func__); + + xscom_read(gcid, PB_WEST_HP_MODE_CURR, ®); + switch (index) { + case 2: + case 3: + /* Configure OPT0 as an OpenCAPI link */ + reg = SETFIELD(PPC_BITMASK(32, 33), reg, 0b01); + break; + case 4: + case 5: + /* Configure OPT3 as an OpenCAPI link */ + reg = SETFIELD(PPC_BITMASK(38, 39), reg, 0b01); + break; + default: + assert(false); + } + + p9_adu_set_pb_hp_mode(gcid, reg); +} + +static void enable_xsl_clocks(uint32_t gcid, uint32_t scom_base, int index) +{ + /* Step 5 - Enable Clocks in XSL */ + + prlog(PR_DEBUG, "OCAPI: %s: Enable clocks in XSL\n", __func__); + + npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(index_to_stack(index), + NPU2_BLOCK_XSL, + NPU2_XSL_WRAP_CFG), + NPU2_MISC_DA_LEN_8B, NPU2_XSL_WRAP_CFG_XSLO_CLOCK_ENABLE); +} + +#define CQ_CTL_STATUS_TIMEOUT 10 /* milliseconds */ + +static int set_fence_control(uint32_t gcid, uint32_t scom_base, + int index, uint8_t status) +{ + int stack, block; + uint64_t reg, status_field; + uint8_t status_val; + uint64_t fence_control; + uint64_t timeout = mftb() + msecs_to_tb(CQ_CTL_STATUS_TIMEOUT); + + stack = index_to_stack(index); + block = index_to_block(index); + + fence_control = NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, + block == NPU2_BLOCK_OTL0 ? + NPU2_CQ_CTL_FENCE_CONTROL_0 : + NPU2_CQ_CTL_FENCE_CONTROL_1); + + reg = SETFIELD(NPU2_CQ_CTL_FENCE_CONTROL_REQUEST_FENCE, 0ull, status); + npu2_scom_write(gcid, scom_base, fence_control, + NPU2_MISC_DA_LEN_8B, reg); + + /* Wait for fence status to update */ + if (index_to_block(index) == NPU2_BLOCK_OTL0) + status_field = NPU2_CQ_CTL_STATUS_BRK0_AM_FENCED; + else + status_field = NPU2_CQ_CTL_STATUS_BRK1_AM_FENCED; + + do { + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(index_to_stack(index), + NPU2_BLOCK_CTL, + NPU2_CQ_CTL_STATUS), + NPU2_MISC_DA_LEN_8B); + status_val = GETFIELD(status_field, reg); + if (status_val == status) + return OPAL_SUCCESS; + time_wait_ms(1); + } while (tb_compare(mftb(), timeout) == TB_ABEFOREB); + + /** + * @fwts-label OCAPIFenceStatusTimeout + * @fwts-advice The NPU fence status did not update as expected. This + * could be the result of a firmware or hardware bug. OpenCAPI + * functionality could be broken. + */ + prlog(PR_ERR, + "OCAPI: Fence status for brick %d stuck: expected 0x%x, got 0x%x\n", + index, status, status_val); + return OPAL_HARDWARE; +} + +static void set_npcq_config(uint32_t gcid, uint32_t scom_base, int index) +{ + uint64_t reg, stack, block; + + prlog(PR_DEBUG, "OCAPI: %s: Set NPCQ Config\n", __func__); + /* Step 6 - Set NPCQ configuration */ + /* CQ_CTL Misc Config Register #0 */ + stack = index_to_stack(index); + block = index_to_block(index); + + /* Enable OTL */ + npu2_scom_write(gcid, scom_base, NPU2_OTL_CONFIG0(stack, block), + NPU2_MISC_DA_LEN_8B, NPU2_OTL_CONFIG0_EN); + set_fence_control(gcid, scom_base, index, 0b01); + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, + NPU2_CQ_CTL_MISC_CFG), + NPU2_MISC_DA_LEN_8B); + /* Set OCAPI mode */ + reg |= NPU2_CQ_CTL_MISC_CFG_CONFIG_OCAPI_MODE; + if (block == NPU2_BLOCK_OTL0) + reg |= NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL0_ENABLE; + else + reg |= NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL1_ENABLE; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, + NPU2_CQ_CTL_MISC_CFG), + NPU2_MISC_DA_LEN_8B, reg); + + /* NPU Fenced */ + set_fence_control(gcid, scom_base, index, 0b11); + + /* NPU Half Fenced */ + set_fence_control(gcid, scom_base, index, 0b10); + + /* CQ_DAT Misc Config Register #1 */ + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_DAT, + NPU2_CQ_DAT_MISC_CFG), + NPU2_MISC_DA_LEN_8B); + /* Set OCAPI mode for bricks 2-5 */ + reg |= NPU2_CQ_DAT_MISC_CFG_CONFIG_OCAPI_MODE; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_DAT, + NPU2_CQ_DAT_MISC_CFG), + NPU2_MISC_DA_LEN_8B, reg); + + /* CQ_SM Misc Config Register #0 */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_CQ_SM_MISC_CFG0), + NPU2_MISC_DA_LEN_8B); + /* Set OCAPI mode for bricks 2-5 */ + reg |= NPU2_CQ_SM_MISC_CFG0_CONFIG_OCAPI_MODE; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_CQ_SM_MISC_CFG0), + NPU2_MISC_DA_LEN_8B, reg); + } +} + +static void enable_xsl_xts_interfaces(uint32_t gcid, uint32_t scom_base, int index) +{ + uint64_t reg; + + prlog(PR_DEBUG, "OCAPI: %s: Enable XSL-XTS Interfaces\n", __func__); + /* Step 7 - Enable XSL-XTS interfaces */ + /* XTS Config Register - Enable XSL-XTS interface */ + reg = npu2_scom_read(gcid, scom_base, NPU2_XTS_CFG, NPU2_MISC_DA_LEN_8B); + reg |= NPU2_XTS_CFG_OPENCAPI; + npu2_scom_write(gcid, scom_base, NPU2_XTS_CFG, NPU2_MISC_DA_LEN_8B, reg); + + /* XTS Config2 Register - Enable XSL1/2 */ + reg = npu2_scom_read(gcid, scom_base, NPU2_XTS_CFG2, NPU2_MISC_DA_LEN_8B); + switch (index_to_stack(index)) { + case NPU2_STACK_STCK_1: + reg |= NPU2_XTS_CFG2_XSL1_ENA; + break; + case NPU2_STACK_STCK_2: + reg |= NPU2_XTS_CFG2_XSL2_ENA; + break; + } + npu2_scom_write(gcid, scom_base, NPU2_XTS_CFG2, NPU2_MISC_DA_LEN_8B, reg); +} + +static void enable_sm_allocation(uint32_t gcid, uint32_t scom_base, int index) +{ + uint64_t reg, block; + int stack = index_to_stack(index); + + prlog(PR_DEBUG, "OCAPI: %s: Enable State Machine Allocation\n", __func__); + /* Step 8 - Enable state-machine allocation */ + /* Low-Water Marks Registers - Enable state machine allocation */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_LOW_WATER_MARKS), + NPU2_MISC_DA_LEN_8B); + reg |= NPU2_LOW_WATER_MARKS_ENABLE_MACHINE_ALLOC; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_LOW_WATER_MARKS), + NPU2_MISC_DA_LEN_8B, reg); + } +} + +static void enable_pb_snooping(uint32_t gcid, uint32_t scom_base, int index) +{ + uint64_t reg, block; + int stack = index_to_stack(index); + + prlog(PR_DEBUG, "OCAPI: %s: Enable PowerBus snooping\n", __func__); + /* Step 9 - Enable PowerBus snooping */ + /* CQ_SM Misc Config Register #0 - Enable PowerBus snooping */ + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_CQ_SM_MISC_CFG0), + NPU2_MISC_DA_LEN_8B); + reg |= NPU2_CQ_SM_MISC_CFG0_CONFIG_ENABLE_PBUS; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, block, + NPU2_CQ_SM_MISC_CFG0), + NPU2_MISC_DA_LEN_8B, reg); + } +} + +static void brick_config(uint32_t gcid, uint32_t scom_base, int index) +{ + set_transport_mux_controls(gcid, scom_base, index, NPU2_DEV_TYPE_OPENCAPI); + enable_odl_phy_mux(gcid, index); + disable_alink_fp(gcid); + set_pb_hp_opencapi(gcid, index); + enable_xsl_clocks(gcid, scom_base, index); + set_npcq_config(gcid, scom_base, index); + enable_xsl_xts_interfaces(gcid, scom_base, index); + enable_sm_allocation(gcid, scom_base, index); + enable_pb_snooping(gcid, scom_base, index); +} + +/* Procedure 13.1.3.5 - TL Configuration */ +static void tl_config(uint32_t gcid, uint32_t scom_base, uint64_t index) +{ + uint64_t reg; + uint64_t stack = index_to_stack(index); + uint64_t block = index_to_block(index); + + prlog(PR_DEBUG, "OCAPI: %s: TL Configuration\n", __func__); + /* OTL Config 0 Register */ + reg = 0; + /* OTL Enable */ + reg |= NPU2_OTL_CONFIG0_EN; + /* Block PE Handle from ERAT Index */ + reg |= NPU2_OTL_CONFIG0_BLOCK_PE_HANDLE; + /* OTL Brick ID */ + reg = SETFIELD(NPU2_OTL_CONFIG0_BRICKID, reg, index - 2); + /* ERAT Hash 0 */ + reg = SETFIELD(NPU2_OTL_CONFIG0_ERAT_HASH_0, reg, 0b011001); + /* ERAT Hash 1 */ + reg = SETFIELD(NPU2_OTL_CONFIG0_ERAT_HASH_1, reg, 0b000111); + /* ERAT Hash 2 */ + reg = SETFIELD(NPU2_OTL_CONFIG0_ERAT_HASH_2, reg, 0b101100); + /* ERAT Hash 3 */ + reg = SETFIELD(NPU2_OTL_CONFIG0_ERAT_HASH_3, reg, 0b100110); + npu2_scom_write(gcid, scom_base, NPU2_OTL_CONFIG0(stack, block), + NPU2_MISC_DA_LEN_8B, reg); + + /* OTL Config 1 Register */ + reg = 0; + /* + * We leave Template 1-3 bits at 0 to force template 0 as required + * for unknown devices. + * + * Template 0 Transmit Rate is set to most conservative setting which + * will always be supported. Other Template Transmit rates are left + * unset and will be set later by OS. + */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_TEMP0_RATE, reg, 0b1111); + /* Extra wait cycles TXI-TXO */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_DRDY_WAIT, reg, 0b001); + /* Minimum Frequency to Return TLX Credits to AFU */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_CRET_FREQ, reg, 0b001); + /* Frequency to add age to Transmit Requests */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_AGE_FREQ, reg, 0b11000); + /* Response High Priority Threshold */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_RS2_HPWAIT, reg, 0b011011); + /* 4-slot Request High Priority Threshold */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_RQ4_HPWAIT, reg, 0b011011); + /* 6-slot Request High Priority */ + reg = SETFIELD(NPU2_OTL_CONFIG1_TX_RQ6_HPWAIT, reg, 0b011011); + /* Stop the OCAPI Link on Uncorrectable Error + * TODO: Confirm final value - disabled for debug */ + + npu2_scom_write(gcid, scom_base, NPU2_OTL_CONFIG1(stack, block), + NPU2_MISC_DA_LEN_8B, reg); + + /* TLX Credit Configuration Register */ + reg = 0; + /* VC0/VC3/DCP0/DCP1 credits to send to AFU */ + reg = SETFIELD(NPU2_OTL_TLX_CREDITS_VC0_CREDITS, reg, 0x40); + reg = SETFIELD(NPU2_OTL_TLX_CREDITS_VC3_CREDITS, reg, 0x40); + reg = SETFIELD(NPU2_OTL_TLX_CREDITS_DCP0_CREDITS, reg, 0x80); + reg = SETFIELD(NPU2_OTL_TLX_CREDITS_DCP1_CREDITS, reg, 0x80); + npu2_scom_write(gcid, scom_base, NPU2_OTL_TLX_CREDITS(stack, block), + NPU2_MISC_DA_LEN_8B, reg); +} + +/* Detect Nimbus DD2.0 and DD2.01 */ +static int get_nimbus_level(void) +{ + struct proc_chip *chip = next_chip(NULL); + + if (chip && chip->type == PROC_CHIP_P9_NIMBUS) + return chip->ec_level & 0xff; + return -1; +} + +/* Procedure 13.1.3.6 - Address Translation Configuration */ +static void address_translation_config(uint32_t gcid, uint32_t scom_base, + uint64_t index) +{ + int chip_level; + uint64_t reg; + uint64_t stack = index_to_stack(index); + + prlog(PR_DEBUG, "OCAPI: %s: Address Translation Configuration\n", __func__); + /* PSL_SCNTL_A0 Register */ + /* + * ERAT shared between multiple AFUs + * + * The workbook has this bit around the wrong way from the hardware. + * + * TODO: handle correctly with link ganging + */ + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, + NPU2_XSL_PSL_SCNTL_A0), + NPU2_MISC_DA_LEN_8B); + reg |= NPU2_XSL_PSL_SCNTL_A0_MULTI_AFU_DIAL; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, + NPU2_XSL_PSL_SCNTL_A0), + NPU2_MISC_DA_LEN_8B, reg); + + chip_level = get_nimbus_level(); + if (chip_level == 0x20) { + /* + * Errata HW408041 (section 15.1.10 of NPU workbook) + * "RA mismatch when both tlbie and checkout response + * are seen in same cycle" + */ + /* XSL_GP Register - Bloom Filter Disable */ + reg = npu2_scom_read(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_GP), + NPU2_MISC_DA_LEN_8B); + /* To update XSL_GP, we must first write a magic value to it */ + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_GP), + NPU2_MISC_DA_LEN_8B, 0x0523790323000000); + reg &= ~NPU2_XSL_GP_BLOOM_FILTER_ENABLE; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_GP), + NPU2_MISC_DA_LEN_8B, reg); + } + + if (chip_level == 0x20 || chip_level == 0x21) { + /* + * DD2.0/2.1 EOA Bug. Fixed in DD2.2 + */ + reg = 0x32F8000000000001; + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, + NPU2_XSL_DEF), + NPU2_MISC_DA_LEN_8B, reg); + } +} + +/* TODO: Merge this with NVLink implementation - we don't use the npu2_bar + * wrapper for the PHY BARs yet */ +static void write_bar(uint32_t gcid, uint32_t scom_base, uint64_t reg, + uint64_t addr, uint64_t size) +{ + uint64_t val; + int block; + switch (NPU2_REG(reg)) { + case NPU2_PHY_BAR: + val = SETFIELD(NPU2_PHY_BAR_ADDR, 0ul, addr >> 21); + val = SETFIELD(NPU2_PHY_BAR_ENABLE, val, 1); + break; + case NPU2_NTL0_BAR: + case NPU2_NTL1_BAR: + val = SETFIELD(NPU2_NTL_BAR_ADDR, 0ul, addr >> 16); + val = SETFIELD(NPU2_NTL_BAR_SIZE, val, ilog2(size >> 16)); + val = SETFIELD(NPU2_NTL_BAR_ENABLE, val, 1); + break; + case NPU2_GENID_BAR: + val = SETFIELD(NPU2_GENID_BAR_ADDR, 0ul, addr >> 16); + val = SETFIELD(NPU2_GENID_BAR_ENABLE, val, 1); + break; + default: + val = 0ul; + } + + for (block = NPU2_BLOCK_SM_0; block <= NPU2_BLOCK_SM_3; block++) { + npu2_scom_write(gcid, scom_base, NPU2_REG_OFFSET(0, block, reg), + NPU2_MISC_DA_LEN_8B, val); + prlog(PR_DEBUG, "OCAPI: Setting BAR %llx to %llx\n", + NPU2_REG_OFFSET(0, block, reg), val); + } +} + +static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, + uint64_t reg[]) +{ + uint64_t addr, size; + + prlog(PR_DEBUG, "OCAPI: patching up PHY0 bar, %s\n", __func__); + phys_map_get(gcid, NPU_PHY, 0, &addr, &size); + write_bar(gcid, scom_base, + NPU2_REG_OFFSET(NPU2_STACK_STCK_2, 0, NPU2_PHY_BAR), + addr, size); + prlog(PR_DEBUG, "OCAPI: patching up PHY1 bar, %s\n", __func__); + phys_map_get(gcid, NPU_PHY, 1, &addr, &size); + write_bar(gcid, scom_base, + NPU2_REG_OFFSET(NPU2_STACK_STCK_1, 0, NPU2_PHY_BAR), + addr, size); + + prlog(PR_DEBUG, "OCAPI: setup global mmio, %s\n", __func__); + phys_map_get(gcid, NPU_REGS, 0, &addr, &size); + write_bar(gcid, scom_base, + NPU2_REG_OFFSET(NPU2_STACK_STCK_0, 0, NPU2_PHY_BAR), + addr, size); + reg[0] = addr; + reg[1] = size; +} + +static void mask_nvlink_fir(struct npu2 *p) +{ + uint64_t reg; + + /* + * From section 13.1.3.10 of the NPU workbook: "the NV-Link + * Datalink Layer Stall and NoStall signals are used for a + * different purpose when the link is configured for + * OpenCAPI. Therefore, the corresponding bits in NPU FIR + * Register 1 must be masked and configured to NOT cause the + * NPU to go into Freeze or Fence mode or send an Interrupt." + * + * FIXME: will need to revisit when mixing nvlink with + * opencapi. Assumes an opencapi-only setup on both PHYs for + * now. + */ + + /* Mask FIRs */ + xscom_read(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, ®); + reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0xFFF); + xscom_write(p->chip_id, p->xscom_base + NPU2_MISC_FIR_MASK1, reg); + + /* freeze disable */ + reg = npu2_scom_read(p->chip_id, p->xscom_base, + NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B); + reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + npu2_scom_write(p->chip_id, p->xscom_base, + NPU2_MISC_FREEZE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); + + /* fence disable */ + reg = npu2_scom_read(p->chip_id, p->xscom_base, + NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B); + reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + npu2_scom_write(p->chip_id, p->xscom_base, + NPU2_MISC_FENCE_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); + + /* irq disable */ + reg = npu2_scom_read(p->chip_id, p->xscom_base, + NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B); + reg = SETFIELD(PPC_BITMASK(0, 11), reg, 0); + npu2_scom_write(p->chip_id, p->xscom_base, + NPU2_MISC_IRQ_ENABLE1, NPU2_MISC_DA_LEN_8B, reg); +} + +static int setup_irq(struct npu2 *p) +{ + uint64_t reg, mmio_addr; + uint32_t base; + + base = xive_alloc_ipi_irqs(p->chip_id, NPU_IRQ_LEVELS, 64); + if (base == XIVE_IRQ_ERROR) { + /** + * @fwts-label OCAPIIRQAllocationFailed + * @fwts-advice OpenCAPI IRQ setup failed. This is probably + * a firmware bug. OpenCAPI functionality will be broken. + */ + prlog(PR_ERR, "OCAPI: Couldn't allocate interrupts for NPU\n"); + return -1; + } + p->irq_base = base; + + xive_register_ipi_source(base, NPU_IRQ_LEVELS, NULL, NULL); + mmio_addr = (uint64_t ) xive_get_trigger_port(base); + prlog(PR_DEBUG, "OCAPI: NPU base irq %d @%llx\n", base, mmio_addr); + reg = (mmio_addr & NPU2_MISC_IRQ_BASE_MASK) << 13; + npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_BASE, + NPU2_MISC_DA_LEN_8B, reg); + /* + * setup page size = 64k + * + * OS type is set to AIX: opal also runs with 2 pages per interrupt, + * so to cover the max offset for 35 levels of interrupt, we need + * bits 41 to 46, which is what the AIX setting does. There's no + * other meaning for that AIX setting. + */ + reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_CFG, + NPU2_MISC_DA_LEN_8B); + reg |= NPU2_MISC_CFG_IPI_PS; + reg &= ~NPU2_MISC_CFG_IPI_OS; + npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_CFG, + NPU2_MISC_DA_LEN_8B, reg); + + /* enable translation interrupts for all bricks */ + reg = npu2_scom_read(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE2, + NPU2_MISC_DA_LEN_8B); + reg |= PPC_BIT(0) | PPC_BIT(1) | PPC_BIT(2) | PPC_BIT(3); + npu2_scom_write(p->chip_id, p->xscom_base, NPU2_MISC_IRQ_ENABLE2, + NPU2_MISC_DA_LEN_8B, reg); + + mask_nvlink_fir(p); + return 0; +} + +static void npu2_opencapi_probe(struct dt_node *dn) +{ + struct dt_node *link; + char *path; + uint32_t gcid, index, links, scom_base; + uint64_t reg[2]; + uint64_t dev_index; + struct npu2 *n; + int rc; + + path = dt_get_path(dn); + gcid = dt_get_chip_id(dn); + index = dt_prop_get_u32(dn, "ibm,npu-index"); + links = dt_prop_get_u32(dn, "ibm,npu-links"); + + /* Don't try to init when we have an NVLink link */ + dt_for_each_compatible(dn, link, "ibm,npu-link") { + prlog(PR_DEBUG, "OCAPI: NPU%d: NVLink link found, skipping\n", + index); + return; + } + + prlog(PR_INFO, "OCAPI: Chip %d Found OpenCAPI NPU%d (%d links) at %s\n", + gcid, index, links, path); + free(path); + + scom_base = dt_get_address(dn, 0, NULL); + prlog(PR_INFO, "OCAPI: SCOM Base: %08x\n", scom_base); + + setup_global_mmio_bar(gcid, scom_base, reg); + + n = zalloc(sizeof(struct npu2) + links * sizeof(struct npu2_dev)); + n->devices = (struct npu2_dev *)(n + 1); + n->chip_id = gcid; + n->xscom_base = scom_base; + n->regs = (void *)reg[0]; + n->dt_node = dn; + + dt_for_each_compatible(dn, link, "ibm,npu-link-opencapi") { + dev_index = dt_prop_get_u32(link, "ibm,npu-link-index"); + prlog(PR_INFO, "OCAPI: Configuring link index %lld\n", + dev_index); + + /* Procedure 13.1.3.1 - Select OCAPI vs NVLink */ + brick_config(gcid, scom_base, dev_index); + + /* Procedure 13.1.3.5 - Transaction Layer Configuration */ + tl_config(gcid, scom_base, dev_index); + + /* Procedure 13.1.3.6 - Address Translation Configuration */ + address_translation_config(gcid, scom_base, dev_index); + } + + /* Procedure 13.1.3.10 - Interrupt Configuration */ + rc = setup_irq(n); + if (rc) + goto failed; + + return; +failed: + free(n); +} + +void probe_npu2_opencapi(void) +{ + struct dt_node *np_npu; + + dt_for_each_compatible(dt_root, np_npu, "ibm,power9-npu") + npu2_opencapi_probe(np_npu); +} diff --git a/hw/npu2.c b/hw/npu2.c index b6eff9cb40c8..2e5e3efc73a9 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1306,6 +1306,15 @@ static void npu2_probe_phb(struct dt_node *dn) uint64_t reg[2], mm_win[2]; char *path; + /* Abort if any OpenCAPI links detected */ + if (dt_find_compatible_node(dn, NULL, "ibm,npu-link-opencapi")) { + /* Die if there's also an NVLink link */ + assert(!dt_find_compatible_node(dn, NULL, "ibm,npu-link")); + prlog(PR_INFO, "NPU2: OpenCAPI link configuration detected, " + "not initialising NVLink\n"); + return; + } + /* Retrieve chip id */ path = dt_get_path(dn); gcid = dt_get_chip_id(dn); diff --git a/include/npu2-regs.h b/include/npu2-regs.h index b6dae7bb13fd..9732ec9654e6 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -90,6 +90,11 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_BLOCK_DAT 0x5 #define NPU2_BLOCK_NTL0 0x9 #define NPU2_BLOCK_NTL1 0xb +#define NPU2_BLOCK_OTL0 0xc +#define NPU2_BLOCK_OTL1 0xd + +/* OpenCAPI blocks */ +#define NPU2_BLOCK_XSL 0xe /* MISC stack blocks */ #define NPU2_BLOCK_ATS 0 @@ -108,6 +113,8 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* SM block registers */ #define NPU2_CQ_SM_MISC_CFG0 0x000 +#define NPU2_CQ_SM_MISC_CFG0_CONFIG_ENABLE_PBUS PPC_BIT(38) +#define NPU2_CQ_SM_MISC_CFG0_CONFIG_OCAPI_MODE PPC_BIT(57) #define NPU2_CQ_SM_MISC_CFG1 0x008 #define NPU2_PB_EPSILON 0x010 #define NPU2_TIMER_CFG 0x018 @@ -138,6 +145,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_GENID_BAR_ADDR PPC_BITMASK(3,35) #define NPU2_GENID_BAR_POISON PPC_BIT(39) #define NPU2_LOW_WATER_MARKS 0x040 +#define NPU2_LOW_WATER_MARKS_ENABLE_MACHINE_ALLOC PPC_BIT(51) #define NPU2_HIGH_WATER_MARKS 0x048 #define NPU2_RELAXED_ORDERING_CFG0 0x050 #define NPU2_RELAXED_ORDERING_CFG1 0x058 @@ -176,6 +184,9 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, /* CTL block registers */ #define NPU2_CQ_CTL_MISC_CFG 0x000 +#define NPU2_CQ_CTL_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(52) +#define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL0_ENABLE PPC_BIT(55) +#define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL1_ENABLE PPC_BIT(56) #define NPU2_CQ_FUTURE_CFG1 0x008 #define NPU2_CQ_FUTURE_CFG2 0x010 #define NPU2_CQ_FUTURE_CFG3 0x018 @@ -198,6 +209,8 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_LPC_THRESHOLD_CFG 0x080 #define NPU2_CQ_INHIBIT_CFG 0x088 #define NPU2_CQ_CTL_STATUS 0x090 +#define NPU2_CQ_CTL_STATUS_BRK0_AM_FENCED PPC_BITMASK(48, 49) +#define NPU2_CQ_CTL_STATUS_BRK1_AM_FENCED PPC_BITMASK(50, 51) #define NPU2_CQ_C_ERR_RPT_MSG0 0x0C0 #define NPU2_CQ_C_ERR_RPT_MSG1 0x0C8 #define NPU2_CQ_C_ERR_RPT_FIRST0 0x0D0 @@ -206,9 +219,13 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_C_ERR_RPT_MASK1 0x0E8 #define NPU2_CQ_C_ERR_RPT_HOLD0 0x0F0 #define NPU2_CQ_C_ERR_RPT_HOLD1 0x0F8 +#define NPU2_CQ_CTL_FENCE_CONTROL_0 0x140 +#define NPU2_CQ_CTL_FENCE_CONTROL_1 0x148 +#define NPU2_CQ_CTL_FENCE_CONTROL_REQUEST_FENCE PPC_BITMASK(0, 1) /* DAT block registers */ #define NPU2_CQ_DAT_MISC_CFG 0x008 +#define NPU2_CQ_DAT_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(40) #define NPU2_CQ_DAT_ECC_CFG 0x010 #define NPU2_CQ_DAT_SCRATCH0 0x018 #define NPU2_CQ_DAT_ECC_STATUS 0x020 @@ -289,6 +306,43 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_NTL_DL_CONFIG(ndev) NPU2_DL_REG_OFFSET(ndev, 0xFFF8) #define NPU2_NTL_DL_CLK_CTRL(ndev) NPU2_DL_REG_OFFSET(ndev, 0x001C) +/* OpenCAPI - XSL registers */ +#define NPU2_XSL_PSL_SCNTL_A0 0x010 +#define NPU2_XSL_PSL_SCNTL_A0_MULTI_AFU_DIAL PPC_BIT(0) +#define NPU2_XSL_DEF 0x040 +#define NPU2_XSL_GP 0x058 +#define NPU2_XSL_GP_BLOOM_FILTER_ENABLE PPC_BIT(16) +#define NPU2_XSL_WRAP_CFG 0x0C0 +#define NPU2_XSL_WRAP_CFG_XSLO_CLOCK_ENABLE PPC_BIT(0) + +/* OpenCAPI - OTL registers */ +#define NPU2_OTL_CONFIG0(stack, block) NPU2_REG_OFFSET(stack, block, 0x000) +#define NPU2_OTL_CONFIG0_EN PPC_BIT(0) +#define NPU2_OTL_CONFIG0_BLOCK_PE_HANDLE PPC_BIT(1) +#define NPU2_OTL_CONFIG0_BRICKID PPC_BITMASK(2, 3) +#define NPU2_OTL_CONFIG0_PE_MASK PPC_BITMASK(4, 7) +#define NPU2_OTL_CONFIG0_ERAT_HASH_0 PPC_BITMASK(8, 13) +#define NPU2_OTL_CONFIG0_ERAT_HASH_1 PPC_BITMASK(14, 19) +#define NPU2_OTL_CONFIG0_ERAT_HASH_2 PPC_BITMASK(20, 25) +#define NPU2_OTL_CONFIG0_ERAT_HASH_3 PPC_BITMASK(26, 31) +#define NPU2_OTL_CONFIG0_BLOCK_TID_OVERRIDE PPC_BIT(50) +#define NPU2_OTL_TLX_CREDITS(stack, block) NPU2_REG_OFFSET(stack, block, 0x050) +#define NPU2_OTL_TLX_CREDITS_VC0_CREDITS PPC_BITMASK(0, 7) +#define NPU2_OTL_TLX_CREDITS_VC3_CREDITS PPC_BITMASK(24, 31) +#define NPU2_OTL_TLX_CREDITS_DCP0_CREDITS PPC_BITMASK(32, 39) +#define NPU2_OTL_TLX_CREDITS_DCP1_CREDITS PPC_BITMASK(56, 63) +#define NPU2_OTL_CONFIG1(stack, block) NPU2_REG_OFFSET(stack, block, 0x058) +#define NPU2_OTL_CONFIG1_TX_DRDY_WAIT PPC_BITMASK(5, 7) +#define NPU2_OTL_CONFIG1_TX_TEMP0_RATE PPC_BITMASK(8, 11) +#define NPU2_OTL_CONFIG1_TX_CRET_FREQ PPC_BITMASK(32, 34) +#define NPU2_OTL_CONFIG1_TX_AGE_FREQ PPC_BITMASK(35, 39) +#define NPU2_OTL_CONFIG1_TX_RS2_HPWAIT PPC_BITMASK(40, 45) +#define NPU2_OTL_CONFIG1_TX_RQ4_HPWAIT PPC_BITMASK(46, 51) +#define NPU2_OTL_CONFIG1_TX_RQ6_HPWAIT PPC_BITMASK(52, 57) +#define NPU2_OTL_CONFIG1_TX_CBUF_ECC_DIS PPC_BIT(58) +#define NPU2_OTL_CONFIG1_TX_STOP_LINK PPC_BIT(59) +#define NPU2_OTL_CONFIG1_TX_STOP_ON_UE PPC_BIT(60) + /* Misc block registers. Unlike the SM/CTL/DAT/NTL registers above * there is only a single instance of each of these in the NPU so we * define them as absolute offsets. */ @@ -296,6 +350,9 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_MISC_RELAXTED_CFG NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x008) #define NPU2_MISC_LCO_CFG NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x010) #define NPU2_MISC_OPTICAL_IO_CFG0 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x018) +#define NPU2_MISC_OPTICAL_IO_CFG0_NDLMUX_BRK0TO2 PPC_BITMASK(0, 2) +#define NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK0TO1 PPC_BITMASK(3, 4) +#define NPU2_MISC_OPTICAL_IO_CFG0_OCMUX_BRK4TO5 PPC_BITMASK(5, 6) #define NPU2_MISC_ERR_RPT_HOLD NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x020) #define NPU2_MISC_ERR_RPT_MASK NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x028) #define NPU2_MISC_CFG NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x030) @@ -320,6 +377,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_MISC_ERR_CTL_CFG NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x088) #define NPU2_MISC_RING_ACK_ERR NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x090) #define NPU2_MISC_IRQ_BASE NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x098) +#define NPU2_MISC_IRQ_BASE_MASK PPC_BITMASK(13, 51) #define NPU2_MISC_ERR_BRICK_GROUP NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x0A0) #define NPU2_MISC_FREEZE_STATE NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x0A8) #define NPU2_MISC_FENCE_STATE NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x0B0) @@ -362,6 +420,10 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_MISC_IRQ_LOG13 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x368) #define NPU2_MISC_IRQ_LOG14 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x370) #define NPU2_MISC_IRQ_LOG15 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x378) +#define NPU2_MISC_IRQ_ENABLE2 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, 0x408) + +/* Misc register, direct access only */ +#define NPU2_MISC_FIR_MASK1 0x2C43 /* ATS block registers */ #define NPU2_ATS_PMU_CTL NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_ATS, 0x000) @@ -406,8 +468,11 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_XTS_CFG NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x020) #define NPU2_XTS_CFG_MMIOSD PPC_BIT(1) #define NPU2_XTS_CFG_TRY_ATR_RO PPC_BIT(6) +#define NPU2_XTS_CFG_OPENCAPI PPC_BIT(15) #define NPU2_XTS_CFG2 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x028) #define NPU2_XTS_CFG2_NO_FLUSH_ENA PPC_BIT(49) +#define NPU2_XTS_CFG2_XSL1_ENA PPC_BIT(54) +#define NPU2_XTS_CFG2_XSL2_ENA PPC_BIT(55) #define NPU2_XTS_DBG_CFG0 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x030) #define NPU2_XTS_DBG_CFG1 NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x038) #define NPU2_XTS_PMU_CNT NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x040) @@ -463,4 +528,29 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_DD1_MISC_SCOM_IND_SCOM_DATA 0x38f #define NPU2_MISC_SCOM_IND_SCOM_DATA 0x68f + +/* OpenCAPI - PowerBus and OBus configuration SCOM addresses */ +#define PU_IOE_PB_MISC_CFG 0x5013823 +#define PU_IOE_PB_MISC_CFG_SEL_03_NPU_NOT_PB PPC_BIT(13) +#define PU_IOE_PB_MISC_CFG_SEL_04_NPU_NOT_PB PPC_BIT(14) +#define PU_IOE_PB_MISC_CFG_SEL_05_NPU_NOT_PB PPC_BIT(15) +#define OBUS_LL0_IOOL_PHY_CONFIG 0x901080C +#define OBUS_LL3_IOOL_PHY_CONFIG 0xC01080C +#define OBUS_IOOL_PHY_CONFIG_ODL0_ENABLED PPC_BIT(52) +#define OBUS_IOOL_PHY_CONFIG_ODL1_ENABLED PPC_BIT(53) +#define OBUS_IOOL_PHY_CONFIG_ODL_PHY_SWAP PPC_BIT(54) +#define OBUS_IOOL_PHY_CONFIG_LINK0_OLL_ENABLED PPC_BIT(58) +#define OBUS_IOOL_PHY_CONFIG_LINK1_OLL_ENABLED PPC_BIT(59) +#define OBUS_IOOL_PHY_CONFIG_NV0_NPU_ENABLED PPC_BIT(61) +#define OBUS_IOOL_PHY_CONFIG_NV1_NPU_ENABLED PPC_BIT(62) +#define OBUS_IOOL_PHY_CONFIG_NV2_NPU_ENABLED PPC_BIT(63) +#define PU_IOE_PB_FP01_CFG 0x501380A +#define PU_IOE_PB_FP23_CFG 0x501380B +#define PU_IOE_PB_FP45_CFG 0x501380C +#define PU_IOE_PB_FP67_CFG 0x501380D +#define PU_IOE_PB_FP_CFG_FP0_FMR_DISABLE PPC_BIT(20) +#define PU_IOE_PB_FP_CFG_FP0_PRS_DISABLE PPC_BIT(25) +#define PU_IOE_PB_FP_CFG_FP1_FMR_DISABLE PPC_BIT(52) +#define PU_IOE_PB_FP_CFG_FP1_PRS_DISABLE PPC_BIT(57) + #endif /* __NPU2_REGS_H */ diff --git a/include/npu2.h b/include/npu2.h index 94e537b4d6ec..7b2b5090ff71 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -138,6 +138,7 @@ struct npu2_dev { struct npu2 { uint32_t index; + struct dt_node *dt_node; uint32_t flags; uint32_t chip_id; uint64_t xscom_base; @@ -146,6 +147,7 @@ struct npu2 { uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; + uint32_t irq_base; uint32_t total_devices; struct npu2_dev *devices; enum phys_map_type gpu_map_type; diff --git a/include/skiboot.h b/include/skiboot.h index e94f2121284b..1218367ec578 100644 --- a/include/skiboot.h +++ b/include/skiboot.h @@ -232,6 +232,7 @@ extern int preload_capp_ucode(void); extern void preload_io_vpd(void); extern void probe_npu(void); extern void probe_npu2(void); +extern void probe_npu2_opencapi(void); extern void uart_init(void); extern void mbox_init(void); extern void early_uart_init(void); From patchwork Wed Feb 7 06:50:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870254 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsVd569Qz9ryk for ; Wed, 7 Feb 2018 17:53:45 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsVd435KzDqZs for ; Wed, 7 Feb 2018 17:53:45 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRp4SwTzF0TH for ; Wed, 7 Feb 2018 17:51:18 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176n2eH044022 for ; Wed, 7 Feb 2018 01:51:16 -0500 Received: from e06smtp12.uk.ibm.com (e06smtp12.uk.ibm.com [195.75.94.108]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyvcv8btm-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:16 -0500 Received: from localhost by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:14 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp12.uk.ibm.com (192.168.101.142) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:10 -0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAkM37617716; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 38B404204D; Wed, 7 Feb 2018 06:44:07 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9400042041; Wed, 7 Feb 2018 06:44:06 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:06 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 87CFCA03A2; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:52 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0008-0000-0000-000004CA36E6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0009-0000-0000-00001E5DE5C0 Message-Id: <3f1f5e29c1b2625e19d912d189b4df8566752bbc.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 06/10] npu2-hw-procedures: Add support for OpenCAPI PHY link training X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Unlike NVLink, which uses the pci-virt framework to fake a PCI configuration space for NVLink devices, the OpenCAPI device model presents us with a real configuration space handled by the device over the OpenCAPI link. As a result, we have to train the OpenCAPI link in skiboot before we do PCI probing, so that config space can be accessed, rather than having link training being triggered by the Linux driver. Add some helper functions to wrap the existing NVLink PHY training sequence so we can easily run it within skiboot. Additionally, we add OpenCAPI-specific lane settings, and a function to "bump" lanes that haven't trained properly (this process isn't documented in the workbook, but the hardware experts assure us that this improves link training reliability...) Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Acked-by: Reza Arbab --- hw/npu2-hw-procedures.c | 77 ++++++++++++++++++++++++++++++++++++++++++- include/npu2-regs.h | 6 +++- include/npu2.h | 3 ++- 3 files changed, 86 insertions(+) diff --git a/hw/npu2-hw-procedures.c b/hw/npu2-hw-procedures.c index d66632fcbdb9..1dc583b86f51 100644 --- a/hw/npu2-hw-procedures.c +++ b/hw/npu2-hw-procedures.c @@ -60,6 +60,7 @@ struct npu2_phy_reg NPU2_PHY_TX_FIFO_INIT = {0x105, 53, 1}; struct npu2_phy_reg NPU2_PHY_TX_RXCAL = {0x103, 57, 1}; struct npu2_phy_reg NPU2_PHY_RX_INIT_DONE = {0x0ca, 48, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_EDGE_TRACK_CNTL = {0x092, 48, 2}; +struct npu2_phy_reg NPU2_PHY_RX_PR_BUMP_SL_1UI = {0x092, 57, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_FW_OFF = {0x08a, 56, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_FW_INERTIA_AMT = {0x08a, 57, 3}; struct npu2_phy_reg NPU2_PHY_RX_CFG_LTE_MC = {0x000, 60, 4}; @@ -68,6 +69,8 @@ struct npu2_phy_reg NPU2_PHY_RX_B_INTEG_COARSE_GAIN = {0x026, 48, 4}; struct npu2_phy_reg NPU2_PHY_RX_E_INTEG_COARSE_GAIN = {0x030, 48, 4}; /* These registers are per-PHY, not per lane */ +struct npu2_phy_reg NPU2_PHY_RX_SPEED_SELECT = {0x262, 51, 2}; +struct npu2_phy_reg NPU2_PHY_RX_AC_COUPLED = {0x262, 53, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_SWO_EN = {0x3c9, 48, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_REQ = {0x3c1, 49, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_DONE = {0x3c1, 50, 1}; @@ -378,6 +381,11 @@ static uint32_t phy_reset_complete(struct npu2_dev *ndev) { int lane; + if (ndev->type == NPU2_DEV_TYPE_OPENCAPI) { + phy_write(ndev, &NPU2_PHY_RX_AC_COUPLED, 1); + phy_write(ndev, &NPU2_PHY_RX_SPEED_SELECT, 1); + } + FOR_EACH_LANE(ndev, lane) { phy_write_lane(ndev, &NPU2_PHY_RX_LANE_ANA_PDWN, lane, 0); phy_write_lane(ndev, &NPU2_PHY_RX_LANE_DIG_PDWN, lane, 0); @@ -887,3 +895,72 @@ void npu2_dev_procedure_reset(struct npu2_dev *dev) { npu2_clear_link_flag(dev, NPU2_DEV_DL_RESET); } + +static uint32_t run_procedure(struct npu2_dev *dev, uint16_t procedure_number) +{ + struct procedure *proc; + const char *name; + uint32_t result; + + assert(procedure_number <= ARRAY_SIZE(npu_procedures)); + proc = npu_procedures[procedure_number]; + assert(proc); + + name = proc->name; + NPU2DEVINF(dev, "Running procedure %s\n", name); + dev->procedure_status = PROCEDURE_INPROGRESS; + dev->procedure_number = procedure_number; + dev->procedure_step = 0; + dev->procedure_data = 0; + dev->procedure_tb = mftb(); + + result = get_procedure_status(dev); + while (!(result & PROCEDURE_COMPLETE)) { + time_wait_ms(1); + result = get_procedure_status(dev); + } + return result; +} + +void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev) +{ + uint64_t reg; + uint64_t status_xscom; + int lane, bit = 7; + + switch (dev->index) { + case 2: + status_xscom = OB0_ODL0_TRAINING_STATUS; + break; + case 3: + status_xscom = OB0_ODL1_TRAINING_STATUS; + break; + case 4: + status_xscom = OB3_ODL0_TRAINING_STATUS; + break; + case 5: + status_xscom = OB3_ODL1_TRAINING_STATUS; + break; + default: + assert(false); + } + xscom_read(dev->npu->chip_id, status_xscom, ®); + reg = GETFIELD(OB_ODL_TRAINING_STATUS_STS_RX_PATTERN_B, reg); + + FOR_EACH_LANE(dev, lane) { + if (reg & (1 << bit--)) + continue; + prlog(PR_TRACE, "OCAPI: bumpui bumping lane %d\n", lane); + for (int i = 0; i < 4; i++) { + phy_write_lane(dev, &NPU2_PHY_RX_PR_BUMP_SL_1UI, lane, 1); + phy_write_lane(dev, &NPU2_PHY_RX_PR_BUMP_SL_1UI, lane, 0); + } + } +} + +void npu2_opencapi_phy_setup(struct npu2_dev *dev) +{ + run_procedure(dev, 4); /* procedure_phy_reset */ + run_procedure(dev, 5); /* procedure_phy_tx_zcal */ + run_procedure(dev, 6); /* procedure_phy_rx_dccal */ +} diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 9732ec9654e6..b219ad77bdc6 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -553,4 +553,10 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define PU_IOE_PB_FP_CFG_FP1_FMR_DISABLE PPC_BIT(52) #define PU_IOE_PB_FP_CFG_FP1_PRS_DISABLE PPC_BIT(57) +#define OB0_ODL0_TRAINING_STATUS 0x901082E +#define OB0_ODL1_TRAINING_STATUS 0x901082F +#define OB3_ODL0_TRAINING_STATUS 0xC01082E +#define OB3_ODL1_TRAINING_STATUS 0xC01082F +#define OB_ODL_TRAINING_STATUS_STS_RX_PATTERN_B PPC_BITMASK(8, 15) + #endif /* __NPU2_REGS_H */ diff --git a/include/npu2.h b/include/npu2.h index 7b2b5090ff71..2922d2185296 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -200,9 +200,12 @@ int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); void npu2_dev_procedure_reset(struct npu2_dev *dev); + void npu2_set_link_flag(struct npu2_dev *ndev, uint8_t flag); void npu2_clear_link_flag(struct npu2_dev *ndev, uint8_t flag); extern int nv_zcal_nominal; bool is_p9dd1(void); +void npu2_opencapi_phy_setup(struct npu2_dev *dev); +void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev); #endif /* __NPU2_H */ From patchwork Wed Feb 7 06:50:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870251 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsTv4ZDMz9ryk for ; Wed, 7 Feb 2018 17:53:07 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsTv3GjYzF13W for ; Wed, 7 Feb 2018 17:53:07 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRm4qXXzF0Sb for ; Wed, 7 Feb 2018 17:51:16 +1100 (AEDT) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176oDH3124114 for ; Wed, 7 Feb 2018 01:51:15 -0500 Received: from e06smtp14.uk.ibm.com (e06smtp14.uk.ibm.com [195.75.94.110]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyp00wfex-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:14 -0500 Received: from localhost by e06smtp14.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:12 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp14.uk.ibm.com (192.168.101.144) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:11 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAe518677872; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 57AA24C044; Wed, 7 Feb 2018 06:45:01 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 747874C04A; Wed, 7 Feb 2018 06:45:00 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:45:00 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 8DD59A03A3; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:53 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0016-0000-0000-00000520334E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0017-0000-0000-0000285CE7C4 Message-Id: <15ab3b16eee16840c6f847b5e2e17b699cd26001.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 07/10] npu2-opencapi: Train OpenCAPI links and setup devices X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Scan the OpenCAPI links under the NPU, and for each link, reset the card, set up a device, train the link and register a PHB. Implement the necessary operations for the OpenCAPI PHB type. Signed-off-by: Andrew Donnellan Signed-off-by: Frederic Barrat Reviewed-by: Frederic Barrat --- v1->v2: - remove all-devices case for device reset (Fred) - remove i2c 1.8v hack (Fred) - find the right I2C port ID for resets only once per device rather than once per reset (Fred) v2->v3: - add a stub ioda_reset callback so we don't see OPAL_UNSUPPORTED in Linux - Remove unused NPU IRQ register macro (Fred) - Fix up PHB device tree properties so we don't get warnings from DTC - use NPU phandle rather than link phandle for ibm,npcq property for consistency with NPU2 - use correct "ibm,config-space-type" property in device tree v3->v4: - Use a separate npu2_opencapi_set_pe() function rather than bolting it on to npu2_set_pe() (Alistair) --- core/pci.c | 3 +- hw/npu2-opencapi.c | 636 ++++++++++++++++++++++++++++++++++++++++++++- hw/npu2.c | 12 +- include/npu2-regs.h | 52 +++- include/npu2.h | 7 +- 5 files changed, 700 insertions(+), 10 deletions(-) diff --git a/core/pci.c b/core/pci.c index 0809521f8baf..c7122639e9ff 100644 --- a/core/pci.c +++ b/core/pci.c @@ -1549,7 +1549,8 @@ static void pci_add_one_device_node(struct phb *phb, pd->dn = np = dt_new(parent_node, name); /* XXX FIXME: make proper "compatible" properties */ - if (pci_has_cap(pd, PCI_CFG_CAP_ID_EXP, false)) { + if (pci_has_cap(pd, PCI_CFG_CAP_ID_EXP, false) || + phb->phb_type == phb_type_npu_v2_opencapi) { snprintf(compat, MAX_NAME, "pciex%x,%x", vdid & 0xffff, vdid >> 16); dt_add_property_cells(np, "ibm,pci-config-space-type", 1); diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 6ff3d3317ee0..c4e263c6d57f 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -53,6 +53,9 @@ #include #define NPU_IRQ_LEVELS 35 +#define NPU_IRQ_LEVELS_XSL 23 + +static const struct phb_ops npu2_opencapi_ops; static inline uint64_t index_to_stack(uint64_t index) { switch (index) { @@ -99,6 +102,28 @@ static inline uint64_t index_to_block(uint64_t index) { } } +static uint64_t get_odl_status(uint32_t gcid, uint64_t index) { + uint64_t reg, status_xscom; + switch (index) { + case 2: + status_xscom = OB0_ODL0_STATUS; + break; + case 3: + status_xscom = OB0_ODL1_STATUS; + break; + case 4: + status_xscom = OB3_ODL0_STATUS; + break; + case 5: + status_xscom = OB3_ODL1_STATUS; + break; + default: + assert(false); + } + xscom_read(gcid, status_xscom, ®); + return reg; +} + /* Procedure 13.1.3.1 - select OCAPI vs NVLink for bricks 2-3/4-5 */ static void set_transport_mux_controls(uint32_t gcid, uint32_t scom_base, @@ -673,6 +698,441 @@ static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, reg[1] = size; } +/* Procedure 13.1.3.8 - AFU MMIO Range BARs */ +static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, + struct npu2_dev *dev) +{ + uint64_t stack = index_to_stack(dev->index); + uint64_t offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + NPU2_NTL0_BAR : NPU2_NTL1_BAR; + uint64_t pa_offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : + NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; + uint64_t addr, size, reg; + + prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); + phys_map_get(gcid, NPU_OCAPI_MMIO, dev->index, &addr, &size); + + prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); + write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, + size); + dev->bars[0].npu2_bar.base = addr; + dev->bars[0].npu2_bar.size = size; + + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_ADDR, 0ull, addr >> 16); + reg = SETFIELD(NPU2_CQ_CTL_MISC_MMIOPA_SIZE, reg, ilog2(size >> 16)); + prlog(PR_DEBUG, "OCAPI: PA translation %llx\n", reg); + npu2_scom_write(gcid, scom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_CTL, + pa_offset), + NPU2_MISC_DA_LEN_8B, reg); +} + +/* Procedure 13.1.3.9 - AFU Config BARs */ +static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, + struct npu2_dev *dev) +{ + uint64_t stack = index_to_stack(dev->index); + int stack_num = stack - NPU2_STACK_STCK_0; + uint64_t addr, size; + + prlog(PR_DEBUG, "OCAPI: %s: Setup AFU Config BARs\n", __func__); + phys_map_get(gcid, NPU_GENID, stack_num, &addr, &size); + prlog(PR_DEBUG, "OCAPI: Assigning GENID BAR: %016llx\n", addr); + write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, NPU2_GENID_BAR), + addr, size); + dev->bars[1].npu2_bar.base = addr; + dev->bars[1].npu2_bar.size = size; +} + +static void otl_enabletx(uint32_t gcid, uint32_t scom_base, uint64_t index) +{ + uint64_t stack = index_to_stack(index); + uint64_t block = index_to_block(index); + uint64_t reg; + + /* OTL Config 2 Register */ + /* Transmit Enable */ + prlog(PR_DEBUG, "OCAPI: %s: Enabling TX\n", __func__); + reg = 0; + reg |= NPU2_OTL_CONFIG2_TX_SEND_EN; + npu2_scom_write(gcid, scom_base, NPU2_OTL_CONFIG2(stack, block), + NPU2_MISC_DA_LEN_8B, reg); + + reg = npu2_scom_read(gcid, scom_base, NPU2_OTL_VC_CREDITS(stack, block), + NPU2_MISC_DA_LEN_8B); + prlog(PR_DEBUG, "OCAPI: credit counter: %llx\n", reg); + /* TODO: Abort if credits are zero */ +} + +static void reset_ocapi_device(struct npu2_dev *dev) +{ + uint8_t data[3]; + int rc; + int i; + + switch (dev->index) { + case 2: + case 4: + memcpy(data, platform.ocapi->i2c_odl0_data, sizeof(data)); + break; + case 3: + case 5: + memcpy(data, platform.ocapi->i2c_odl1_data, sizeof(data)); + break; + default: + assert(false); + } + + for (i = 0; i < 3; i++) { + rc = i2c_request_send(dev->i2c_port_id_ocapi, 0x20, SMBUS_WRITE, + platform.ocapi->i2c_offset[i], 1, + &data[i], sizeof(data[i]), 120); + if (rc) { + /** + * @fwts-label OCAPIDeviceResetFailed + * @fwts-advice There was an error attempting to send + * a reset signal over I2C to the OpenCAPI device. + */ + prlog(PR_ERR, "OCAPI: Error writing I2C reset signal: %d\n", rc); + break; + } + if (i != 0) + time_wait_ms(5); + } +} + +static int odl_train(uint32_t gcid, uint32_t index, struct npu2_dev *dev) +{ + uint64_t reg, config_xscom; + int timeout = 3000; + prlog(PR_DEBUG, "OCAPI: %s: Training ODL\n", __func__); + + switch (index) { + case 2: + config_xscom = OB0_ODL0_CONFIG; + break; + case 3: + config_xscom = OB0_ODL1_CONFIG; + break; + case 4: + config_xscom = OB3_ODL0_CONFIG; + break; + case 5: + config_xscom = OB3_ODL1_CONFIG; + break; + default: + assert(false); + } + + /* Reset ODL */ + reg = OB_ODL_CONFIG_RESET; + reg = SETFIELD(OB_ODL_CONFIG_VERSION, reg, 0b000001); + reg = SETFIELD(OB_ODL_CONFIG_TRAIN_MODE, reg, 0b0110); + reg = SETFIELD(OB_ODL_CONFIG_SUPPORTED_MODES, reg, 0b0010); + reg |= OB_ODL_CONFIG_X4_BACKOFF_ENABLE; + reg = SETFIELD(OB_ODL_CONFIG_PHY_CNTR_LIMIT, reg, 0b1111); + reg |= OB_ODL_CONFIG_DEBUG_ENABLE; + reg = SETFIELD(OB_ODL_CONFIG_FWD_PROGRESS_TIMER, reg, 0b0110); + xscom_write(gcid, config_xscom, reg); + + reg &= ~OB_ODL_CONFIG_RESET; + xscom_write(gcid, config_xscom, reg); + + reset_ocapi_device(dev); + + /* Transmit Pattern A */ + reg = SETFIELD(OB_ODL_CONFIG_TRAIN_MODE, reg, 0b0001); + xscom_write(gcid, config_xscom, reg); + time_wait_ms(5); + + /* Bump lanes - this improves training reliability */ + npu2_opencapi_bump_ui_lane(dev); + + /* Start training */ + reg = SETFIELD(OB_ODL_CONFIG_TRAIN_MODE, reg, 0b1000); + xscom_write(gcid, config_xscom, reg); + + do { + reg = get_odl_status(gcid, index); + if (GETFIELD(OB_ODL_STATUS_TRAINING_STATE_MACHINE, reg) == 0x7) { + prlog(PR_NOTICE, + "OCAPI: Link %d on chip %u trained in %dms\n", + index, gcid, 3000 - timeout); + return OPAL_SUCCESS; + } + time_wait_ms(1); + } while (timeout--); + prlog(PR_INFO, "OCAPI: Link %d on chip %u failed to train, retrying\n", + index, gcid); + prlog(PR_INFO, "OCAPI: Link status: %016llx\n", reg); + return OPAL_HARDWARE; +} + +static int64_t npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) +{ + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); + uint64_t reg; + int64_t link_width, rc = OPAL_SUCCESS; + + reg = get_odl_status(dev->npu->chip_id, dev->index); + link_width = GETFIELD(OB_ODL_STATUS_TRAINED_MODE, reg); + switch (link_width) { + case 0b0001: + *val = OPAL_SHPC_LINK_UP_x4; + break; + case 0b0010: + *val = OPAL_SHPC_LINK_UP_x8; + break; + default: + rc = OPAL_HARDWARE; + } + return rc; +} + +static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) +{ + struct pci_slot *slot; + + slot = pci_slot_alloc(phb, NULL); + if (!slot) + return slot; + + /* TODO: Figure out other slot functions */ + slot->ops.get_presence_state = NULL; + slot->ops.get_link_state = npu2_opencapi_get_link_state; + slot->ops.get_power_state = NULL; + slot->ops.get_attention_state = NULL; + slot->ops.get_latch_state = NULL; + slot->ops.set_power_state = NULL; + slot->ops.set_attention_state = NULL; + + return slot; +} + +static int64_t npu2_opencapi_pcicfg_check(struct npu2_dev *dev, uint32_t offset, + uint32_t size) +{ + if (!dev || offset > 0xfff || (offset & (size - 1))) + return OPAL_PARAMETER; + + return OPAL_SUCCESS; +} + +static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, + uint32_t offset, uint32_t size, + void *data) +{ + uint64_t cfg_addr; + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(phb); + uint64_t genid_base; + int64_t rc; + + rc = npu2_opencapi_pcicfg_check(dev, offset, size); + if (rc) + return rc; + + genid_base = dev->bars[1].npu2_bar.base + + (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + + cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; + cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | + NPU2_CQ_CTL_CONFIG_ADDR_DEVICE_NUMBER | + NPU2_CQ_CTL_CONFIG_ADDR_FUNCTION_NUMBER, + cfg_addr, bdfn); + cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_REGISTER_NUMBER, + cfg_addr, offset & ~3u); + + out_be64((uint64_t *)genid_base, cfg_addr); + sync(); + + switch (size) { + case 1: + *((uint8_t *)data) = + in_8((volatile uint8_t *)(genid_base + 128 + (offset & 3))); + break; + case 2: + *((uint16_t *)data) = + in_le16((volatile uint16_t *)(genid_base + 128 + (offset & 2))); + break; + case 4: + *((uint32_t *)data) = in_le32((volatile uint32_t *)(genid_base + 128)); + break; + default: + return OPAL_PARAMETER; + } + + return OPAL_SUCCESS; +} + +#define NPU2_OPENCAPI_PCI_CFG_READ(size, type) \ +static int64_t npu2_opencapi_pcicfg_read##size(struct phb *phb, \ + uint32_t bdfn, \ + uint32_t offset, \ + type *data) \ +{ \ + /* Initialize data in case of error */ \ + *data = (type)0xffffffff; \ + return npu2_opencapi_pcicfg_read(phb, bdfn, offset, \ + sizeof(type), data); \ +} + +static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, + uint32_t offset, uint32_t size, + uint32_t data) +{ + uint64_t cfg_addr; + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(phb); + uint64_t genid_base; + int64_t rc; + + rc = npu2_opencapi_pcicfg_check(dev, offset, size); + if (rc) + return rc; + + genid_base = dev->bars[1].npu2_bar.base + + (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + + cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; + cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | + NPU2_CQ_CTL_CONFIG_ADDR_DEVICE_NUMBER | + NPU2_CQ_CTL_CONFIG_ADDR_FUNCTION_NUMBER, + cfg_addr, bdfn); + cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_REGISTER_NUMBER, + cfg_addr, offset & ~3u); + + out_be64((uint64_t *)genid_base, cfg_addr); + sync(); + + switch (size) { + case 1: + out_8((volatile uint8_t *)(genid_base + 128 + (offset & 3)), + data); + break; + case 2: + out_le16((volatile uint16_t *)(genid_base + 128 + (offset & 2)), + data); + break; + case 4: + out_le32((volatile uint32_t *)(genid_base + 128), data); + break; + default: + return OPAL_PARAMETER; + } + + return OPAL_SUCCESS; +} + +#define NPU2_OPENCAPI_PCI_CFG_WRITE(size, type) \ +static int64_t npu2_opencapi_pcicfg_write##size(struct phb *phb, \ + uint32_t bdfn, \ + uint32_t offset, \ + type data) \ +{ \ + return npu2_opencapi_pcicfg_write(phb, bdfn, offset, \ + sizeof(type), data); \ +} + +NPU2_OPENCAPI_PCI_CFG_READ(8, u8) +NPU2_OPENCAPI_PCI_CFG_READ(16, u16) +NPU2_OPENCAPI_PCI_CFG_READ(32, u32) +NPU2_OPENCAPI_PCI_CFG_WRITE(8, u8) +NPU2_OPENCAPI_PCI_CFG_WRITE(16, u16) +NPU2_OPENCAPI_PCI_CFG_WRITE(32, u32) + +static int64_t npu2_opencapi_ioda_reset(struct phb __unused *phb, + bool __unused purge) +{ + /* Not relevant to OpenCAPI - we do this just to silence the error */ + return OPAL_SUCCESS; +} + +static int64_t npu2_opencapi_set_pe(struct phb *phb, + uint64_t pe_num, + uint64_t bdfn, + uint8_t bcompare, + uint8_t dcompare, + uint8_t fcompare, + uint8_t action) +{ + struct npu2 *p; + struct npu2_dev *dev; + uint64_t reg, val, pe_bdfn; + + /* Sanity check */ + if (action != OPAL_MAP_PE && action != OPAL_UNMAP_PE) + return OPAL_PARAMETER; + if (pe_num >= NPU2_MAX_PE_NUM) + return OPAL_PARAMETER; + if (bdfn >> 8) + return OPAL_PARAMETER; + if (bcompare != OpalPciBusAll || + dcompare != OPAL_COMPARE_RID_DEVICE_NUMBER || + fcompare != OPAL_COMPARE_RID_FUNCTION_NUMBER) + return OPAL_UNSUPPORTED; + + /* Get the NPU2 device */ + dev = phb_to_npu2_dev_ocapi(phb); + if (!dev) + return OPAL_PARAMETER; + + p = dev->npu; + + pe_bdfn = dev->bdfn; + + val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); + val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, pe_bdfn); + reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, + NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); + p->bdf2pe_cache[dev->index] = val; + npu2_write(p, reg, val); + + return OPAL_SUCCESS; +} + +static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, + void *data __unused) +{ + uint32_t irq; + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(phb); + uint64_t block = index_to_block(dev->index); + uint64_t stacku = index_to_stacku(dev->index); + uint64_t dsisr, dar, tfc, handle; + + /* + * Pass the hw irq number for the translation fault irq + * irq levels 23 -> 26 are for translation faults, 1 per brick + */ + irq = dev->npu->irq_base + NPU_IRQ_LEVELS_XSL; + if (stacku == NPU2_STACK_STCK_2U) + irq += 2; + if (block == NPU2_BLOCK_OTL1) + irq++; + + /* + * Add the addresses of the registers needed by the OS to handle + * faults. The OS accesses them by mmio. + */ + dsisr = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_DSISR(stacku, block); + dar = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_DAR(stacku, block); + tfc = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_TFC(stacku, block); + handle = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_PEHANDLE(stacku, + block); + dt_add_property_cells(pd->dn, "ibm,opal-xsl-irq", irq); + dt_add_property_cells(pd->dn, "ibm,opal-xsl-mmio", + hi32(dsisr), lo32(dsisr), + hi32(dar), lo32(dar), + hi32(tfc), lo32(tfc), + hi32(handle), lo32(handle)); + return 0; +} + +static void npu2_opencapi_final_fixup(struct phb *phb) +{ + pci_walk_dev(phb, NULL, npu2_add_mmio_regs, NULL); +} + static void mask_nvlink_fir(struct npu2 *p) { uint64_t reg; @@ -766,6 +1226,137 @@ static int setup_irq(struct npu2 *p) return 0; } +static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, + struct npu2_dev *dev) +{ + uint32_t dev_index, npu_index; + struct dt_node *dn_phb, *dn; + struct pci_slot *slot; + char port_name[17]; + uint64_t mm_win[2]; + int retries = 20; + int rc; + + dev_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + npu_index = dt_prop_get_u32(n->dt_node, "ibm,npu-index"); + + /* Populate PHB device node */ + phys_map_get(n->chip_id, NPU_OCAPI_MMIO, dev_index, &mm_win[0], + &mm_win[1]); + prlog(PR_DEBUG, "OCAPI: Setting MMIO window to %016llx + %016llx\n", + mm_win[0], mm_win[1]); + dn_phb = dt_new_addr(dt_root, "pciex", mm_win[0]); + assert(dn_phb); + dt_add_property_strings(dn_phb, + "compatible", + "ibm,power9-npu-opencapi-pciex", + "ibm,ioda2-npu2-opencapi-phb"); + + dt_add_property_cells(dn_phb, "#address-cells", 3); + dt_add_property_cells(dn_phb, "#size-cells", 2); + dt_add_property_cells(dn_phb, "#interrupt-cells", 1); + dt_add_property_cells(dn_phb, "bus-range", 0, 0xff); + dt_add_property_cells(dn_phb, "clock-frequency", 0x200, 0); + dt_add_property_cells(dn_phb, "interrupt-parent", get_ics_phandle()); + + dt_add_property_strings(dn_phb, "device_type", "pciex"); + dt_add_property(dn_phb, "reg", mm_win, sizeof(mm_win)); + dt_add_property_cells(dn_phb, "ibm,npu-index", npu_index); + dt_add_property_cells(dn_phb, "ibm,chip-id", n->chip_id); + dt_add_property_cells(dn_phb, "ibm,xscom-base", n->xscom_base); + dt_add_property_cells(dn_phb, "ibm,npcq", n->dt_node->phandle); + dt_add_property_cells(dn_phb, "ibm,links", 1); + dt_add_property(dn_phb, "ibm,mmio-window", mm_win, sizeof(mm_win)); + dt_add_property_cells(dn_phb, "ibm,phb-diag-data-size", 0); + dt_add_property_cells(dn_phb, "ibm,opal-num-pes", NPU2_MAX_PE_NUM); + + n->mm_base = mm_win[0]; + n->mm_size = mm_win[1]; + + dt_add_property_cells(dn_phb, "ranges", 0x02000000, + hi32(n->mm_base), lo32(n->mm_base), + hi32(n->mm_base), lo32(n->mm_base), + hi32(n->mm_size), lo32(n->mm_size)); + + dev->type = NPU2_DEV_TYPE_OPENCAPI; + dev->npu = n; + dev->dt_node = dn_link; + dev->phb_ocapi.dt_node = dn_phb; + dev->phb_ocapi.ops = &npu2_opencapi_ops; + dev->phb_ocapi.phb_type = phb_type_npu_v2_opencapi; + dev->phb_ocapi.scan_map = 1; + dev->index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + dev->pl_xscom_base = dt_prop_get_u64(dn_link, "ibm,npu-phy"); + dev->lane_mask = dt_prop_get_u32(dn_link, "ibm,npu-lane-mask"); + dev->bdfn = 0; + n->total_devices++; + + /* Find I2C port for handling device reset */ + snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", + dev->npu->chip_id, platform.ocapi->i2c_engine, + platform.ocapi->i2c_port); + prlog(PR_DEBUG, "OCAPI: Looking for I2C port %s\n", port_name); + + dt_for_each_compatible(dt_root, dn, "ibm,power9-i2c-port") { + if (streq(port_name, dt_prop_get(dn, "ibm,port-name"))) { + dev->i2c_port_id_ocapi = dt_prop_get_u32(dn, "ibm,opal-id"); + break; + } + } + + if (!dev->i2c_port_id_ocapi) { + prlog(PR_ERR, "OCAPI: Couldn't find I2C port %s\n", port_name); + goto failed; + } + + /* TODO: Procedure 13.1.3.7 - AFU Memory Range BARs */ + /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ + setup_afu_mmio_bars(n->chip_id, n->xscom_base, dev); + /* Procedure 13.1.3.9 - AFU Config BARs */ + setup_afu_config_bars(n->chip_id, n->xscom_base, dev); + + set_fence_control(n->chip_id, n->xscom_base, dev->index, 0b00); + + npu2_opencapi_phy_setup(dev); + + do { + rc = odl_train(n->chip_id, dev->index, dev); + } while (rc != OPAL_SUCCESS && --retries); + + if (rc != OPAL_SUCCESS && retries == 0) { + /** + * @fwts-label OCAPILinkTrainingFailed + * @fwts-advice The OpenCAPI link training procedure failed. + * This indicates a hardware or firmware bug. OpenCAPI + * functionality will not be available on this link. + */ + prlog(PR_ERR, "OCAPI: Link %d on chip %u failed to train\n", + dev->index, n->chip_id); + prlog(PR_ERR, "OCAPI: Final link status: %016llx\n", + get_odl_status(n->chip_id, dev->index)); + goto failed; + } + + otl_enabletx(n->chip_id, n->xscom_base, dev->index); + + slot = npu2_opencapi_slot_create(&dev->phb_ocapi); + if (!slot) + { + /** + * @fwts-label OCAPICannotCreatePHBSlot + * @fwts-advice Firmware probably ran out of memory creating + * NPU slot. OpenCAPI functionality could be broken. + */ + prlog(PR_ERR, "OCAPI: Cannot create PHB slot\n"); + } + + pci_register_phb(&dev->phb_ocapi, OPAL_DYNAMIC_PHB_ID); + return; +failed: + dt_add_property_string(dn_phb, "status", "error"); + return; +} + static void npu2_opencapi_probe(struct dt_node *dn) { struct dt_node *link; @@ -774,7 +1365,7 @@ static void npu2_opencapi_probe(struct dt_node *dn) uint64_t reg[2]; uint64_t dev_index; struct npu2 *n; - int rc; + int rc, i = 0; path = dt_get_path(dn); gcid = dt_get_chip_id(dn); @@ -792,6 +1383,8 @@ static void npu2_opencapi_probe(struct dt_node *dn) gcid, index, links, path); free(path); + assert(platform.ocapi); + scom_base = dt_get_address(dn, 0, NULL); prlog(PR_INFO, "OCAPI: SCOM Base: %08x\n", scom_base); @@ -824,6 +1417,11 @@ static void npu2_opencapi_probe(struct dt_node *dn) if (rc) goto failed; + dt_for_each_compatible(dn, link, "ibm,npu-link-opencapi") { + npu2_opencapi_setup_device(link, n, &n->devices[i]); + i++; + } + return; failed: free(n); @@ -836,3 +1434,39 @@ void probe_npu2_opencapi(void) dt_for_each_compatible(dt_root, np_npu, "ibm,power9-npu") npu2_opencapi_probe(np_npu); } + +static const struct phb_ops npu2_opencapi_ops = { + .cfg_read8 = npu2_opencapi_pcicfg_read8, + .cfg_read16 = npu2_opencapi_pcicfg_read16, + .cfg_read32 = npu2_opencapi_pcicfg_read32, + .cfg_write8 = npu2_opencapi_pcicfg_write8, + .cfg_write16 = npu2_opencapi_pcicfg_write16, + .cfg_write32 = npu2_opencapi_pcicfg_write32, + .choose_bus = NULL, + .device_init = NULL, + .phb_final_fixup = npu2_opencapi_final_fixup, + .ioda_reset = npu2_opencapi_ioda_reset, + .papr_errinjct_reset = NULL, + .pci_reinit = NULL, + .set_phb_mem_window = NULL, + .phb_mmio_enable = NULL, + .map_pe_mmio_window = NULL, + .map_pe_dma_window = NULL, + .map_pe_dma_window_real = NULL, + .pci_msi_eoi = NULL, + .set_xive_pe = NULL, + .get_msi_32 = NULL, + .get_msi_64 = NULL, + .set_pe = npu2_opencapi_set_pe, + .set_peltv = NULL, + .eeh_freeze_status = npu2_freeze_status, /* TODO */ + .eeh_freeze_clear = NULL, + .eeh_freeze_set = NULL, + .next_error = NULL, + .err_inject = NULL, + .get_diag_data = NULL, + .get_diag_data2 = NULL, + .set_capi_mode = NULL, + .set_capp_recovery = NULL, + .tce_kill = NULL, +}; diff --git a/hw/npu2.c b/hw/npu2.c index 2e5e3efc73a9..1f7a1e0cebc7 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1110,12 +1110,12 @@ static struct pci_slot *npu2_slot_create(struct phb *phb) return slot; } -static int64_t npu2_freeze_status(struct phb *phb __unused, - uint64_t pe_number __unused, - uint8_t *freeze_state, - uint16_t *pci_error_type __unused, - uint16_t *severity __unused, - uint64_t *phb_status __unused) +int64_t npu2_freeze_status(struct phb *phb __unused, + uint64_t pe_number __unused, + uint8_t *freeze_state, + uint16_t *pci_error_type __unused, + uint16_t *severity __unused, + uint64_t *phb_status __unused) { /* * FIXME: When it's called by skiboot PCI config accessor, diff --git a/include/npu2-regs.h b/include/npu2-regs.h index b219ad77bdc6..71c8d8f07af0 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -116,6 +116,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_SM_MISC_CFG0_CONFIG_ENABLE_PBUS PPC_BIT(38) #define NPU2_CQ_SM_MISC_CFG0_CONFIG_OCAPI_MODE PPC_BIT(57) #define NPU2_CQ_SM_MISC_CFG1 0x008 +#define NPU2_CQ_SM_MISC_CFG2 0x148 #define NPU2_PB_EPSILON 0x010 #define NPU2_TIMER_CFG 0x018 #define NPU2_GPU0_MEM_BAR 0x020 @@ -187,7 +188,11 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OCAPI_MODE PPC_BIT(52) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL0_ENABLE PPC_BIT(55) #define NPU2_CQ_CTL_MISC_CFG_CONFIG_OTL1_ENABLE PPC_BIT(56) -#define NPU2_CQ_FUTURE_CFG1 0x008 +#define NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG 0x0B0 +#define NPU2_CQ_CTL_MISC_MMIOPA_ADDR PPC_BITMASK(1,35) +#define NPU2_CQ_CTL_MISC_MMIOPA_SIZE PPC_BITMASK(39,43) +#define NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG 0x0B8 +#define NPU2_CQ_CTL_MISC_CFG1 0x008 #define NPU2_CQ_FUTURE_CFG2 0x010 #define NPU2_CQ_FUTURE_CFG3 0x018 #define NPU2_CQ_PERF_MATCH 0x020 @@ -219,6 +224,15 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_CQ_C_ERR_RPT_MASK1 0x0E8 #define NPU2_CQ_C_ERR_RPT_HOLD0 0x0F0 #define NPU2_CQ_C_ERR_RPT_HOLD1 0x0F8 +#define NPU2_CQ_CTL_CONFIG_ADDR0 0x120 +#define NPU2_CQ_CTL_CONFIG_ADDR1 0x128 +#define NPU2_CQ_CTL_CONFIG_ADDR_ENABLE PPC_BIT(0) +#define NPU2_CQ_CTL_CONFIG_ADDR_STATUS PPC_BITMASK(1, 3) +#define NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER PPC_BITMASK(4, 11) +#define NPU2_CQ_CTL_CONFIG_ADDR_DEVICE_NUMBER PPC_BITMASK(12, 16) +#define NPU2_CQ_CTL_CONFIG_ADDR_FUNCTION_NUMBER PPC_BITMASK(17, 19) +#define NPU2_CQ_CTL_CONFIG_ADDR_REGISTER_NUMBER PPC_BITMASK(20, 31) +#define NPU2_CQ_CTL_CONFIG_ADDR_TYPE PPC_BIT(32) #define NPU2_CQ_CTL_FENCE_CONTROL_0 0x140 #define NPU2_CQ_CTL_FENCE_CONTROL_1 0x148 #define NPU2_CQ_CTL_FENCE_CONTROL_REQUEST_FENCE PPC_BITMASK(0, 1) @@ -331,9 +345,16 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_OTL_TLX_CREDITS_VC3_CREDITS PPC_BITMASK(24, 31) #define NPU2_OTL_TLX_CREDITS_DCP0_CREDITS PPC_BITMASK(32, 39) #define NPU2_OTL_TLX_CREDITS_DCP1_CREDITS PPC_BITMASK(56, 63) +#define NPU2_OTL_VC_CREDITS(stack, block) NPU2_REG_OFFSET(stack, block, 0x090) #define NPU2_OTL_CONFIG1(stack, block) NPU2_REG_OFFSET(stack, block, 0x058) +#define NPU2_OTL_CONFIG1_TX_TEMP1_EN PPC_BIT(1) +#define NPU2_OTL_CONFIG1_TX_TEMP2_EN PPC_BIT(2) +#define NPU2_OTL_CONFIG1_TX_TEMP3_EN PPC_BIT(3) #define NPU2_OTL_CONFIG1_TX_DRDY_WAIT PPC_BITMASK(5, 7) #define NPU2_OTL_CONFIG1_TX_TEMP0_RATE PPC_BITMASK(8, 11) +#define NPU2_OTL_CONFIG1_TX_TEMP1_RATE PPC_BITMASK(12, 15) +#define NPU2_OTL_CONFIG1_TX_TEMP2_RATE PPC_BITMASK(16, 19) +#define NPU2_OTL_CONFIG1_TX_TEMP3_RATE PPC_BITMASK(20, 23) #define NPU2_OTL_CONFIG1_TX_CRET_FREQ PPC_BITMASK(32, 34) #define NPU2_OTL_CONFIG1_TX_AGE_FREQ PPC_BITMASK(35, 39) #define NPU2_OTL_CONFIG1_TX_RS2_HPWAIT PPC_BITMASK(40, 45) @@ -342,6 +363,15 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_OTL_CONFIG1_TX_CBUF_ECC_DIS PPC_BIT(58) #define NPU2_OTL_CONFIG1_TX_STOP_LINK PPC_BIT(59) #define NPU2_OTL_CONFIG1_TX_STOP_ON_UE PPC_BIT(60) +#define NPU2_OTL_CONFIG1_TX_T0_MASK_CRTN0 PPC_BIT(61) +#define NPU2_OTL_CONFIG1_TX_T123_MASK_CRTN0 PPC_BIT(62) +#define NPU2_OTL_CONFIG2(stack, block) NPU2_REG_OFFSET(stack, block, 0x0C0) +#define NPU2_OTL_CONFIG2_TX_SEND_EN PPC_BIT(0) + +#define NPU2_OTL_OSL_DSISR(stack, block) NPU2_REG_OFFSET(stack, block, 0x000) +#define NPU2_OTL_OSL_DAR(stack, block) NPU2_REG_OFFSET(stack, block, 0x008) +#define NPU2_OTL_OSL_TFC(stack, block) NPU2_REG_OFFSET(stack, block, 0x010) +#define NPU2_OTL_OSL_PEHANDLE(stack, block) NPU2_REG_OFFSET(stack, block, 0x018) /* Misc block registers. Unlike the SM/CTL/DAT/NTL registers above * there is only a single instance of each of these in the NPU so we @@ -553,6 +583,26 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define PU_IOE_PB_FP_CFG_FP1_FMR_DISABLE PPC_BIT(52) #define PU_IOE_PB_FP_CFG_FP1_PRS_DISABLE PPC_BIT(57) +#define OB0_ODL0_CONFIG 0x901082A +#define OB0_ODL1_CONFIG 0x901082B +#define OB3_ODL0_CONFIG 0xC01082A +#define OB3_ODL1_CONFIG 0xC01082B +#define OB_ODL_CONFIG_RESET PPC_BIT(0) +#define OB_ODL_CONFIG_VERSION PPC_BITMASK(2, 7) +#define OB_ODL_CONFIG_TRAIN_MODE PPC_BITMASK(8, 11) +#define OB_ODL_CONFIG_SUPPORTED_MODES PPC_BITMASK(12, 15) +#define OB_ODL_CONFIG_X4_BACKOFF_ENABLE PPC_BIT(16) +#define OB_ODL_CONFIG_PHY_CNTR_LIMIT PPC_BITMASK(20, 23) +#define OB_ODL_CONFIG_DEBUG_ENABLE PPC_BIT(33) +#define OB_ODL_CONFIG_FWD_PROGRESS_TIMER PPC_BITMASK(40, 43) + +#define OB0_ODL0_STATUS 0x901082C +#define OB0_ODL1_STATUS 0x901082D +#define OB3_ODL0_STATUS 0xC01082C +#define OB3_ODL1_STATUS 0xC01082D +#define OB_ODL_STATUS_TRAINED_MODE PPC_BITMASK(0,3) +#define OB_ODL_STATUS_TRAINING_STATE_MACHINE PPC_BITMASK(49, 51) + #define OB0_ODL0_TRAINING_STATUS 0x901082E #define OB0_ODL1_TRAINING_STATUS 0x901082F #define OB3_ODL0_TRAINING_STATUS 0xC01082E diff --git a/include/npu2.h b/include/npu2.h index 2922d2185296..b9de3c1b77e2 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -207,5 +207,10 @@ extern int nv_zcal_nominal; bool is_p9dd1(void); void npu2_opencapi_phy_setup(struct npu2_dev *dev); void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev); - +int64_t npu2_freeze_status(struct phb *phb __unused, + uint64_t pe_number __unused, + uint8_t *freeze_state, + uint16_t *pci_error_type __unused, + uint16_t *severity __unused, + uint64_t *phb_status __unused); #endif /* __NPU2_H */ From patchwork Wed Feb 7 06:50:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870252 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsV92Ph1z9ryk for ; Wed, 7 Feb 2018 17:53:21 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsV90yR2zF0yq for ; Wed, 7 Feb 2018 17:53:21 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRn6bb2zF0TH for ; Wed, 7 Feb 2018 17:51:16 +1100 (AEDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176oemd025474 for ; Wed, 7 Feb 2018 01:51:15 -0500 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0b-001b2d01.pphosted.com with ESMTP id 2fytafve6e-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:14 -0500 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:12 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:10 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAxM33489062; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F16EDA4053; Wed, 7 Feb 2018 06:44:35 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5A79DA404D; Wed, 7 Feb 2018 06:44:35 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:35 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 9D951A03A5; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:54 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0020-0000-0000-000003F31B67 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0021-0000-0000-0000428594BD Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 08/10] npu2-opencapi: Add OpenCAPI OPAL API calls X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Frederic Barrat Add three OPAL API calls that are required by the ocxl driver. - OPAL_NPU_SPA_SETUP The Shared Process Area (SPA) is a table containing one entry (a "Process Element") per memory context which can be accessed by the OpenCAPI device. - OPAL_NPU_SPA_CLEAR_CACHE The NPU keeps a cache of recently accessed memory contexts. When a Process Element is removed from the SPA, the cache for the link must be cleared. - OPAL_NPU_TL_SET The Transaction Layer specification defines several templates for messages to be exchanged on the link. During link setup, the host and device must negotiate what templates are supported on both sides and at what rates those messages can be sent. Signed-off-by: Frederic Barrat Signed-off-by: Andrew Donnellan --- doc/opal-api/opal-npu2-opencapi-159-160-161.rst | 126 ++++++++++- hw/npu2-opencapi.c | 206 +++++++++++++++++- include/npu2-regs.h | 4 +- include/opal-api.h | 5 +- 4 files changed, 340 insertions(+), 1 deletion(-) create mode 100644 doc/opal-api/opal-npu2-opencapi-159-160-161.rst diff --git a/doc/opal-api/opal-npu2-opencapi-159-160-161.rst b/doc/opal-api/opal-npu2-opencapi-159-160-161.rst new file mode 100644 index 000000000000..4db3d3e8961e --- /dev/null +++ b/doc/opal-api/opal-npu2-opencapi-159-160-161.rst @@ -0,0 +1,126 @@ +.. _OPAL_NPU_SPA_SETUP: + +OPAL_NPU_SPA_SETUP +================== + +OpenCAPI devices only. + +Sets up a Shared Process Area (SPA) with the Shared Process Area +Pointer (SPAP) set to the provided address `addr`, and sets the OTL PE +mask (used for PASID to PE handle conversion) to `PE_mask`. + +If `addr` is NULL, the SPA will be disabled. `addr` must be 4K aligned. + +Parameters +---------- +:: + + uint64_t phb_id + int bdfn + uint64_t addr + uint64_t PE_mask + +``phb_id`` + OPAL ID of PHB + +``bdfn`` + Bus-Device-Function number of OpenCAPI AFU + +``addr`` + Address of Shared Process Area, or NULL to disable SPA. Must be 4K aligned. + +``PE_mask`` + Process Element mask for PASID to PE handle conversion + +Return Values +------------- + +OPAL_SUCCESS + SPAP and PE mask were successfully set + +OPAL_PARAMETER + A provided parameter was invalid + +OPAL_BUSY + SPA is already enabled (or if addr is NULL, SPA is already disabled) + +.. _OPAL_NPU_SPA_CLEAR_CACHE: + +OPAL_NPU_SPA_CLEAR_CACHE +======================== + +OpenCAPI devices only. + +Invalidates the Process Element with the given `PE_handle` from the NPU's SPA cache. + +Parameters +---------- +:: + + uint64_t phb_id + uint32_t bdfn + uint64_t PE_handle + +``phb_id`` + OPAL ID of PHB + +``bdfn`` + Bus-Device-Function number of OpenCAPI AFU + +``PE_handle`` + Handle of Process Element being cleared from SPA cache + +Return Values +------------- + +OPAL_SUCCESS + PE was successfully cleared from SPA cache + +OPAL_PARAMETER + A provided parameter was invalid + +OPAL_BUSY + XSLO is currently invalidating a previously requested entry + +.. _OPAL_NPU_TL_SET: + +OPAL_NPU_TL_SET +=============== + +OpenCAPI devices only. + +Update the NPU OTL configuration with device capabilities. + +Parameters +---------- +:: + + uint64_t phb_id + uint32_t bdfn + long capabilities + uint64_t rate_phys + int rate_sz + +``phb_id`` + OPAL ID of PHB + +``bdfn`` + Bus-Device-Function number of OpenCAPI AFU + +``capabilities`` + Bitmap of TL templates the device can receive + +``rate_phys`` + Physical address of rates buffer + +``rate_sz`` + Size of rates buffer (must be equal to 32) + +Return Values +------------- + +OPAL_SUCCESS + OTL configuration was successfully updated + +OPAL_PARAMETER + A provided parameter was invalid diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index c4e263c6d57f..a2bfea8a04cd 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -54,6 +54,9 @@ #define NPU_IRQ_LEVELS 35 #define NPU_IRQ_LEVELS_XSL 23 +#define MAX_PE_HANDLE ((1 << 15) - 1) +#define TL_MAX_TEMPLATE 63 +#define TL_RATE_BUF_SIZE 32 static const struct phb_ops npu2_opencapi_ops; @@ -1470,3 +1473,206 @@ static const struct phb_ops npu2_opencapi_ops = { .set_capp_recovery = NULL, .tce_kill = NULL, }; + +static int64_t opal_npu_spa_setup(uint64_t phb_id, uint32_t __unused bdfn, + uint64_t addr, uint64_t PE_mask) +{ + uint64_t stack, block, offset, reg; + struct phb *phb = pci_get_phb(phb_id); + struct npu2_dev *dev; + int rc; + + if (!phb || phb->phb_type != phb_type_npu_v2_opencapi) + return OPAL_PARAMETER; + + /* 4k aligned */ + if (addr & 0xFFF) + return OPAL_PARAMETER; + + if (PE_mask > 15) + return OPAL_PARAMETER; + + dev = phb_to_npu2_dev_ocapi(phb); + if (!dev) + return OPAL_PARAMETER; + + block = index_to_block(dev->index); + stack = index_to_stack(dev->index); + if (block == NPU2_BLOCK_OTL1) + offset = NPU2_XSL_PSL_SPAP_A1; + else + offset = NPU2_XSL_PSL_SPAP_A0; + + + lock(&dev->npu->lock); + /* + * set the SPAP used by the device + */ + reg = npu2_scom_read(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, offset), + NPU2_MISC_DA_LEN_8B); + if ((addr && (reg & NPU2_XSL_PSL_SPAP_EN)) || + (!addr && !(reg & NPU2_XSL_PSL_SPAP_EN))) { + rc = OPAL_BUSY; + goto out; + } + /* SPA is disabled by passing a NULL address */ + reg = addr; + if (addr) + reg = addr | NPU2_XSL_PSL_SPAP_EN; + + npu2_scom_write(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, offset), + NPU2_MISC_DA_LEN_8B, reg); + + /* + * set the PE mask that the OS uses for PASID -> PE handle + * conversion + */ + reg = npu2_scom_read(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_OTL_CONFIG0(stack, block), NPU2_MISC_DA_LEN_8B); + reg &= ~NPU2_OTL_CONFIG0_PE_MASK; + reg |= (PE_mask << (63-7)); + npu2_scom_write(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_OTL_CONFIG0(stack, block), NPU2_MISC_DA_LEN_8B, + reg); + rc = OPAL_SUCCESS; +out: + unlock(&dev->npu->lock); + return rc; +} +opal_call(OPAL_NPU_SPA_SETUP, opal_npu_spa_setup, 4); + +static int64_t opal_npu_spa_clear_cache(uint64_t phb_id, uint32_t __unused bdfn, + uint64_t PE_handle) +{ + uint64_t cc_inv, stack, block, reg, rc; + uint32_t retries = 5; + struct phb *phb = pci_get_phb(phb_id); + struct npu2_dev *dev; + + if (!phb || phb->phb_type != phb_type_npu_v2_opencapi) + return OPAL_PARAMETER; + + if (PE_handle > MAX_PE_HANDLE) + return OPAL_PARAMETER; + + dev = phb_to_npu2_dev_ocapi(phb); + if (!dev) + return OPAL_PARAMETER; + + block = index_to_block(dev->index); + stack = index_to_stack(dev->index); + cc_inv = NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_PSL_LLCMD_A0); + + lock(&dev->npu->lock); + reg = npu2_scom_read(dev->npu->chip_id, dev->npu->xscom_base, cc_inv, + NPU2_MISC_DA_LEN_8B); + if (reg & PPC_BIT(16)) { + rc = OPAL_BUSY; + goto out; + } + + reg = PE_handle | PPC_BIT(15); + if (block == NPU2_BLOCK_OTL1) + reg |= PPC_BIT(48); + npu2_scom_write(dev->npu->chip_id, dev->npu->xscom_base, cc_inv, + NPU2_MISC_DA_LEN_8B, reg); + + rc = OPAL_HARDWARE; + while (retries--) { + reg = npu2_scom_read(dev->npu->chip_id, dev->npu->xscom_base, + cc_inv, NPU2_MISC_DA_LEN_8B); + if (!(reg & PPC_BIT(16))) { + rc = OPAL_SUCCESS; + break; + } + /* the bit expected to flip in less than 200us */ + time_wait_us(200); + } +out: + unlock(&dev->npu->lock); + return rc; +} +opal_call(OPAL_NPU_SPA_CLEAR_CACHE, opal_npu_spa_clear_cache, 3); + +static int get_template_rate(unsigned int templ, char *rate_buf) +{ + int shift, idx, val; + + /* + * Each rate is encoded over 4 bits (0->15), with 15 being the + * slowest. The buffer is a succession of rates for all the + * templates. The first 4 bits are for template 63, followed + * by 4 bits for template 62, ... etc. So the rate for + * template 0 is at the very end of the buffer. + */ + idx = (TL_MAX_TEMPLATE - templ) / 2; + shift = 4 * (1 - ((TL_MAX_TEMPLATE - templ) % 2)); + val = rate_buf[idx] >> shift; + return val; +} + +static bool is_template_supported(unsigned int templ, long capabilities) +{ + return !!(capabilities & (1ull << templ)); +} + +static int64_t opal_npu_tl_set(uint64_t phb_id, uint32_t bdfn, + long capabilities, uint64_t rate_phys, int rate_sz) +{ + struct phb *phb = pci_get_phb(phb_id); + struct npu2_dev *dev; + uint64_t stack, block, reg, templ_rate; + int i, rate_pos; + char *rate = (char *) rate_phys; + + if (!phb || phb->phb_type != phb_type_npu_v2_opencapi) + return OPAL_PARAMETER; + if (!opal_addr_valid(rate) || rate_sz != TL_RATE_BUF_SIZE) + return OPAL_PARAMETER; + + dev = phb_to_npu2_dev_ocapi(phb); + if (!dev) + return OPAL_PARAMETER; + + block = index_to_block(dev->index); + stack = index_to_stack(dev->index); + /* + * The 'capabilities' argument defines what TL template the + * device can receive. OpenCAPI 3.0 and 4.0 define 64 templates, so + * that's one bit per template. + * + * For each template, the device processing time may vary, so + * the device advertises at what rate a message of a given + * template can be sent. That's encoded in the 'rate' buffer. + * + * On P9, NPU only knows about TL templates 0 -> 3. + * Per the spec, template 0 must be supported. + */ + if (!is_template_supported(0, capabilities)) + return OPAL_PARAMETER; + + reg = npu2_scom_read(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_OTL_CONFIG1(stack, block), + NPU2_MISC_DA_LEN_8B); + reg &= ~(NPU2_OTL_CONFIG1_TX_TEMP1_EN | NPU2_OTL_CONFIG1_TX_TEMP3_EN | + NPU2_OTL_CONFIG1_TX_TEMP1_EN); + for (i = 0; i < 4; i++) { + /* Skip template 0 as it is implicitly enabled */ + if (i && is_template_supported(i, capabilities)) + reg |= PPC_BIT(i); + /* The tx rate should still be set for template 0 */ + templ_rate = get_template_rate(i, rate); + rate_pos = 8 + i * 4; + reg = SETFIELD(PPC_BITMASK(rate_pos, rate_pos + 3), reg, + templ_rate); + } + npu2_scom_write(dev->npu->chip_id, dev->npu->xscom_base, + NPU2_OTL_CONFIG1(stack, block), NPU2_MISC_DA_LEN_8B, + reg); + prlog(PR_DEBUG, "OCAPI: Link %llx:%x, TL conf1 register set to %llx\n", + phb_id, bdfn, reg); + return OPAL_SUCCESS; +} +opal_call(OPAL_NPU_TL_SET, opal_npu_tl_set, 5); diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 71c8d8f07af0..6efab0843eca 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -321,6 +321,7 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_NTL_DL_CLK_CTRL(ndev) NPU2_DL_REG_OFFSET(ndev, 0x001C) /* OpenCAPI - XSL registers */ +#define NPU2_XSL_PSL_LLCMD_A0 0x008 #define NPU2_XSL_PSL_SCNTL_A0 0x010 #define NPU2_XSL_PSL_SCNTL_A0_MULTI_AFU_DIAL PPC_BIT(0) #define NPU2_XSL_DEF 0x040 @@ -328,6 +329,9 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define NPU2_XSL_GP_BLOOM_FILTER_ENABLE PPC_BIT(16) #define NPU2_XSL_WRAP_CFG 0x0C0 #define NPU2_XSL_WRAP_CFG_XSLO_CLOCK_ENABLE PPC_BIT(0) +#define NPU2_XSL_PSL_SPAP_A0 0 +#define NPU2_XSL_PSL_SPAP_A1 0x18 +#define NPU2_XSL_PSL_SPAP_EN PPC_BIT(63) /* OpenCAPI - OTL registers */ #define NPU2_OTL_CONFIG0(stack, block) NPU2_REG_OFFSET(stack, block, 0x000) diff --git a/include/opal-api.h b/include/opal-api.h index f0ed5f6c69db..1496f389ba24 100644 --- a/include/opal-api.h +++ b/include/opal-api.h @@ -215,7 +215,10 @@ #define OPAL_SENSOR_GROUP_CLEAR 156 #define OPAL_PCI_SET_P2P 157 #define OPAL_QUIESCE 158 -#define OPAL_LAST 158 +#define OPAL_NPU_SPA_SETUP 159 +#define OPAL_NPU_SPA_CLEAR_CACHE 160 +#define OPAL_NPU_TL_SET 161 +#define OPAL_LAST 161 #define QUIESCE_HOLD 1 /* Spin all calls at entry */ #define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */ From patchwork Wed Feb 7 06:50:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870253 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsVQ1Pw8z9ryk for ; Wed, 7 Feb 2018 17:53:34 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsVQ043YzF13f for ; Wed, 7 Feb 2018 17:53:34 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRp1qJKzF0ZD for ; Wed, 7 Feb 2018 17:51:18 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176n1F5144144 for ; Wed, 7 Feb 2018 01:51:16 -0500 Received: from e06smtp11.uk.ibm.com (e06smtp11.uk.ibm.com [195.75.94.107]) by mx0a-001b2d01.pphosted.com with ESMTP id 2fyt87vx7j-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:15 -0500 Received: from localhost by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:13 -0000 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp11.uk.ibm.com (192.168.101.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:10 -0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAe841353440; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3490D4C046; Wed, 7 Feb 2018 06:45:01 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8DDF54C040; Wed, 7 Feb 2018 06:45:00 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:45:00 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B07ECA03A8; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:55 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0040-0000-0000-0000042D37F9 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0041-0000-0000-000020D0E945 Message-Id: <2ab263cfa4e9eed497fe6600c1816501ec9f620d.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 09/10] platforms: Add OpenCAPI platform data and device tree nodes X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Add OpenCAPI platform data for the zaius and zz platforms, as well as the generic platform that's used for BML boots. Currently, all three platforms are identical, but that won't always be the case (e.g. Witherspoon). Additionally, for Zaius, hardcode link information and the I2C bus that's required to reset OpenCAPI devices, because this currently can't be extracted from HDAT. Eventually this will go away once Hostboot adds the relevant data in HDAT. A later patch will add this for ZZ once we've done more testing. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - Remove extraneous break that stops things after the first CPU (Fred) - Remove i2c odl01 data (Fred) - Remove i2c 1.8v hack (Fred) - Don't try to create device tree nodes on zaius if they already exist (Fred) v2->v3: - Reverse bit order of lane masks --- core/platform.c | 11 +++++- platforms/astbmc/zaius.c | 96 +++++++++++++++++++++++++++++++++++++++++- platforms/ibm-fsp/zz.c | 11 +++++- 3 files changed, 118 insertions(+) diff --git a/core/platform.c b/core/platform.c index 6816fe58a252..f09ea3c132b6 100644 --- a/core/platform.c +++ b/core/platform.c @@ -168,6 +168,16 @@ static int generic_start_preload_resource(enum resource_id id, uint32_t subid, return OPAL_EMPTY; } +/* These values will work for a ZZ booted using BML */ +const struct platform_ocapi generic_ocapi = { + .i2c_engine = 1, + .i2c_port = 4, + .i2c_offset = { 0x3, 0x1, 0x1 }, + .i2c_odl0_data = { 0xFD, 0xFD, 0xFF }, + .i2c_odl1_data = { 0xBF, 0xBF, 0xFF }, + .odl_phy_swap = true, +}; + static struct bmc_platform generic_bmc = { .name = "generic", }; @@ -183,6 +193,7 @@ static struct platform generic_platform = { .cec_power_down = generic_cec_power_down, .start_preload_resource = generic_start_preload_resource, .resource_loaded = generic_resource_loaded, + .ocapi = &generic_ocapi, }; const struct bmc_platform *bmc_platform = &generic_bmc; diff --git a/platforms/astbmc/zaius.c b/platforms/astbmc/zaius.c index 74c7f13e35e6..d7064647a8b3 100644 --- a/platforms/astbmc/zaius.c +++ b/platforms/astbmc/zaius.c @@ -24,6 +24,98 @@ #include "astbmc.h" +const struct platform_ocapi zaius_ocapi = { + .i2c_engine = 1, + .i2c_port = 4, + .i2c_offset = { 0x3, 0x1, 0x1 }, + .i2c_odl0_data = { 0xFD, 0xFD, 0xFF }, + .i2c_odl1_data = { 0xBF, 0xBF, 0xFF }, + .odl_phy_swap = true, +}; + +#define NPU_BASE 0x5011000 +#define NPU_SIZE 0x2c +#define NPU_INDIRECT0 0x8000000009010c3f /* OB0 - no OB3 on Zaius */ + +/* OpenCAPI only */ +static void create_link(struct dt_node *npu, int group, int index) +{ + struct dt_node *link; + uint32_t lane_mask; + char namebuf[32]; + + snprintf(namebuf, sizeof(namebuf), "link@%x", index); + link = dt_new(npu, namebuf); + + dt_add_property_string(link, "compatible", "ibm,npu-link-opencapi"); + dt_add_property_cells(link, "ibm,npu-link-index", index); + + switch (index) { + case 2: + lane_mask = 0xf1e000; /* 0-3, 7-10 */ + break; + case 3: + lane_mask = 0x00078f; /* 13-16, 20-23 */ + break; + default: + assert(0); + } + + dt_add_property_u64s(link, "ibm,npu-phy", NPU_INDIRECT0); + dt_add_property_cells(link, "ibm,npu-lane-mask", lane_mask); + dt_add_property_cells(link, "ibm,npu-group-id", group); +} + +/* FIXME: Get rid of this after we get NPU information properly via HDAT/MRW */ +static void zaius_create_npu(void) +{ + struct dt_node *xscom, *npu; + int npu_index = 0; + int phb_index = 7; + char namebuf[32]; + + /* Abort if there's already an NPU in the device tree */ + if (dt_find_compatible_node(dt_root, NULL, "ibm,power9-npu")) + return; + + prlog(PR_DEBUG, "OCAPI: Adding NPU device nodes\n"); + dt_for_each_compatible(dt_root, xscom, "ibm,xscom") { + snprintf(namebuf, sizeof(namebuf), "npu@%x", NPU_BASE); + npu = dt_new(xscom, namebuf); + dt_add_property_cells(npu, "reg", NPU_BASE, NPU_SIZE); + dt_add_property_strings(npu, "compatible", "ibm,power9-npu"); + dt_add_property_cells(npu, "ibm,npu-index", npu_index++); + dt_add_property_cells(npu, "ibm,phb-index", phb_index++); + dt_add_property_cells(npu, "ibm,npu-links", 2); + create_link(npu, 1, 2); + create_link(npu, 2, 3); + } +} + +/* FIXME: Get rid of this after we get NPU information properly via HDAT/MRW */ +static void zaius_create_ocapi_i2c_bus(void) +{ + struct dt_node *xscom, *i2cm, *i2c_bus; + prlog(PR_DEBUG, "OCAPI: Adding I2C bus device node for OCAPI reset\n"); + dt_for_each_compatible(dt_root, xscom, "ibm,xscom") { + i2cm = dt_find_by_name(xscom, "i2cm@a1000"); + if (!i2cm) { + prlog(PR_ERR, "OCAPI: Failed to add I2C bus device node\n"); + continue; + } + + if (dt_find_by_name(i2cm, "i2c-bus@4")) + continue; + + i2c_bus = dt_new_addr(i2cm, "i2c-bus", 4); + dt_add_property_cells(i2c_bus, "reg", 4); + dt_add_property_cells(i2c_bus, "bus-frequency", 0x61a80); + dt_add_property_strings(i2c_bus, "compatible", + "ibm,opal-i2c", "ibm,power8-i2c-port", + "ibm,power9-i2c-port"); + } +} + static bool zaius_probe(void) { if (!dt_node_is_compatible(dt_root, "ingrasys,zaius")) @@ -35,6 +127,9 @@ static bool zaius_probe(void) /* Setup UART for direct use by Linux */ uart_set_console_policy(UART_CONSOLE_OS); + zaius_create_npu(); + zaius_create_ocapi_i2c_bus(); + return true; } @@ -52,4 +147,5 @@ DECLARE_PLATFORM(zaius) = { .elog_commit = ipmi_elog_commit, .exit = ipmi_wdt_final_reset, .terminate = ipmi_terminate, + .ocapi = &zaius_ocapi, }; diff --git a/platforms/ibm-fsp/zz.c b/platforms/ibm-fsp/zz.c index a8150b81363a..9a8492907a7e 100644 --- a/platforms/ibm-fsp/zz.c +++ b/platforms/ibm-fsp/zz.c @@ -27,6 +27,16 @@ #include "ibm-fsp.h" #include "lxvpd.h" +/* We don't yet create NPU device nodes on ZZ, but these values are correct */ +const struct platform_ocapi zz_ocapi = { + .i2c_engine = 1, + .i2c_port = 4, + .i2c_offset = { 0x3, 0x1, 0x1 }, + .i2c_odl0_data = { 0xFD, 0xFD, 0xFF }, + .i2c_odl1_data = { 0xBF, 0xBF, 0xFF }, + .odl_phy_swap = true, +}; + static bool zz_probe(void) { /* FIXME: make this neater when the dust settles */ @@ -71,4 +81,5 @@ DECLARE_PLATFORM(zz) = { .resource_loaded = fsp_resource_loaded, .sensor_read = ibm_fsp_sensor_read, .terminate = ibm_fsp_terminate, + .ocapi = &zz_ocapi, }; From patchwork Wed Feb 7 06:50:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 870250 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zbsTg2rcCz9ryk for ; Wed, 7 Feb 2018 17:52:55 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zbsTg1tjczF13W for ; Wed, 7 Feb 2018 17:52:55 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zbsRm5g9pzF0Yd for ; Wed, 7 Feb 2018 17:51:16 +1100 (AEDT) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w176n2WP124883 for ; Wed, 7 Feb 2018 01:51:14 -0500 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0b-001b2d01.pphosted.com with ESMTP id 2fypywbe3e-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 07 Feb 2018 01:51:14 -0500 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 7 Feb 2018 06:51:12 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 7 Feb 2018 06:51:10 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w176pAKQ46399562; Wed, 7 Feb 2018 06:51:10 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5960442054; Wed, 7 Feb 2018 06:44:07 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BFFDF42049; Wed, 7 Feb 2018 06:44:06 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 7 Feb 2018 06:44:06 +0000 (GMT) Received: from intelligence.au.ibm.com (unknown [9.192.255.83]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B9053A03AA; Wed, 7 Feb 2018 17:51:06 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org, fbarrat@linux.vnet.ibm.com Date: Wed, 7 Feb 2018 17:50:56 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18020706-0020-0000-0000-000003F31B69 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18020706-0021-0000-0000-0000428594BE Message-Id: <191721e7a8969d5ef8f8fd181f20b2b088a0f30a.1517986252.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-07_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802070086 Subject: [Skiboot] [PATCH v5 10/10] doc/device-tree: Add OpenCAPI device tree bindings X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au, arbab@linux.vnet.ibm.com MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Suggested-by: Stewart Smith Signed-off-by: Andrew Donnellan --- doc/device-tree/opencapi.rst | 95 +++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+) create mode 100644 doc/device-tree/opencapi.rst diff --git a/doc/device-tree/opencapi.rst b/doc/device-tree/opencapi.rst new file mode 100644 index 000000000000..54ac36cf477d --- /dev/null +++ b/doc/device-tree/opencapi.rst @@ -0,0 +1,95 @@ +.. _device-tree/opencapi: + +============================= +OpenCAPI Device Tree Bindings +============================= + +NPU bindings: + +.. code-block:: dts + + xscom@603fc00000000 { + npu@5011000 { + compatible = "ibm,power9-npu"; + phandle = <0xe6>; + ibm,phb-index = <0x7>; + reg = <0x5011000 0x2c>; + ibm,npu-index = <0x0>; + ibm,npu-links = <0x2>; /* Number of links wired up to this npu. */ + + link@2 { + compatible = "ibm,npu-link-opencapi"; + ibm,npu-group-id = <0x1>; + ibm,npu-lane-mask = <0xf1e000>; /* Mask specifying which IBM PHY lanes + * are used for this link. 24-bit, + * lane 0 is most significant bit */ + ibm,npu-phy = <0x80000000 0x9010c3f>; /* SCOM address of the IBM PHY + * controlling this link. */ + ibm,npu-link-index = <0x2>; /* Hardware link index. + * Used to calculate various address offsets. */ + phandle = <0xe7>; + }; + + link@3 { + compatible = "ibm,npu-link-opencapi"; + ibm,npu-group-id = <0x2>; + ibm,npu-lane-mask = <0x78f>; + ibm,npu-phy = <0x80000000 0x9010c3f>; + ibm,npu-link-index = <0x3>; + phandle = <0xe8>; + }; + }; + }; + +PCI device bindings +------------------- + +.. code-block:: dts + + pciex@600e800000000 { + compatible = "ibm,power9-npu-opencapi-pciex", "ibm,ioda2-npu2-opencapi-phb"; + ibm,mmio-window = <0x600e8 0x0 0x8 0x0>; + ibm,opal-num-pes = <0x10>; + device_type = "pciex"; + ibm,links = <0x1>; + ibm,phb-diag-data-size = <0x0>; + ibm,xscom-base = <0x5011000>; + ranges = <0x2000000 0x600e8 0x0 0x600e8 0x0 0x8 0x0>; + #interrupt-cells = <0x1>; + bus-range = <0x0 0xff>; + interrupt-parent = <0xdf>; + #address-cells = <0x3>; + ibm,opal-phbid = <0x0 0x5>; + ibm,npcq = <0xe6>; /* phandle to the NPU node */ + ibm,chip-id = <0x0>; + #size-cells = <0x2>; + phandle = <0x581>; + reg = <0x600e8 0x0 0x8 0x0>; + clock-frequency = <0x200 0x0>; + ibm,npu-index = <0x0>; + + device@0 { + ibm,opal-xsl-irq = <0x58>; + revision-id = <0x0>; + ibm,opal-xsl-mmio = <0x60302 0x1d0000 0x60302 0x1d0008 0x60302 0x1d0010 0x60302 0x1d0018>; + ibm,pci-config-space-type = <0x1>; + class-code = <0x120000>; + vendor-id = <0x1014>; + device-id = <0x62b>; + phandle = <0x58d>; + reg = <0x0 0x0 0x0 0x0 0x0>; + }; + + device@0,1 { + ibm,opal-xsl-irq = <0x58>; + revision-id = <0x0>; + ibm,opal-xsl-mmio = <0x60302 0x1d0000 0x60302 0x1d0008 0x60302 0x1d0010 0x60302 0x1d0018>; + ibm,pci-config-space-type = <0x1>; + class-code = <0x120000>; + vendor-id = <0x1014>; + device-id = <0x62b>; + phandle = <0x58e>; + reg = <0x100 0x0 0x0 0x0 0x0>; + }; + }; +