From patchwork Mon Aug 27 08:55:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962377 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQjF5fCRz9s2P for ; Mon, 27 Aug 2018 18:56:17 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQjF4WJ8zDrM8 for ; Mon, 27 Aug 2018 18:56:17 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj71rd9zDrGC for ; Mon, 27 Aug 2018 18:56:10 +1000 (AEST) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8mpou103080 for ; Mon, 27 Aug 2018 04:56:08 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m4dj9henf-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:08 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:06 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:04 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u3x741287746 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:03 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F3BC411C04A; Mon, 27 Aug 2018 11:56:00 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AC40311C06C; Mon, 27 Aug 2018 11:56:00 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:56:00 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 73F9EA0278; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:50 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-0008-0000-0000-0000026729EC X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-0009-0000-0000-000021CF7CBC Message-Id: <543316e7a0efa5d60fe6196d4aa1ed6a5cbef9e5.1535359753.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=569 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 1/6] occ: Wait if OCC GPU presence status not immediately available X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" It takes a few seconds for the OCC to set everything up in order to read GPU presence. At present, we try to kick off OCC initialisation as early as possible to maximise the time it has to read GPU presence. Unfortunately sometimes that's not enough, so add a loop in occ_get_gpu_presence() so that on the first time we try to get GPU presence we keep trying for up to 2 seconds. Experimentally this seems to be adequate. Fixes: 9b394a32c8ea ("occ: Add support for GPU presence detection") Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/occ.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/hw/occ.c b/hw/occ.c index a55bf8ed4f54..9fcac3f9581c 100644 --- a/hw/occ.c +++ b/hw/occ.c @@ -1238,14 +1238,26 @@ exit: bool occ_get_gpu_presence(struct proc_chip *chip, int gpu_num) { struct occ_dynamic_data *ddata; + static int max_retries = 20; + static bool found = false; assert(gpu_num <= 2); ddata = get_occ_dynamic_data(chip); - - if (ddata->major_version != 0 || ddata->minor_version < 1) { + while (!found && max_retries) { + if (ddata->major_version == 0 && ddata->minor_version >= 1) { + found = true; + break; + } prlog(PR_INFO, "OCC: OCC not reporting GPU slot presence, " - "assuming device is present\n"); + "waiting\n"); + time_wait_ms(100); + max_retries--; + ddata = get_occ_dynamic_data(chip); + } + + if (!found) { + prlog(PR_INFO, "OCC: No GPU slot presence, assuming GPU present\n"); return true; } From patchwork Mon Aug 27 08:55:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962381 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQll5BHPz9s2P for ; Mon, 27 Aug 2018 18:58:27 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQll3r16zDrZC for ; Mon, 27 Aug 2018 18:58:27 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj83CZ3zDrGC for ; Mon, 27 Aug 2018 18:56:12 +1000 (AEST) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8moFD084433 for ; Mon, 27 Aug 2018 04:56:09 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m4cp8bvww-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:09 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:07 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:05 +0100 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u4xv32243840 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:04 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B2C46A4055; Mon, 27 Aug 2018 11:56:01 +0100 (BST) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CAE0FA4059; Mon, 27 Aug 2018 11:56:00 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:56:00 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 7FE60A030E; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:51 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-0028-0000-0000-000002F02CAD X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-0029-0000-0000-000023A97EBA Message-Id: <13603320ad3e6fe3127bf0fe63a27f805f95995b.1535359753.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 2/6] npu2: Split device index into brick and link index X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" On Witherspoon, OpenCAPI devices attached to link indexes 0 and 1 are handled by bricks 2 and 3. Rename index to brick_index, and add a new field, link_index, to refer to the link index. For now, we set those values identically. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- hw/npu2-hw-procedures.c | 8 ++-- hw/npu2-opencapi.c | 77 +++++++++++++++++++++--------------------- hw/npu2.c | 21 ++++++----- include/npu2-regs.h | 14 ++++---- include/npu2.h | 7 ++-- 5 files changed, 67 insertions(+), 60 deletions(-) diff --git a/hw/npu2-hw-procedures.c b/hw/npu2-hw-procedures.c index 86864629a66b..8767d9259984 100644 --- a/hw/npu2-hw-procedures.c +++ b/hw/npu2-hw-procedures.c @@ -198,7 +198,7 @@ DEFINE_PROCEDURE(nop); /* Return the brick number (0-2) within an obus chiplet */ static int obus_brick_index(struct npu2_dev *ndev) { - int index = ndev->index % 3; + int index = ndev->brick_index % 3; /* On the second obus chiplet, index is reversed */ if ((ndev->pl_xscom_base & 0x3F000000) != 0x09000000) @@ -432,7 +432,7 @@ DEFINE_PROCEDURE(phy_reset, phy_reset_wait, phy_reset_complete); /* Procedure 1.2.6 - I/O PHY Tx Impedance Calibration */ static uint32_t phy_tx_zcal(struct npu2_dev *ndev) { - if (ndev->npu->tx_zcal_complete[ndev->index > 2]) + if (ndev->npu->tx_zcal_complete[ndev->brick_index > 2]) return PROCEDURE_COMPLETE; /* Turn off SW enable and enable zcal state machine */ @@ -603,7 +603,7 @@ static uint32_t phy_tx_zcal_calculate(struct npu2_dev *ndev) phy_write(ndev, &NPU2_PHY_TX_MARGINPU_SELECT, therm(margin_select + 1)/2); phy_write(ndev, &NPU2_PHY_TX_MARGINPD_SELECT, therm(margin_select + 1)/2); - ndev->npu->tx_zcal_complete[ndev->index > 2] = 1; + ndev->npu->tx_zcal_complete[ndev->brick_index > 2] = 1; return PROCEDURE_COMPLETE; } DEFINE_PROCEDURE(phy_tx_zcal, phy_tx_zcal_wait, phy_tx_zcal_calculate); @@ -977,7 +977,7 @@ void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev) uint64_t status_xscom; int lane, bit = 7; - switch (dev->index) { + switch (dev->brick_index) { case 2: status_xscom = OB0_ODL0_TRAINING_STATUS; break; diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 57c2f2fcc075..68ae93a2f836 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -53,11 +53,11 @@ #include #define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define NPU_IRQ_LEVELS 35 @@ -753,16 +753,16 @@ static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); - uint64_t offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + uint64_t stack = index_to_stack(dev->brick_index); + uint64_t offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_NTL0_BAR : NPU2_NTL1_BAR; - uint64_t pa_offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + uint64_t pa_offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; uint64_t addr, size, reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - phys_map_get(gcid, NPU_OCAPI_MMIO, dev->index, &addr, &size); + phys_map_get(gcid, NPU_OCAPI_MMIO, dev->brick_index, &addr, &size); prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, @@ -783,7 +783,7 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); + uint64_t stack = index_to_stack(dev->brick_index); int stack_num = stack - NPU2_STACK_STCK_0; uint64_t addr, size; @@ -799,8 +799,8 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, static void otl_enabletx(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); - uint64_t block = index_to_block(dev->index); + uint64_t stack = index_to_stack(dev->brick_index); + uint64_t block = index_to_block(dev->brick_index); uint64_t reg; /* OTL Config 2 Register */ @@ -822,7 +822,7 @@ static void assert_reset(struct npu2_dev *dev) uint8_t pin, data; int rc; - switch (dev->index) { + switch (dev->brick_index) { case 2: case 4: pin = platform.ocapi->i2c_reset_odl0; @@ -910,7 +910,7 @@ static bool i2c_presence_detect(struct npu2_dev *dev) OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); - switch (dev->index) { + switch (dev->brick_index) { // TODO(ajd): Link or brick index? case 2: data = platform.ocapi->i2c_presence_odl0; break; @@ -929,7 +929,7 @@ static void reset_odl(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -965,7 +965,7 @@ static void set_init_pattern(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -992,7 +992,7 @@ static void start_training(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -1035,7 +1035,7 @@ static int64_t npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) uint64_t reg; int64_t link_width, training_status, rc = OPAL_SUCCESS; - reg = get_odl_status(dev->npu->chip_id, dev->index); + reg = get_odl_status(dev->npu->chip_id, dev->brick_index); link_width = GETFIELD(OB_ODL_STATUS_TRAINED_MODE, reg); training_status = GETFIELD(OB_ODL_STATUS_TRAINING_STATE_MACHINE, reg); @@ -1071,7 +1071,7 @@ static int64_t npu2_opencapi_retry_state(struct pci_slot *slot) */ OCAPIERR(dev, "Link failed to train, final link status: %016llx\n", - get_odl_status(chip_id, dev->index)); + get_odl_status(chip_id, dev->brick_index)); return OPAL_HARDWARE; } @@ -1093,7 +1093,7 @@ static int64_t npu2_opencapi_poll_link(struct pci_slot *slot) pci_slot_set_state(slot, OCAPI_SLOT_LINK_WAIT); /* fall-through */ case OCAPI_SLOT_LINK_WAIT: - reg = get_odl_status(chip_id, dev->index); + reg = get_odl_status(chip_id, dev->brick_index); if (GETFIELD(OB_ODL_STATUS_TRAINING_STATE_MACHINE, reg) == OCAPI_LINK_STATE_TRAINED) { OCAPIINF(dev, "link trained in %lld ms\n", @@ -1153,9 +1153,9 @@ static int64_t npu2_opencapi_freset(struct pci_slot *slot) if (dev->train_need_fence) { OCAPIDBG(dev, "Fencing OTL during reset\n"); set_fence_control(chip_id, dev->npu->xscom_base, - dev->index, 0b11); + dev->brick_index, 0b11); npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, - PPC_BIT(dev->index + 6)); + PPC_BIT(dev->brick_index + 6)); dev->train_fenced = true; } dev->train_need_fence = true; @@ -1180,9 +1180,10 @@ static int64_t npu2_opencapi_freset(struct pci_slot *slot) case OCAPI_SLOT_FRESET_DEASSERT_DELAY: if (dev->train_fenced) { OCAPIDBG(dev, "Unfencing OTL after reset\n"); - npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, PPC_BIT(dev->index)); + npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, + PPC_BIT(dev->brick_index)); set_fence_control(chip_id, dev->npu->xscom_base, - dev->index, 0b00); + dev->brick_index, 0b00); dev->train_fenced = false; } @@ -1263,7 +1264,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, return rc; genid_base = dev->bars[1].npu2_bar.base + - (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | @@ -1321,7 +1322,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, return rc; genid_base = dev->bars[1].npu2_bar.base + - (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | @@ -1409,13 +1410,14 @@ static int64_t npu2_opencapi_set_pe(struct phb *phb, p = dev->npu; pe_bdfn = dev->bdfn; - + val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, pe_bdfn); reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); - p->bdf2pe_cache[dev->index] = val; + NPU2_MISC_BRICK0_BDF2PE_MAP0 + + (dev->brick_index * 0x18)); + p->bdf2pe_cache[dev->brick_index] = val; npu2_write(p, reg, val); return OPAL_SUCCESS; @@ -1426,8 +1428,8 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, { uint32_t irq; struct npu2_dev *dev = phb_to_npu2_dev_ocapi(phb); - uint64_t block = index_to_block(dev->index); - uint64_t stacku = index_to_stacku(dev->index); + uint64_t block = index_to_block(dev->brick_index); + uint64_t stacku = index_to_stacku(dev->brick_index); uint64_t dsisr, dar, tfc, handle; /* @@ -1631,7 +1633,8 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, dev->phb_ocapi.ops = &npu2_opencapi_ops; dev->phb_ocapi.phb_type = phb_type_npu_v2_opencapi; dev->phb_ocapi.scan_map = 0; - dev->index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + dev->link_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + dev->brick_index = dev->link_index; dev->pl_xscom_base = dt_prop_get_u64(dn_link, "ibm,npu-phy"); dev->lane_mask = dt_prop_get_u32(dn_link, "ibm,npu-lane-mask"); dev->link_speed = dt_prop_get_u64(dn_link, "ibm,link-speed"); @@ -1664,7 +1667,7 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, /* Procedure 13.1.3.9 - AFU Config BARs */ setup_afu_config_bars(n->chip_id, n->xscom_base, dev); - set_fence_control(n->chip_id, n->xscom_base, dev->index, 0b00); + set_fence_control(n->chip_id, n->xscom_base, dev->brick_index, 0b00); if (npu2_ocapi_training_state != NPU2_TRAIN_DEFAULT) { setup_debug_training_state(dev); @@ -1847,8 +1850,8 @@ static int64_t opal_npu_spa_setup(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); if (block == NPU2_BLOCK_OTL1) offset = NPU2_XSL_PSL_SPAP_A1; else @@ -1912,8 +1915,8 @@ static int64_t opal_npu_spa_clear_cache(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); cc_inv = NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_PSL_LLCMD_A0); lock(&dev->npu->lock); @@ -1987,8 +1990,8 @@ static int64_t opal_npu_tl_set(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); /* * The 'capabilities' argument defines what TL template the * device can receive. OpenCAPI 3.0 and 4.0 define 64 templates, so diff --git a/hw/npu2.c b/hw/npu2.c index 2b3540583ac1..be1c304420fc 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1075,10 +1075,10 @@ static int64_t npu2_set_pe(struct phb *phb, val = SETFIELD(NPU2_CQ_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); if (!NPU2DEV_BRICK(dev)) - reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->index/2, + reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->brick_index/2, NPU2_BLOCK_CTL, NPU2_CQ_BRICK0_BDF2PE_MAP0); else - reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->index/2, + reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->brick_index/2, NPU2_BLOCK_CTL, NPU2_CQ_BRICK1_BDF2PE_MAP0); npu2_write(p, reg, val); @@ -1086,8 +1086,8 @@ static int64_t npu2_set_pe(struct phb *phb, val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); - p->bdf2pe_cache[dev->index] = val; + NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->brick_index * 0x18)); + p->bdf2pe_cache[dev->brick_index] = val; npu2_write(p, reg, val); return OPAL_SUCCESS; @@ -1601,7 +1601,7 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, NULL); /* Link index */ - PCI_VIRT_CFG_INIT_RO(pvd, start + 0xc, 1, dev->index); + PCI_VIRT_CFG_INIT_RO(pvd, start + 0xc, 1, dev->link_index); return start + VENDOR_CAP_LEN; } @@ -1725,7 +1725,8 @@ static void npu2_populate_devices(struct npu2 *p, dev->type = NPU2_DEV_TYPE_NVLINK; dev->npu = p; dev->dt_node = link; - dev->index = dt_prop_get_u32(link, "ibm,npu-link-index"); + dev->link_index = dt_prop_get_u32(link, "ibm,npu-link-index"); + dev->brick_index = dev->link_index; group_id = dt_prop_get_u32(link, "ibm,npu-group-id"); dev->bdfn = npu_allocate_bdfn(p, group_id); @@ -1742,7 +1743,7 @@ static void npu2_populate_devices(struct npu2 *p, stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); npu2_bar = &dev->bars[0].npu2_bar; npu2_bar->type = NPU_NTL; - npu2_bar->index = dev->index; + npu2_bar->index = dev->brick_index; npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? NPU2_NTL0_BAR : NPU2_NTL1_BAR); npu2_get_bar(p->chip_id, npu2_bar); @@ -2248,8 +2249,10 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, goto out; } - xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_STACK, xts_bdf_lpar, 0x4 >> (ndev->index / 2)); - xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BRICK, xts_bdf_lpar, (ndev->index % 2)); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_STACK, xts_bdf_lpar, + 0x4 >> (ndev->brick_index / 2)); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BRICK, xts_bdf_lpar, + (ndev->brick_index % 2)); NPU2DBG(p, "XTS_BDF_MAP[%03d] = 0x%08llx\n", id, xts_bdf_lpar); npu2_write(p, NPU2_XTS_BDF_MAP + id*8, xts_bdf_lpar); diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 6bd77e48bf4f..8c1ba5fff37a 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -44,19 +44,19 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, (((stack) << 20) | ((block) << 16) | (offset)) #define NPU2_NTL_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->index >> 1), \ - NPU2_BLOCK_NTL0 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->brick_index >> 1), \ + NPU2_BLOCK_NTL0 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_NTLU_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0U + ((ndev)->index >> 1), \ - NPU2_BLOCK_NTL0 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(NPU2_STACK_STCK_0U + ((ndev)->brick_index >> 1), \ + NPU2_BLOCK_NTL0 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_DL_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(((ndev)->index >> 1), \ - 8 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(((ndev)->brick_index >> 1), \ + 8 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_SM_REG_OFFSET(ndev, sm, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->index >> 1), \ + NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->brick_index >> 1), \ NPU2_BLOCK_SM_0 + (sm), offset) /* Get the offset for this register */ diff --git a/include/npu2.h b/include/npu2.h index 4c2e20e0e2f7..10742031ec0f 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -47,10 +47,10 @@ #define NPU2_DEV_DL_RESET 0x2 /* Return the stack (0-2) of a device */ -#define NPU2DEV_STACK(ndev) ((ndev)->index / 2) +#define NPU2DEV_STACK(ndev) ((ndev)->brick_index / 2) /* Return the brick number (0-1) within a stack */ -#define NPU2DEV_BRICK(ndev) ((ndev)->index % 2) +#define NPU2DEV_BRICK(ndev) ((ndev)->brick_index % 2) /* This represents the state of the actual hardware BARs not the * emulated PCIe BARs. The is a subtle difference between the two as @@ -111,7 +111,8 @@ struct npu2_dev_nvlink { struct npu2_dev { enum npu2_dev_type type; - uint32_t index; + uint32_t link_index; + uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; struct npu2_pcie_bar bars[2]; From patchwork Mon Aug 27 08:55:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962385 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQmq3TBfz9s2P for ; Mon, 27 Aug 2018 18:59:23 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQmq26jDzDrS7 for ; Mon, 27 Aug 2018 18:59:23 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj954qGzDrJ7 for ; Mon, 27 Aug 2018 18:56:13 +1000 (AEST) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8mnbr106175 for ; Mon, 27 Aug 2018 04:56:11 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m4dmwsbg8-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:10 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:08 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:05 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u4Da46465056 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:04 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 14AB1AE05A; Mon, 27 Aug 2018 11:55:37 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 27EADAE045; Mon, 27 Aug 2018 11:55:36 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:55:36 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 8E1DFA034A; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:52 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-0008-0000-0000-0000026729ED X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-0009-0000-0000-000021CF7CBD Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 3/6] hw/npu2: Common NPU2 init routine between NVLink and OpenCAPI X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Replace probe_npu2() and probe_npu2_opencapi() with a new shared probe_npu2(). Refactor some of the common NPU setup code into shared code. No functional change. This patch does not implement support for using both types of devices simultaneously on the same NPU - we expect to add this sometime in the future. Signed-off-by: Andrew Donnellan --- v1->v2: - cleaned up unneeded null check (Alistair) --- core/init.c | 2 +- hw/npu2-common.c | 118 +++++++++++++++++++++++- hw/npu2-opencapi.c | 169 +++++++++++---------------------- hw/npu2.c | 236 ++++++++++++++-------------------------------- include/npu2.h | 6 +- include/skiboot.h | 1 +- 6 files changed, 257 insertions(+), 275 deletions(-) diff --git a/core/init.c b/core/init.c index ca6c468c3941..9095981e67e0 100644 --- a/core/init.c +++ b/core/init.c @@ -1126,8 +1126,6 @@ void __noreturn __nomcount main_cpu_entry(const void *fdt) /* Probe NPUs */ probe_npu(); probe_npu2(); - /* TODO: Eventually, we'll do NVLink and OpenCAPI together */ - probe_npu2_opencapi(); /* Initialize PCI */ pci_init_slots(); diff --git a/hw/npu2-common.c b/hw/npu2-common.c index d076b4906fcc..7346e704fc93 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -20,6 +20,7 @@ #include #include #include +#include enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link) { @@ -107,3 +108,120 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, (uint64_t)new_val << 32); } + +static struct npu2 *setup_npu(struct dt_node *dn) +{ + struct npu2 *npu; + struct npu2_dev *dev; + struct dt_node *np; + uint32_t num_links; + void *npumem; + char *path; + int gcid; + struct proc_chip *chip; + int i = 0; + + /* Retrieve chip ID */ + path = dt_get_path(dn); + gcid = dt_get_chip_id(dn); + chip = get_chip(gcid); + assert(chip); + + num_links = dt_prop_get_u32(dn, "ibm,npu-links"); + npumem = zalloc(sizeof(struct npu2) + num_links * + sizeof(struct npu2_dev)); + assert(npumem); + npu = npumem; + + npu->dt_node = dn; + npu->index = dt_prop_get_u32(dn, "ibm,npu-index"); + npu->chip_id = gcid; + npu->xscom_base = dt_get_address(dn, 0, NULL); + npu->phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); + npu->devices = npumem + sizeof(struct npu2); + + dt_for_each_compatible(dn, np, "ibm,npu-link") { + dev = &npu->devices[i]; + dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); + /* May be overridden by platform presence detection */ + dev->brick_index = dev->link_index; + dev->type = npu2_dt_link_dev_type(np); + dev->npu = npu; + dev->dt_node = np; + dev->pl_xscom_base = dt_prop_get_u64(np, "ibm,npu-phy"); + dev->lane_mask = dt_prop_get_u32(np, "ibm,npu-lane-mask"); + dev->link_speed = dt_prop_get_u64(np, "ibm,link-speed"); + i++; + }; + npu->total_devices = i; + + prlog(PR_INFO, "NPU: Chip %d Found NPU2#%d (%d links) at %s\n", + npu->chip_id, npu->index, npu->total_devices, path); + prlog(PR_INFO, " SCOM Base: %08llx\n", npu->xscom_base); + free(path); + return npu; +} + +static void setup_devices(struct npu2 *npu) +{ + bool nvlink_detected = false, ocapi_detected = false; + struct npu2_dev *dev; + + /* + * TODO: In future, we'll do brick configuration here to support mixed + * setups. + */ + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + switch (dev->type) { + case NPU2_DEV_TYPE_NVLINK: + nvlink_detected = true; + break; + case NPU2_DEV_TYPE_OPENCAPI: + ocapi_detected = true; + break; + default: + prlog(PR_INFO, "NPU: Link %d device not present\n", + npu->devices[i].link_index); + } + } + + if (nvlink_detected && ocapi_detected) { + prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported, aborting NPU init\n"); + return; + } + + if (nvlink_detected) + npu2_nvlink_init_npu(npu); + else if (ocapi_detected) + npu2_opencapi_init_npu(npu); +} + +void probe_npu2(void) +{ + struct proc_chip *chip = next_chip(NULL); + struct npu2 *npu; + struct dt_node *np; + const char *zcal; + + /* Abort if we're running on DD1 */ + if (chip && + (chip->type == PROC_CHIP_P9_NIMBUS || + chip->type == PROC_CHIP_P9_CUMULUS) && + (chip->ec_level & 0xf0) == 0x10) { + prlog(PR_INFO, "NPU2: DD1 not supported\n"); + return; + } + + /* Check for a zcal override */ + zcal = nvram_query("nv_zcal_override"); + if (zcal) { + nv_zcal_nominal = atoi(zcal); + prlog(PR_WARNING, "NPU2: Using ZCAL impedance override = %d\n", nv_zcal_nominal); + } + + dt_for_each_compatible(dt_root, np, "ibm,power9-npu") { + npu = setup_npu(np); + setup_devices(npu); + } +} diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 68ae93a2f836..ead8ec011341 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1446,11 +1446,14 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, * Add the addresses of the registers needed by the OS to handle * faults. The OS accesses them by mmio. */ - dsisr = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_DSISR(stacku, block); - dar = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_DAR(stacku, block); - tfc = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_TFC(stacku, block); - handle = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_PEHANDLE(stacku, - block); + dsisr = (uint64_t) dev->npu->ocapi_global_mmio_base + + NPU2_OTL_OSL_DSISR(stacku, block); + dar = (uint64_t) dev->npu->ocapi_global_mmio_base + + NPU2_OTL_OSL_DAR(stacku, block); + tfc = (uint64_t) dev->npu->ocapi_global_mmio_base + + NPU2_OTL_OSL_TFC(stacku, block); + handle = (uint64_t) dev->npu->ocapi_global_mmio_base + + NPU2_OTL_OSL_PEHANDLE(stacku, block); dt_add_property_cells(pd->dn, "ibm,opal-xsl-irq", irq); dt_add_property_cells(pd->dn, "ibm,opal-xsl-mmio", hi32(dsisr), lo32(dsisr), @@ -1576,20 +1579,15 @@ static void setup_debug_training_state(struct npu2_dev *dev) } } -static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, - struct npu2_dev *dev) +static void setup_device(struct npu2_dev *dev) { - uint32_t dev_index, npu_index; struct dt_node *dn_phb, *dn; struct pci_slot *slot; char port_name[17]; uint64_t mm_win[2]; - dev_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); - npu_index = dt_prop_get_u32(n->dt_node, "ibm,npu-index"); - /* Populate PHB device node */ - phys_map_get(n->chip_id, NPU_OCAPI_MMIO, dev_index, &mm_win[0], + phys_map_get(dev->npu->chip_id, NPU_OCAPI_MMIO, dev->brick_index, &mm_win[0], &mm_win[1]); prlog(PR_DEBUG, "OCAPI: Setting MMIO window to %016llx + %016llx\n", mm_win[0], mm_win[1]); @@ -1609,40 +1607,28 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, dt_add_property_strings(dn_phb, "device_type", "pciex"); dt_add_property(dn_phb, "reg", mm_win, sizeof(mm_win)); - dt_add_property_cells(dn_phb, "ibm,npu-index", npu_index); - dt_add_property_cells(dn_phb, "ibm,chip-id", n->chip_id); - dt_add_property_cells(dn_phb, "ibm,xscom-base", n->xscom_base); - dt_add_property_cells(dn_phb, "ibm,npcq", n->dt_node->phandle); + dt_add_property_cells(dn_phb, "ibm,npu-index", dev->npu->index); + dt_add_property_cells(dn_phb, "ibm,chip-id", dev->npu->chip_id); + dt_add_property_cells(dn_phb, "ibm,xscom-base", dev->npu->xscom_base); + dt_add_property_cells(dn_phb, "ibm,npcq", dev->npu->dt_node->phandle); dt_add_property_cells(dn_phb, "ibm,links", 1); dt_add_property(dn_phb, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(dn_phb, "ibm,phb-diag-data-size", 0); dt_add_property_cells(dn_phb, "ibm,opal-num-pes", NPU2_MAX_PE_NUM); - n->mm_base = mm_win[0]; - n->mm_size = mm_win[1]; - dt_add_property_cells(dn_phb, "ranges", 0x02000000, - hi32(n->mm_base), lo32(n->mm_base), - hi32(n->mm_base), lo32(n->mm_base), - hi32(n->mm_size), lo32(n->mm_size)); + hi32(mm_win[0]), lo32(mm_win[0]), + hi32(mm_win[0]), lo32(mm_win[0]), + hi32(mm_win[1]), lo32(mm_win[1])); - dev->type = NPU2_DEV_TYPE_OPENCAPI; - dev->npu = n; - dev->dt_node = dn_link; dev->phb_ocapi.dt_node = dn_phb; dev->phb_ocapi.ops = &npu2_opencapi_ops; dev->phb_ocapi.phb_type = phb_type_npu_v2_opencapi; dev->phb_ocapi.scan_map = 0; - dev->link_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); - dev->brick_index = dev->link_index; - dev->pl_xscom_base = dt_prop_get_u64(dn_link, "ibm,npu-phy"); - dev->lane_mask = dt_prop_get_u32(dn_link, "ibm,npu-lane-mask"); - dev->link_speed = dt_prop_get_u64(dn_link, "ibm,link-speed"); + dev->bdfn = 0; dev->train_need_fence = false; dev->train_fenced = false; - n->total_devices++; - /* Find I2C port for handling device reset */ snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", dev->npu->chip_id, platform.ocapi->i2c_engine, @@ -1663,11 +1649,11 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, /* TODO: Procedure 13.1.3.7 - AFU Memory Range BARs */ /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ - setup_afu_mmio_bars(n->chip_id, n->xscom_base, dev); + setup_afu_mmio_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); /* Procedure 13.1.3.9 - AFU Config BARs */ - setup_afu_config_bars(n->chip_id, n->xscom_base, dev); + setup_afu_config_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); - set_fence_control(n->chip_id, n->xscom_base, dev->brick_index, 0b00); + set_fence_control(dev->npu->chip_id, dev->npu->xscom_base, dev->brick_index, 0b00); if (npu2_ocapi_training_state != NPU2_TRAIN_DEFAULT) { setup_debug_training_state(dev); @@ -1689,107 +1675,72 @@ failed: return; } -static void npu2_opencapi_probe(struct dt_node *dn) +static void read_nvram_training_state(void) { - struct dt_node *link; - char *path; - uint32_t gcid, index, links, scom_base; - uint64_t reg[2]; - uint64_t dev_index; - struct npu2 *n; - int rc, i = 0; - - gcid = dt_get_chip_id(dn); - index = dt_prop_get_u32(dn, "ibm,npu-index"); - links = dt_prop_get_u32(dn, "ibm,npu-links"); - - /* Don't try to init when we have an NVLink link */ - dt_for_each_compatible(dn, link, "ibm,npu-link") { - if (npu2_dt_link_dev_type(link) != NPU2_DEV_TYPE_OPENCAPI) { - prlog(PR_DEBUG, - "OCAPI: NPU%d: Non-OpenCAPI link found, skipping OpenCAPI init\n", - index); - return; - } + const char *state; + + state = nvram_query("opencapi-link-training"); + if (state) { + if (!strcmp(state, "prbs31")) + npu2_ocapi_training_state = NPU2_TRAIN_PRBS31; + else if (!strcmp(state, "none")) + npu2_ocapi_training_state = NPU2_TRAIN_NONE; + else + prlog(PR_WARNING, + "OCAPI: invalid training state in NVRAM: %s\n", + state); } +} - path = dt_get_path(dn); - prlog(PR_INFO, "OCAPI: Chip %d Found OpenCAPI NPU%d (%d links) at %s\n", - gcid, index, links, path); - free(path); +int npu2_opencapi_init_npu(struct npu2 *npu) +{ + struct npu2_dev *dev; + uint64_t reg[2]; + int rc; assert(platform.ocapi); + read_nvram_training_state(); /* TODO: Test OpenCAPI with fast reboot and make it work */ disable_fast_reboot("OpenCAPI device enabled"); - scom_base = dt_get_address(dn, 0, NULL); - prlog(PR_INFO, "OCAPI: SCOM Base: %08x\n", scom_base); + setup_global_mmio_bar(npu->chip_id, npu->xscom_base, reg); - setup_global_mmio_bar(gcid, scom_base, reg); + npu->ocapi_global_mmio_base = (void *)reg[0]; - n = zalloc(sizeof(struct npu2) + links * sizeof(struct npu2_dev)); - n->devices = (struct npu2_dev *)(n + 1); - n->chip_id = gcid; - n->xscom_base = scom_base; - n->regs = (void *)reg[0]; - n->dt_node = dn; + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; - dt_for_each_compatible(dn, link, "ibm,npu-link") { - dev_index = dt_prop_get_u32(link, "ibm,npu-link-index"); - prlog(PR_INFO, "OCAPI: Configuring link index %lld\n", - dev_index); + prlog(PR_INFO, "OCAPI: Configuring link index %d, brick %d\n", + dev->link_index, dev->brick_index); /* Procedure 13.1.3.1 - Select OCAPI vs NVLink */ - brick_config(gcid, scom_base, dev_index); + brick_config(npu->chip_id, npu->xscom_base, dev->brick_index); /* Procedure 13.1.3.5 - Transaction Layer Configuration */ - tl_config(gcid, scom_base, dev_index); + tl_config(npu->chip_id, npu->xscom_base, dev->brick_index); /* Procedure 13.1.3.6 - Address Translation Configuration */ - address_translation_config(gcid, scom_base, dev_index); + address_translation_config(npu->chip_id, npu->xscom_base, dev->brick_index); } /* Procedure 13.1.3.10 - Interrupt Configuration */ - rc = setup_irq(n); + rc = setup_irq(npu); if (rc) goto failed; - dt_for_each_compatible(dn, link, "ibm,npu-link") { - npu2_opencapi_setup_device(link, n, &n->devices[i]); - i++; + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + setup_device(dev); } - return; + return 0; failed: - free(n); -} - -static void read_nvram_training_state(void) -{ - const char *state; - - state = nvram_query("opencapi-link-training"); - if (state) { - if (!strcmp(state, "prbs31")) - npu2_ocapi_training_state = NPU2_TRAIN_PRBS31; - else if (!strcmp(state, "none")) - npu2_ocapi_training_state = NPU2_TRAIN_NONE; - else - prlog(PR_WARNING, - "OCAPI: invalid training state in NVRAM: %s\n", - state); - } -} - -void probe_npu2_opencapi(void) -{ - struct dt_node *np_npu; - - read_nvram_training_state(); - - dt_for_each_compatible(dt_root, np_npu, "ibm,power9-npu") - npu2_opencapi_probe(np_npu); + return -1; } static const struct phb_ops npu2_opencapi_ops = { diff --git a/hw/npu2.c b/hw/npu2.c index be1c304420fc..419dd37a24b1 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1357,44 +1357,13 @@ static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint } /* - * Probe NPU2 device node and create PCI root device node - * accordingly. The NPU2 device node should specify number - * of links and xscom base address to access links. + * Set up NPU for NVLink and create PCI root device node + * accordingly. */ -static void npu2_probe_phb(struct dt_node *dn) +int npu2_nvlink_init_npu(struct npu2 *npu) { - struct proc_chip *proc_chip; - struct dt_node *np, *link; - bool ocapi_detected = false, nvlink_detected = false; - uint32_t gcid, scom, index, phb_index, links; + struct dt_node *np; uint64_t reg[2], mm_win[2], val; - char *path; - - /* Abort if any OpenCAPI links detected */ - dt_for_each_compatible(dn, link, "ibm,npu-link") { - if (npu2_dt_link_dev_type(link) == NPU2_DEV_TYPE_OPENCAPI) - ocapi_detected = true; - else - nvlink_detected = true; - } - - if (ocapi_detected && nvlink_detected) { - prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported\n"); - return; - } else if (ocapi_detected) { - prlog(PR_INFO, "NPU: OpenCAPI link configuration detected, not initialising NVLink\n"); - return; - } - - /* Retrieve chip id */ - path = dt_get_path(dn); - gcid = dt_get_chip_id(dn); - proc_chip = get_chip(gcid); - assert(proc_chip); - if ((proc_chip->ec_level & 0xf0) > 0x20) { - prerror("NPU: unsupported ec level on Chip 0x%x!\n", gcid); - return; - } /* TODO: Clean this up with register names, etc. when we get * time. This just turns NVLink mode on in each brick and should @@ -1403,64 +1372,56 @@ static void npu2_probe_phb(struct dt_node *dn) * * Obviously if the year is now 2020 that didn't happen and you * should fix this :-) */ - xscom_write_mask(gcid, 0x5011000, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011030, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011060, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011090, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011200, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011230, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011260, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011290, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011400, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011430, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011460, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011490, PPC_BIT(58), PPC_BIT(58)); - - xscom_write_mask(gcid, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(gcid, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(gcid, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x5011000, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011030, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011060, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011090, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011200, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011230, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011260, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011290, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011400, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011430, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011460, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011490, PPC_BIT(58), PPC_BIT(58)); + + xscom_write_mask(npu->chip_id, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); val = NPU2_NTL_MISC_CFG2_BRICK_ENABLE | NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA | NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA | NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; - xscom_write_mask(gcid, 0x5011110, val, val); - xscom_write_mask(gcid, 0x5011130, val, val); - xscom_write_mask(gcid, 0x5011310, val, val); - xscom_write_mask(gcid, 0x5011330, val, val); - xscom_write_mask(gcid, 0x5011510, val, val); - xscom_write_mask(gcid, 0x5011530, val, val); + xscom_write_mask(npu->chip_id, 0x5011110, val, val); + xscom_write_mask(npu->chip_id, 0x5011130, val, val); + xscom_write_mask(npu->chip_id, 0x5011310, val, val); + xscom_write_mask(npu->chip_id, 0x5011330, val, val); + xscom_write_mask(npu->chip_id, 0x5011510, val, val); + xscom_write_mask(npu->chip_id, 0x5011530, val, val); val = PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); - xscom_write_mask(gcid, 0x5011009, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011039, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011069, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011099, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011209, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011239, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011269, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011299, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011409, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011439, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011469, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011499, val, PPC_BITMASK(6,11)); - - index = dt_prop_get_u32(dn, "ibm,npu-index"); - phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); - links = dt_prop_get_u32(dn, "ibm,npu-links"); - prlog(PR_INFO, "NPU: Chip %d Found NPU2#%d (%d links) at %s\n", - gcid, index, links, path); - free(path); - - /* Retrieve scom base address */ - scom = dt_get_address(dn, 0, NULL); - prlog(PR_INFO, " SCOM Base: %08x\n", scom); + xscom_write_mask(npu->chip_id, 0x5011009, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011039, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011069, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011099, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011209, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011239, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011269, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011299, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011409, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011439, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); /* Reassign the BARs */ - assign_mmio_bars(gcid, scom, reg, mm_win); + assign_mmio_bars(npu->chip_id, npu->xscom_base, reg, mm_win); + npu->regs = (void *)reg[0]; + npu->mm_base = mm_win[0]; + npu->mm_size = mm_win[1]; if (reg[0] && reg[1]) prlog(PR_INFO, " Global MMIO BAR: %016llx (%lldMB)\n", @@ -1477,17 +1438,21 @@ static void npu2_probe_phb(struct dt_node *dn) "ibm,ioda2-npu2-phb"); dt_add_property_strings(np, "device_type", "pciex"); dt_add_property(np, "reg", reg, sizeof(reg)); - dt_add_property_cells(np, "ibm,phb-index", phb_index); - dt_add_property_cells(np, "ibm,npu-index", index); - dt_add_property_cells(np, "ibm,chip-id", gcid); - dt_add_property_cells(np, "ibm,xscom-base", scom); - dt_add_property_cells(np, "ibm,npcq", dn->phandle); - dt_add_property_cells(np, "ibm,links", links); + dt_add_property_cells(np, "ibm,phb-index", npu->phb_index); + dt_add_property_cells(np, "ibm,npu-index", npu->index); + dt_add_property_cells(np, "ibm,chip-id", npu->chip_id); + dt_add_property_cells(np, "ibm,xscom-base", npu->xscom_base); + dt_add_property_cells(np, "ibm,npcq", npu->dt_node->phandle); + dt_add_property_cells(np, "ibm,links", npu->total_devices); dt_add_property(np, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(np, "ibm,phb-diag-data-size", 0); /* Disable fast reboot - not currently supported */ disable_fast_reboot("NVLink device enabled"); + + npu2_nvlink_create_phb(npu, np); + + return 0; } static uint32_t npu2_populate_pcie_cap(struct npu2_dev *dev, @@ -1949,49 +1914,25 @@ static void npu2_setup_irqs(struct npu2 *p) npu2_write(p, reg, val); } -static void npu2_create_phb(struct dt_node *dn) +void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) { - const struct dt_property *prop; - struct npu2 *p; struct pci_slot *slot; - uint32_t links; - void *pmem; - - /* Retrieve number of devices */ - links = dt_prop_get_u32(dn, "ibm,links"); - pmem = zalloc(sizeof(struct npu2) + links * sizeof(struct npu2_dev)); - assert(pmem); - - /* Populate PHB */ - p = pmem; - p->index = dt_prop_get_u32(dn, "ibm,phb-index"); - p->chip_id = dt_prop_get_u32(dn, "ibm,chip-id"); - p->xscom_base = dt_prop_get_u32(dn, "ibm,xscom-base"); - p->total_devices = links; - p->regs = (void *)dt_get_address(dn, 0, NULL); - - prop = dt_require_property(dn, "ibm,mmio-window", -1); - assert(prop->len >= (2 * sizeof(uint64_t))); - p->mm_base = ((const uint64_t *)prop->prop)[0]; - p->mm_size = ((const uint64_t *)prop->prop)[1]; - - p->devices = pmem + sizeof(struct npu2); /* Generic PHB */ - p->phb_nvlink.dt_node = dn; - p->phb_nvlink.ops = &npu_ops; - p->phb_nvlink.phb_type = phb_type_npu_v2; - init_lock(&p->lock); - init_lock(&p->phb_nvlink.lock); - list_head_init(&p->phb_nvlink.devices); - list_head_init(&p->phb_nvlink.virt_devices); - - npu2_setup_irqs(p); - npu2_populate_devices(p, dn); - npu2_add_interrupt_map(p, dn); - npu2_add_phb_properties(p); - - slot = npu2_slot_create(&p->phb_nvlink); + npu->phb_nvlink.dt_node = dn; + npu->phb_nvlink.ops = &npu_ops; + npu->phb_nvlink.phb_type = phb_type_npu_v2; + init_lock(&npu->lock); + init_lock(&npu->phb_nvlink.lock); + list_head_init(&npu->phb_nvlink.devices); + list_head_init(&npu->phb_nvlink.virt_devices); + + npu2_setup_irqs(npu); + npu2_populate_devices(npu, dn); + npu2_add_interrupt_map(npu, dn); + npu2_add_phb_properties(npu); + + slot = npu2_slot_create(&npu->phb_nvlink); if (!slot) { /** @@ -2002,41 +1943,10 @@ static void npu2_create_phb(struct dt_node *dn) prlog(PR_ERR, "NPU: Cannot create PHB slot\n"); } - pci_register_phb(&p->phb_nvlink, OPAL_DYNAMIC_PHB_ID); - - npu2_init_ioda_cache(p); - npu2_hw_init(p); -} - -void probe_npu2(void) -{ - struct proc_chip *chip = next_chip(NULL); - struct dt_node *np; - const char *zcal; - - /* Abort if we're running on DD1 */ - if (chip && - (chip->type == PROC_CHIP_P9_NIMBUS || - chip->type == PROC_CHIP_P9_CUMULUS) && - (chip->ec_level & 0xf0) == 0x10) { - prlog(PR_INFO, "NPU: DD1 not supported\n"); - return; - } - - /* Check for a zcal override */ - zcal = nvram_query("nv_zcal_override"); - if (zcal) { - nv_zcal_nominal = atoi(zcal); - prlog(PR_WARNING, "NPU: Using ZCAL impedance override = %d\n", nv_zcal_nominal); - } - - /* Scan NPU2 XSCOM nodes */ - dt_for_each_compatible(dt_root, np, "ibm,power9-npu") - npu2_probe_phb(np); + pci_register_phb(&npu->phb_nvlink, OPAL_DYNAMIC_PHB_ID); - /* Scan newly created PHB nodes */ - dt_for_each_compatible(dt_root, np, "ibm,power9-npu-pciex") - npu2_create_phb(np); + npu2_init_ioda_cache(npu); + npu2_hw_init(npu); } /* diff --git a/include/npu2.h b/include/npu2.h index 10742031ec0f..9fdc438bb4c1 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -147,6 +147,7 @@ struct npu2 { uint32_t chip_id; uint64_t xscom_base; void *regs; + void *ocapi_global_mmio_base; uint64_t mm_base; uint64_t mm_size; uint32_t base_lsi; @@ -167,6 +168,7 @@ struct npu2 { /* NVLink */ struct phb phb_nvlink; + uint32_t phb_index; }; static inline struct npu2 *phb_to_npu2_nvlink(struct phb *phb) @@ -193,6 +195,10 @@ static inline struct phb *npu2_dev_to_phb(struct npu2_dev *ndev) } } +int npu2_opencapi_init_npu(struct npu2 *npu); +int npu2_nvlink_init_npu(struct npu2 *npu); +void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn); + enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link); void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val); uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg); diff --git a/include/skiboot.h b/include/skiboot.h index bba76c12c39e..f2818b1a2326 100644 --- a/include/skiboot.h +++ b/include/skiboot.h @@ -201,7 +201,6 @@ extern int preload_capp_ucode(void); extern void preload_io_vpd(void); extern void probe_npu(void); extern void probe_npu2(void); -extern void probe_npu2_opencapi(void); extern void uart_init(void); extern void mbox_init(void); extern void early_uart_init(void); From patchwork Mon Aug 27 08:55:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962382 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQm16p7xz9s3Z for ; Mon, 27 Aug 2018 18:58:41 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQm14Lh7zDrS7 for ; Mon, 27 Aug 2018 18:58:41 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj84ktSzDrJ7 for ; Mon, 27 Aug 2018 18:56:12 +1000 (AEST) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8mnew097418 for ; Mon, 27 Aug 2018 04:56:10 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m4ccvvmph-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:09 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:08 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:05 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u4mK38600886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:04 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1596DAE05F; Mon, 27 Aug 2018 11:55:37 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 283D4AE051; Mon, 27 Aug 2018 11:55:36 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:55:36 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 942A6A037F; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:53 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-0020-0000-0000-000002BC2B4E X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-0021-0000-0000-0000210995F6 Message-Id: <8ab5f180a2dd766f7f2f55e42e06e0740bf7f551.1535359753.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 4/6] hw/npu2, platform: Add NPU2 platform device detection callback X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" There is no standardised way to determine the presence and type of devices connected to an NPU on POWER9. Currently, we hardcode device types based on platform type (as no platform currently supports both OpenCAPI and NVLink), and for OpenCAPI platforms we use I2C to detect presence. Witherspoon (and potentially other platforms later on) supports both NVLink and OpenCAPI, and additionally uses SXM2 connectors which can carry more than one link, rather than the SlimSAS connectors used for OpenCAPI on Zaius and ZZ. This necessitates some special handling. Add a platform callback for NPU device detection. In a later patch, we will use this to implement Witherspoon-specific device detection. For now, add a Witherspoon stub that sets all links to NVLink (i.e. current behaviour). Move the existing I2C-based presence detection for OpenCAPI devices on Zaius/ZZ into common code, which we use by default for platforms which do not define a callback. Clean up the use of the ibm,npu-link-type property, which will now be exposed solely for debugging and not consumed internally. Signed-off-by: Andrew Donnellan --- v1->v2: - turn the "generic" i2c detection code into a callback (Alistair) --- core/platform.c | 2 +- hdata/spira.c | 1 +- hw/npu2-common.c | 103 +++++++++++++++++++++++++++++----- hw/npu2-opencapi.c | 85 +--------------------------- include/npu2.h | 14 ++++- include/platform.h | 4 +- platforms/astbmc/witherspoon.c | 10 +++- platforms/astbmc/zaius.c | 3 +- 8 files changed, 125 insertions(+), 97 deletions(-) diff --git a/core/platform.c b/core/platform.c index b32cbf5ce9d1..4b3eaa4bd1d1 100644 --- a/core/platform.c +++ b/core/platform.c @@ -25,6 +25,7 @@ #include #include #include +#include #include bool manufacturing_mode = false; @@ -204,6 +205,7 @@ static struct platform generic_platform = { .start_preload_resource = generic_start_preload_resource, .resource_loaded = generic_resource_loaded, .ocapi = &generic_ocapi, + .npu2_device_detect = npu2_i2c_presence_detect, /* Assumes ZZ */ }; const struct bmc_platform *bmc_platform = &generic_bmc; diff --git a/hdata/spira.c b/hdata/spira.c index c820c4daf26e..189584d5ddb6 100644 --- a/hdata/spira.c +++ b/hdata/spira.c @@ -1490,7 +1490,6 @@ static void add_npu(struct dt_node *xscom, const struct HDIF_array_hdr *links, } dt_add_property_string(node, "compatible", "ibm,npu-link"); - dt_add_property_string(node, "ibm,npu-link-type", "nvlink"); dt_add_property_cells(node, "reg", link_count); dt_add_property_cells(node, "ibm,npu-link-index", link_count); dt_add_property_cells(node, "ibm,workbook-link-id", link_id); diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 7346e704fc93..b87d43469d5b 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -21,19 +21,7 @@ #include #include #include - -enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link) -{ - const char *link_type = dt_prop_get(link, "ibm,npu-link-type") ?: - "unknown"; - if (streq(link_type, "nvlink")) { - return NPU2_DEV_TYPE_NVLINK; - } else if (streq(link_type, "opencapi")) { - return NPU2_DEV_TYPE_OPENCAPI; - } else { - return NPU2_DEV_TYPE_UNKNOWN; - } -} +#include /* * We use the indirect method because it uses the same addresses as @@ -109,12 +97,62 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas (uint64_t)new_val << 32); } +static bool _i2c_presence_detect(struct npu2_dev *dev) +{ + uint8_t state, data; + int rc; + + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, + platform.ocapi->i2c_presence_addr, + SMBUS_READ, 0, 1, + &state, 1, 120); + if (rc) { + OCAPIERR(dev, "error detecting link presence: %d\n", rc); + return true; /* assume link exists */ + } + + OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); + + switch (dev->link_index) { + case 2: + data = platform.ocapi->i2c_presence_odl0; + break; + case 3: + data = platform.ocapi->i2c_presence_odl1; + break; + default: + OCAPIERR(dev, "presence detection on invalid link\n"); + return true; + } + /* Presence detect bits are active low */ + return !(state & data); +} + +/* + * A default presence detection implementation for platforms like ZZ and Zaius + * that don't implement their own. Assumes all devices found will be OpenCAPI. + */ +void npu2_i2c_presence_detect(struct npu2 *npu) +{ + struct npu2_dev *dev; + assert(platform.ocapi); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (platform.ocapi->force_presence || + _i2c_presence_detect(dev)) + dev->type = NPU2_DEV_TYPE_OPENCAPI; + else + dev->type = NPU2_DEV_TYPE_UNKNOWN; + } +} + static struct npu2 *setup_npu(struct dt_node *dn) { struct npu2 *npu; struct npu2_dev *dev; struct dt_node *np; uint32_t num_links; + char port_name[17]; void *npumem; char *path; int gcid; @@ -138,6 +176,27 @@ static struct npu2 *setup_npu(struct dt_node *dn) npu->chip_id = gcid; npu->xscom_base = dt_get_address(dn, 0, NULL); npu->phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); + + if (platform.ocapi) { + /* Find I2C port for handling device presence/reset */ + snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", + gcid, platform.ocapi->i2c_engine, + platform.ocapi->i2c_port); + prlog(PR_DEBUG, "NPU: Looking for I2C port %s\n", port_name); + + dt_for_each_compatible(dt_root, np, "ibm,power9-i2c-port") { + if (streq(port_name, dt_prop_get(np, "ibm,port-name"))) { + npu->i2c_port_id_ocapi = dt_prop_get_u32(np, "ibm,opal-id"); + break; + } + } + + if (!npu->i2c_port_id_ocapi) { + prlog(PR_ERR, "NPU: Couldn't find I2C port %s\n", + port_name); + } + } + npu->devices = npumem + sizeof(struct npu2); dt_for_each_compatible(dn, np, "ibm,npu-link") { @@ -145,7 +204,8 @@ static struct npu2 *setup_npu(struct dt_node *dn) dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); /* May be overridden by platform presence detection */ dev->brick_index = dev->link_index; - dev->type = npu2_dt_link_dev_type(np); + /* Will be overridden by presence detection */ + dev->type = NPU2_DEV_TYPE_UNKNOWN; dev->npu = npu; dev->dt_node = np; dev->pl_xscom_base = dt_prop_get_u64(np, "ibm,npu-phy"); @@ -176,13 +236,22 @@ static void setup_devices(struct npu2 *npu) switch (dev->type) { case NPU2_DEV_TYPE_NVLINK: nvlink_detected = true; + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "nvlink"); break; case NPU2_DEV_TYPE_OPENCAPI: ocapi_detected = true; + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "opencapi"); break; default: prlog(PR_INFO, "NPU: Link %d device not present\n", npu->devices[i].link_index); + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "unknown"); } } @@ -220,8 +289,14 @@ void probe_npu2(void) prlog(PR_WARNING, "NPU2: Using ZCAL impedance override = %d\n", nv_zcal_nominal); } + if (!platform.npu2_device_detect) { + prlog(PR_INFO, "NPU: Platform does not support NPU\n"); + return; + } + dt_for_each_compatible(dt_root, np, "ibm,power9-npu") { npu = setup_npu(np); + platform.npu2_device_detect(npu); setup_devices(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index ead8ec011341..fa28633bb150 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -52,14 +52,6 @@ #include #include -#define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) -#define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) -#define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) - - #define NPU_IRQ_LEVELS 35 #define NPU_IRQ_LEVELS_XSL 23 #define MAX_PE_HANDLE ((1 << 15) - 1) @@ -842,7 +834,7 @@ static void assert_reset(struct npu2_dev *dev) * register and a pin is in output mode if its value is 0 */ data = ~pin; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x3, 1, &data, sizeof(data), 120); @@ -851,7 +843,7 @@ static void assert_reset(struct npu2_dev *dev) /* register 1 controls the signal, reset is active low */ data = ~pin; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x1, 1, &data, sizeof(data), 120); @@ -874,7 +866,7 @@ static void deassert_reset(struct npu2_dev *dev) int rc; data = 0xFF; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x1, 1, &data, sizeof(data), 120); @@ -888,43 +880,6 @@ static void deassert_reset(struct npu2_dev *dev) } } -static bool i2c_presence_detect(struct npu2_dev *dev) -{ - uint8_t state, data; - int rc; - - /* - * Opencapi presence detection is done through i2c - * - * Lagrange platforms (ZZ, Zaius) use the same default mechanism. - * Witherspoon will need a specific implementation, TBD. - */ - rc = i2c_request_send(dev->i2c_port_id_ocapi, - platform.ocapi->i2c_presence_addr, - SMBUS_READ, 0, 1, - &state, 1, 120); - if (rc) { - OCAPIERR(dev, "error detecting link presence: %d\n", rc); - return true; /* assume link exists */ - } - - OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); - - switch (dev->brick_index) { // TODO(ajd): Link or brick index? - case 2: - data = platform.ocapi->i2c_presence_odl0; - break; - case 3: - data = platform.ocapi->i2c_presence_odl1; - break; - default: - OCAPIERR(dev, "presence detection on invalid link\n"); - return true; - } - /* Presence detect bits are active low */ - return !(state & data); -} - static void reset_odl(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; @@ -1015,17 +970,10 @@ static void start_training(uint32_t gcid, struct npu2_dev *dev) xscom_write(gcid, config_xscom, reg); } -static int64_t npu2_opencapi_get_presence_state(struct pci_slot *slot, +static int64_t npu2_opencapi_get_presence_state(struct pci_slot __unused *slot, uint8_t *val) { - bool present; - struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); - - if (platform.ocapi->force_presence) - present = true; - else - present = i2c_presence_detect(dev); - *val = present; + *val = true; return OPAL_SUCCESS; } @@ -1581,9 +1529,8 @@ static void setup_debug_training_state(struct npu2_dev *dev) static void setup_device(struct npu2_dev *dev) { - struct dt_node *dn_phb, *dn; + struct dt_node *dn_phb; struct pci_slot *slot; - char port_name[17]; uint64_t mm_win[2]; /* Populate PHB device node */ @@ -1629,23 +1576,6 @@ static void setup_device(struct npu2_dev *dev) dev->bdfn = 0; dev->train_need_fence = false; dev->train_fenced = false; - /* Find I2C port for handling device reset */ - snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", - dev->npu->chip_id, platform.ocapi->i2c_engine, - platform.ocapi->i2c_port); - prlog(PR_DEBUG, "OCAPI: Looking for I2C port %s\n", port_name); - - dt_for_each_compatible(dt_root, dn, "ibm,power9-i2c-port") { - if (streq(port_name, dt_prop_get(dn, "ibm,port-name"))) { - dev->i2c_port_id_ocapi = dt_prop_get_u32(dn, "ibm,opal-id"); - break; - } - } - - if (!dev->i2c_port_id_ocapi) { - prlog(PR_ERR, "OCAPI: Couldn't find I2C port %s\n", port_name); - goto failed; - } /* TODO: Procedure 13.1.3.7 - AFU Memory Range BARs */ /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ @@ -1670,9 +1600,6 @@ static void setup_device(struct npu2_dev *dev) } pci_register_phb(&dev->phb_ocapi, OPAL_DYNAMIC_PHB_ID); return; -failed: - dt_add_property_string(dn_phb, "status", "error"); - return; } static void read_nvram_training_state(void) diff --git a/include/npu2.h b/include/npu2.h index 9fdc438bb4c1..6d28fd3e58f4 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -17,6 +17,7 @@ #ifndef __NPU2_H #define __NPU2_H +#include #include /* Debugging options */ @@ -36,6 +37,14 @@ #define NPU2DEVINF(p, fmt, a...) NPU2DEVLOG(PR_INFO, p, fmt, ##a) #define NPU2DEVERR(p, fmt, a...) NPU2DEVLOG(PR_ERR, p, fmt, ##a) +#define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) +#define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) +#define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) + + /* Number of PEs supported */ #define NPU2_MAX_PE_NUM 16 #define NPU2_RESERVED_PE_NUM 15 @@ -136,7 +145,6 @@ struct npu2_dev { /* OpenCAPI */ struct phb phb_ocapi; - uint64_t i2c_port_id_ocapi; bool train_need_fence; bool train_fenced; }; @@ -169,6 +177,8 @@ struct npu2 { /* NVLink */ struct phb phb_nvlink; uint32_t phb_index; + + uint64_t i2c_port_id_ocapi; }; static inline struct npu2 *phb_to_npu2_nvlink(struct phb *phb) @@ -195,11 +205,11 @@ static inline struct phb *npu2_dev_to_phb(struct npu2_dev *ndev) } } +void npu2_i2c_presence_detect(struct npu2 *npu); int npu2_opencapi_init_npu(struct npu2 *npu); int npu2_nvlink_init_npu(struct npu2 *npu); void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn); -enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link); void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val); uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg); void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val); diff --git a/include/platform.h b/include/platform.h index 1a35a86a6f8b..efafbd2239b9 100644 --- a/include/platform.h +++ b/include/platform.h @@ -22,6 +22,7 @@ struct phb; struct pci_device; struct pci_slot; struct errorlog; +struct npu2; enum resource_id { RESOURCE_ID_KERNEL, @@ -76,6 +77,9 @@ struct platform { /* OpenCAPI platform-specific I2C information */ const struct platform_ocapi *ocapi; + /* NPU2 device detection */ + void (*npu2_device_detect)(struct npu2 *npu); + /* * Probe platform, return true on a match, called before * any allocation has been performed outside of the heap diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index d663709f8766..ce83ff9701d3 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "astbmc.h" #include "ast.h" @@ -153,6 +154,14 @@ static void witherspoon_pre_pci_fixup(void) phb4_pre_pci_fixup_witherspoon(); } +static void witherspoon_npu2_device_detect(struct npu2 *npu) +{ + /* Stub until we implement real device detection */ + for (int i = 0; i < npu->total_devices; i++) { + npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + } +} + /* The only difference between these is the PCI slot handling */ DECLARE_PLATFORM(witherspoon) = { @@ -170,4 +179,5 @@ DECLARE_PLATFORM(witherspoon) = { .terminate = ipmi_terminate, .pci_get_slot_info = dt_slot_get_slot_info, + .npu2_device_detect = witherspoon_npu2_device_detect, }; diff --git a/platforms/astbmc/zaius.c b/platforms/astbmc/zaius.c index 2f296c3679ac..267e6578bbe7 100644 --- a/platforms/astbmc/zaius.c +++ b/platforms/astbmc/zaius.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "astbmc.h" @@ -51,7 +52,6 @@ static void create_link(struct dt_node *npu, int group, int index) link = dt_new(npu, namebuf); dt_add_property_string(link, "compatible", "ibm,npu-link"); - dt_add_property_string(link, "ibm,npu-link-type", "opencapi"); dt_add_property_cells(link, "ibm,npu-link-index", index); switch (index) { @@ -153,4 +153,5 @@ DECLARE_PLATFORM(zaius) = { .exit = ipmi_wdt_final_reset, .terminate = ipmi_terminate, .ocapi = &zaius_ocapi, + .npu2_device_detect = npu2_i2c_presence_detect, }; From patchwork Mon Aug 27 08:55:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962383 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQmH20dVz9s2P for ; Mon, 27 Aug 2018 18:58:55 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQmH0l4tzF0PW for ; Mon, 27 Aug 2018 18:58:55 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj93tZ3zDrGC for ; Mon, 27 Aug 2018 18:56:13 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8nOpq040814 for ; Mon, 27 Aug 2018 04:56:11 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m4cyhb00j-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:10 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:08 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:06 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u5Rh25296902 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:05 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CD08811C05E; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3423F11C066; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 991D7A03A3; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:54 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-0020-0000-0000-000002BC2B50 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-0021-0000-0000-0000210995F8 Message-Id: <749a8dffca0a81102a9e4289dbb1100e5e394796.1535359753.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 5/6] hw/npu2, platform: Restructure OpenCAPI i2c reset/presence pins X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" In platform_ocapi, we define i2c_{reset,presence}_odl{0,1} to specify the appropriate reset/presence GPIO pins for devices connected to ODL0 and ODL1 respectively. This is obviously wrong, because a device connected to brick 2 and a device connected to brick 4 are going to be different devices connected to different I2C pins, but rather conveniently we haven't had to deal with systems that can use the full 4 bricks as yet. Now that we're adding OpenCAPI support for Witherspoon, we should change this to specify pins separately for all 4 bricks. Replace i2c_{reset,presence}_odl{0,1} with i2c_{reset,presence}_pin{2,3,4,5} and update the presence detection code, device reset code, and existing platforms accordingly. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Reviewed-by: Oliver O'Halloran --- v1->v2: - rename the i2c fields from pin* to brick* to be clearer (Oliver) --- core/platform.c | 24 ++++++++++++++---------- hw/npu2-common.c | 9 +++++++-- hw/npu2-opencapi.c | 10 +++++++--- include/platform.h | 12 ++++++++---- platforms/astbmc/zaius.c | 22 +++++++++++++--------- platforms/ibm-fsp/zz.c | 24 ++++++++++++++---------- 6 files changed, 63 insertions(+), 38 deletions(-) diff --git a/core/platform.c b/core/platform.c index 4b3eaa4bd1d1..7985ce564509 100644 --- a/core/platform.c +++ b/core/platform.c @@ -172,21 +172,25 @@ static int generic_start_preload_resource(enum resource_id id, uint32_t subid, /* These values will work for a ZZ booted using BML */ const struct platform_ocapi generic_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ /* * The ZZs we typically use for BML/generic platform tend to * have old planars and presence detection is broken there, so * force presence. */ - .force_presence = true, - .odl_phy_swap = true, + .force_presence = true, + .odl_phy_swap = true, }; static struct bmc_platform generic_bmc = { diff --git a/hw/npu2-common.c b/hw/npu2-common.c index b87d43469d5b..d60dda19dd3c 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -115,11 +115,16 @@ static bool _i2c_presence_detect(struct npu2_dev *dev) switch (dev->link_index) { case 2: - data = platform.ocapi->i2c_presence_odl0; + data = platform.ocapi->i2c_presence_brick2; break; case 3: - data = platform.ocapi->i2c_presence_odl1; + data = platform.ocapi->i2c_presence_brick3; break; + case 4: + data = platform.ocapi->i2c_presence_brick4; + break; + case 5: + data = platform.ocapi->i2c_presence_brick5; default: OCAPIERR(dev, "presence detection on invalid link\n"); return true; diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index fa28633bb150..decdfafb07db 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -816,12 +816,16 @@ static void assert_reset(struct npu2_dev *dev) switch (dev->brick_index) { case 2: - case 4: - pin = platform.ocapi->i2c_reset_odl0; + pin = platform.ocapi->i2c_reset_brick2; break; case 3: + pin = platform.ocapi->i2c_reset_brick3; + break; + case 4: + pin = platform.ocapi->i2c_reset_brick4; + break; case 5: - pin = platform.ocapi->i2c_reset_odl1; + pin = platform.ocapi->i2c_reset_brick5; break; default: assert(false); diff --git a/include/platform.h b/include/platform.h index efafbd2239b9..fee5a76cf9e8 100644 --- a/include/platform.h +++ b/include/platform.h @@ -50,11 +50,15 @@ struct platform_ocapi { uint8_t i2c_engine; /* I2C engine number */ uint8_t i2c_port; /* I2C port number */ uint8_t i2c_reset_addr; /* I2C address for reset */ - uint8_t i2c_reset_odl0; /* I2C pin to write to reset ODL0 */ - uint8_t i2c_reset_odl1; /* I2C pin to write to reset ODL1 */ + uint8_t i2c_reset_brick2; /* I2C pin to write to reset brick 2 */ + uint8_t i2c_reset_brick3; /* I2C pin to write to reset brick 3 */ + uint8_t i2c_reset_brick4; /* I2C pin to write to reset brick 4 */ + uint8_t i2c_reset_brick5; /* I2C pin to write to reset brick 5 */ uint8_t i2c_presence_addr; /* I2C address for presence detection */ - uint8_t i2c_presence_odl0; /* I2C mask for detection on ODL0 */ - uint8_t i2c_presence_odl1; /* I2C mask for detection on ODL1 */ + uint8_t i2c_presence_brick2; /* I2C pin to read for presence on brick 2 */ + uint8_t i2c_presence_brick3; /* I2C pin to read for presence on brick 3 */ + uint8_t i2c_presence_brick4; /* I2C pin to read for presence on brick 4 */ + uint8_t i2c_presence_brick5; /* I2C pin to read for presence on brick 5 */ bool force_presence; /* don't use i2c detection */ bool odl_phy_swap; /* Swap ODL1 to use brick 2 rather than * brick 1 lanes */ diff --git a/platforms/astbmc/zaius.c b/platforms/astbmc/zaius.c index 267e6578bbe7..0eed10c706d0 100644 --- a/platforms/astbmc/zaius.c +++ b/platforms/astbmc/zaius.c @@ -26,15 +26,19 @@ #include "astbmc.h" const struct platform_ocapi zaius_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ - .odl_phy_swap = true, + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ + .odl_phy_swap = true, }; #define NPU_BASE 0x5011000 diff --git a/platforms/ibm-fsp/zz.c b/platforms/ibm-fsp/zz.c index dc48768b8a96..e4ff981c9c1b 100644 --- a/platforms/ibm-fsp/zz.c +++ b/platforms/ibm-fsp/zz.c @@ -29,20 +29,24 @@ /* We don't yet create NPU device nodes on ZZ, but these values are correct */ const struct platform_ocapi zz_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ /* * i2c presence detection is broken on ZZ planar < v4 so we * force the presence until all our systems are upgraded */ - .force_presence = true, - .odl_phy_swap = true, + .force_presence = true, + .odl_phy_swap = true, }; static bool zz_probe(void) From patchwork Mon Aug 27 08:55:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 962384 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41zQmX50fVz9s3x for ; Mon, 27 Aug 2018 18:59:08 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41zQmX3l7VzF0Pd for ; Mon, 27 Aug 2018 18:59:08 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41zQj9580mzDrM8 for ; Mon, 27 Aug 2018 18:56:13 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7R8nPP3040905 for ; Mon, 27 Aug 2018 04:56:11 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m4cyhb00h-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 04:56:10 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 27 Aug 2018 09:56:08 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 27 Aug 2018 09:56:06 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w7R8u5O244433506 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 27 Aug 2018 08:56:05 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C866711C04A; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 316E311C058; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 27 Aug 2018 11:56:02 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id A5F9BA03A5; Mon, 27 Aug 2018 18:56:01 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 27 Aug 2018 18:55:55 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18082708-4275-0000-0000-000002B077EE X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18082708-4276-0000-0000-000037B97A23 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-08-27_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1808270096 Subject: [Skiboot] [PATCH v2 6/6] platforms/astbmc/witherspoon: Implement OpenCAPI support X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" OpenCAPI on Witherspoon is slightly more involved than on Zaius and ZZ, due to the OpenCAPI links using the SXM2 connectors that are used for NVLink GPUs. This patch adds the regular OpenCAPI platform information, and also a Witherspoon-specific presence detection callback that uses the previously added OCC GPU presence detection to figure out the device types plugged into each SXM2 socket. The SXM2 connectors are capable of carrying 2 OpenCAPI links, and future OpenCAPI devices are expected to make use of this. However, we don't yet support ganged links and the various implications that has for handling things like device reset, so for now, we only enable 1 brick per device. Signed-off-by: Andrew Donnellan --- v1->v2: - update platform field names --- platforms/astbmc/witherspoon.c | 216 +++++++++++++++++++++++++++++++++- 1 file changed, 213 insertions(+), 3 deletions(-) diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index ce83ff9701d3..2d9cccb63def 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -28,10 +28,18 @@ #include #include #include +#include +#include #include "astbmc.h" #include "ast.h" +static enum { + WITHERSPOON_TYPE_UNKNOWN, + WITHERSPOON_TYPE_SEQUOIA, + WITHERSPOON_TYPE_REDBUD +} witherspoon_type; + /* * HACK: Hostboot doesn't export the correct data for the system VPD EEPROM * for this system. So we need to work around it here. @@ -50,8 +58,35 @@ static void vpd_dt_fixup(void) } } +static void witherspoon_create_ocapi_i2c_bus(void) +{ + struct dt_node *xscom, *i2cm, *i2c_bus; + prlog(PR_DEBUG, "OCAPI: Adding I2C bus device node for OCAPI reset\n"); + dt_for_each_compatible(dt_root, xscom, "ibm,xscom") { + i2cm = dt_find_by_name(xscom, "i2cm@a1000"); + if (!i2cm) { + prlog(PR_ERR, "OCAPI: Failed to add I2C bus device node\n"); + continue; + } + + if (dt_find_by_name(i2cm, "i2c-bus@4")) + continue; + + i2c_bus = dt_new_addr(i2cm, "i2c-bus", 4); + dt_add_property_cells(i2c_bus, "reg", 4); + dt_add_property_cells(i2c_bus, "bus-frequency", 0x61a80); + dt_add_property_strings(i2c_bus, "compatible", + "ibm,opal-i2c", "ibm,power8-i2c-port", + "ibm,power9-i2c-port"); + } +} + static bool witherspoon_probe(void) { + struct dt_node *np; + int highest_gpu_group_id = 0; + int gpu_group_id; + if (!dt_node_is_compatible(dt_root, "ibm,witherspoon")) return false; @@ -63,6 +98,26 @@ static bool witherspoon_probe(void) vpd_dt_fixup(); + witherspoon_create_ocapi_i2c_bus(); + + dt_for_each_compatible(dt_root, np, "ibm,npu-link") { + gpu_group_id = dt_prop_get_u32(np, "ibm,npu-group-id"); + if (gpu_group_id > highest_gpu_group_id) + highest_gpu_group_id = gpu_group_id; + }; + + switch (highest_gpu_group_id) { + case 1: + witherspoon_type = WITHERSPOON_TYPE_REDBUD; + break; + case 2: + witherspoon_type = WITHERSPOON_TYPE_SEQUOIA; + break; + default: + witherspoon_type = WITHERSPOON_TYPE_UNKNOWN; + prlog(PR_NOTICE, "PLAT: Unknown Witherspoon variant detected\n"); + } + return true; } @@ -154,14 +209,168 @@ static void witherspoon_pre_pci_fixup(void) phb4_pre_pci_fixup_witherspoon(); } -static void witherspoon_npu2_device_detect(struct npu2 *npu) +static struct npu2_dev *find_link_index(struct npu2 *npu, uint32_t link_index) { - /* Stub until we implement real device detection */ for (int i = 0; i < npu->total_devices; i++) { - npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + if (npu->devices[i].link_index == link_index) + return &npu->devices[i]; } + prlog(PR_ERR, "PLAT: Could not find NPU link index %d\n", link_index); + return NULL; } +static void witherspoon_npu2_device_detect(struct npu2 *npu) +{ + struct proc_chip *chip; + struct npu2_dev *dev; + uint8_t state; + uint64_t i2c_port_id = 0; + char port_name[17]; + struct dt_node *dn; + int rc; + + bool gpu0_present, gpu1_present; + + if (witherspoon_type != WITHERSPOON_TYPE_REDBUD) { + prlog(PR_DEBUG, "PLAT: Setting all NPU links to NVLink, OpenCAPI only supported on Redbud\n"); + for (int i = 0; i < npu->total_devices; i++) { + npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + } + return; + } + assert(npu->total_devices == 6); + + chip = get_chip(npu->chip_id); + + /* Find I2C port */ + snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", + chip->id, platform.ocapi->i2c_engine, + platform.ocapi->i2c_port); + dt_for_each_compatible(dt_root, dn, "ibm,power9-i2c-port") { + if (streq(port_name, dt_prop_get(dn, "ibm,port-name"))) { + i2c_port_id = dt_prop_get_u32(dn, "ibm,opal-id"); + break; + } + } + + if (!i2c_port_id) { + prlog(PR_ERR, "PLAT: Could not find NPU presence I2C port\n"); + return; + } + + gpu0_present = occ_get_gpu_presence(chip, 0); + if (gpu0_present) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 slot present\n", chip->id); + } + + gpu1_present = occ_get_gpu_presence(chip, 1); + if (gpu1_present) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 slot present\n", chip->id); + } + + /* Set pins to input */ + state = 0xff; + rc = i2c_request_send(i2c_port_id, + platform.ocapi->i2c_presence_addr, SMBUS_WRITE, 3, + 1, &state, 1, 120); + if (rc) + goto i2c_failed; + + /* Read the presence value */ + state = 0x00; + rc = i2c_request_send(i2c_port_id, + platform.ocapi->i2c_presence_addr, SMBUS_READ, 0, + 1, &state, 1, 120); + if (rc) + goto i2c_failed; + + if (gpu0_present) { + if (state & (1 << 0)) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is OpenCAPI\n", + chip->id); + /* + * On witherspoon, bricks 2 and 3 are connected to + * the lanes matching links 1 and 0 in OpenCAPI mode. + */ + dev = find_link_index(npu, 0); + if (dev) { + dev->type = NPU2_DEV_TYPE_OPENCAPI; + dev->brick_index = 3; + } + + dev = find_link_index(npu, 1); + if (dev) { + /* dev->type = NPU2_DEV_TYPE_OPENCAPI; */ + /* We current don't support using the second link */ + dev->type = NPU2_DEV_TYPE_UNKNOWN; + dev->brick_index = 2; + } + } else { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is NVLink\n", + chip->id); + dev = find_link_index(npu, 0); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + dev = find_link_index(npu, 1); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + dev = find_link_index(npu, 2); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + } + } + + if (gpu1_present) { + if (state & (1 << 1)) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is OpenCAPI\n", + chip->id); + dev = find_link_index(npu, 4); + if (dev) + dev->type = NPU2_DEV_TYPE_OPENCAPI; + dev = find_link_index(npu, 5); + if (dev) + /* dev->type = NPU2_DEV_TYPE_OPENCAPI; */ + /* We current don't support using the second link */ + dev->type = NPU2_DEV_TYPE_UNKNOWN; + } else { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is NVLink\n", + chip->id); + dev = find_link_index(npu, 3); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + dev = find_link_index(npu, 4); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + dev = find_link_index(npu, 5); + if (dev) + dev->type = NPU2_DEV_TYPE_NVLINK; + } + } + + return; + +i2c_failed: + prlog(PR_ERR, "PLAT: NPU device type detection failed, rc=%d\n", rc); + return; +} + +const struct platform_ocapi witherspoon_ocapi = { + .i2c_engine = 1, + .i2c_port = 4, + .odl_phy_swap = false, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = 1 << 0, + .i2c_reset_brick3 = 1 << 0, + .i2c_reset_brick4 = 1 << 1, + .i2c_reset_brick5 = 1 << 1, + .i2c_presence_addr = 0x20, + /* unused, we do this in custom presence detect */ + .i2c_presence_brick2 = 0, + .i2c_presence_brick3 = 0, + .i2c_presence_brick4 = 0, + .i2c_presence_brick5 = 0, +}; + /* The only difference between these is the PCI slot handling */ DECLARE_PLATFORM(witherspoon) = { @@ -179,5 +388,6 @@ DECLARE_PLATFORM(witherspoon) = { .terminate = ipmi_terminate, .pci_get_slot_info = dt_slot_get_slot_info, + .ocapi = &witherspoon_ocapi, .npu2_device_detect = witherspoon_npu2_device_detect, };