From patchwork Mon Sep 3 04:52:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965225 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423czy1GT5z9s2P for ; Mon, 3 Sep 2018 14:53:34 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423czx6GK4zF33f for ; Mon, 3 Sep 2018 14:53:33 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czq6W4KzF38W for ; Mon, 3 Sep 2018 14:53:27 +1000 (AEST) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834mxMT136648 for ; Mon, 3 Sep 2018 00:53:25 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m8vqvjsbh-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:24 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:23 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:21 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rK6e41353282 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:20 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 57B8E11C052; Mon, 3 Sep 2018 07:53:14 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0681D11C050; Mon, 3 Sep 2018 07:53:14 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 3 Sep 2018 07:53:13 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id E4C33A0253; Mon, 3 Sep 2018 14:53:17 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:52:59 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-0008-0000-0000-0000026B2C0A X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-0009-0000-0000-000021D33A42 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=597 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 1/6] occ: Wait if OCC GPU presence status not immediately available X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" It takes a few seconds for the OCC to set everything up in order to read GPU presence. At present, we try to kick off OCC initialisation as early as possible to maximise the time it has to read GPU presence. Unfortunately sometimes that's not enough, so add a loop in occ_get_gpu_presence() so that on the first time we try to get GPU presence we keep trying for up to 2 seconds. Experimentally this seems to be adequate. Fixes: 9b394a32c8ea ("occ: Add support for GPU presence detection") Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v2->v3: - Remove noisy print (Rashmica/Fred) --- hw/occ.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/hw/occ.c b/hw/occ.c index a55bf8ed4f54..9580bb873639 100644 --- a/hw/occ.c +++ b/hw/occ.c @@ -1238,14 +1238,24 @@ exit: bool occ_get_gpu_presence(struct proc_chip *chip, int gpu_num) { struct occ_dynamic_data *ddata; + static int max_retries = 20; + static bool found = false; assert(gpu_num <= 2); ddata = get_occ_dynamic_data(chip); + while (!found && max_retries) { + if (ddata->major_version == 0 && ddata->minor_version >= 1) { + found = true; + break; + } + time_wait_ms(100); + max_retries--; + ddata = get_occ_dynamic_data(chip); + } - if (ddata->major_version != 0 || ddata->minor_version < 1) { - prlog(PR_INFO, "OCC: OCC not reporting GPU slot presence, " - "assuming device is present\n"); + if (!found) { + prlog(PR_INFO, "OCC: No GPU slot presence, assuming GPU present\n"); return true; } From patchwork Mon Sep 3 04:53:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965228 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423d0f5jNGz9s2P for ; Mon, 3 Sep 2018 14:54:10 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423d0f2vCszF397 for ; Mon, 3 Sep 2018 14:54:10 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czs1ggdzF38Q for ; Mon, 3 Sep 2018 14:53:28 +1000 (AEST) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834mwij029860 for ; Mon, 3 Sep 2018 00:53:26 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m8u1snq6u-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:26 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:24 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:22 +0100 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rLLU41353288 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:21 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0A84F5204E; Mon, 3 Sep 2018 07:53:15 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 21EE352050; Mon, 3 Sep 2018 07:53:14 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id F3C5DA026D; Mon, 3 Sep 2018 14:53:17 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:53:00 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-0008-0000-0000-0000026B2C09 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-0009-0000-0000-000021D33A43 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 2/6] npu2: Split device index into brick and link index X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" On Witherspoon, OpenCAPI devices attached to link indexes 0 and 1 are handled by bricks 2 and 3. Rename index to brick_index, and add a new field, link_index, to refer to the link index. For now, we set those values identically. Signed-off-by: Andrew Donnellan Acked-by: Reza Arbab Reviewed-by: Alistair Popple Reviewed-by: Frederic Barrat --- hw/npu2-hw-procedures.c | 8 ++-- hw/npu2-opencapi.c | 77 +++++++++++++++++++++--------------------- hw/npu2.c | 21 ++++++----- include/npu2-regs.h | 14 ++++---- include/npu2.h | 7 ++-- 5 files changed, 67 insertions(+), 60 deletions(-) diff --git a/hw/npu2-hw-procedures.c b/hw/npu2-hw-procedures.c index 86864629a66b..8767d9259984 100644 --- a/hw/npu2-hw-procedures.c +++ b/hw/npu2-hw-procedures.c @@ -198,7 +198,7 @@ DEFINE_PROCEDURE(nop); /* Return the brick number (0-2) within an obus chiplet */ static int obus_brick_index(struct npu2_dev *ndev) { - int index = ndev->index % 3; + int index = ndev->brick_index % 3; /* On the second obus chiplet, index is reversed */ if ((ndev->pl_xscom_base & 0x3F000000) != 0x09000000) @@ -432,7 +432,7 @@ DEFINE_PROCEDURE(phy_reset, phy_reset_wait, phy_reset_complete); /* Procedure 1.2.6 - I/O PHY Tx Impedance Calibration */ static uint32_t phy_tx_zcal(struct npu2_dev *ndev) { - if (ndev->npu->tx_zcal_complete[ndev->index > 2]) + if (ndev->npu->tx_zcal_complete[ndev->brick_index > 2]) return PROCEDURE_COMPLETE; /* Turn off SW enable and enable zcal state machine */ @@ -603,7 +603,7 @@ static uint32_t phy_tx_zcal_calculate(struct npu2_dev *ndev) phy_write(ndev, &NPU2_PHY_TX_MARGINPU_SELECT, therm(margin_select + 1)/2); phy_write(ndev, &NPU2_PHY_TX_MARGINPD_SELECT, therm(margin_select + 1)/2); - ndev->npu->tx_zcal_complete[ndev->index > 2] = 1; + ndev->npu->tx_zcal_complete[ndev->brick_index > 2] = 1; return PROCEDURE_COMPLETE; } DEFINE_PROCEDURE(phy_tx_zcal, phy_tx_zcal_wait, phy_tx_zcal_calculate); @@ -977,7 +977,7 @@ void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev) uint64_t status_xscom; int lane, bit = 7; - switch (dev->index) { + switch (dev->brick_index) { case 2: status_xscom = OB0_ODL0_TRAINING_STATUS; break; diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 57c2f2fcc075..68ae93a2f836 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -53,11 +53,11 @@ #include #define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->index, ## a) + dev->npu->chip_id, dev->brick_index, ## a) #define NPU_IRQ_LEVELS 35 @@ -753,16 +753,16 @@ static void setup_global_mmio_bar(uint32_t gcid, uint32_t scom_base, static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); - uint64_t offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + uint64_t stack = index_to_stack(dev->brick_index); + uint64_t offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_NTL0_BAR : NPU2_NTL1_BAR; - uint64_t pa_offset = index_to_block(dev->index) == NPU2_BLOCK_OTL0 ? + uint64_t pa_offset = index_to_block(dev->brick_index) == NPU2_BLOCK_OTL0 ? NPU2_CQ_CTL_MISC_MMIOPA0_CONFIG : NPU2_CQ_CTL_MISC_MMIOPA1_CONFIG; uint64_t addr, size, reg; prlog(PR_DEBUG, "OCAPI: %s: Setup AFU MMIO BARs\n", __func__); - phys_map_get(gcid, NPU_OCAPI_MMIO, dev->index, &addr, &size); + phys_map_get(gcid, NPU_OCAPI_MMIO, dev->brick_index, &addr, &size); prlog(PR_DEBUG, "OCAPI: AFU MMIO set to %llx, size %llx\n", addr, size); write_bar(gcid, scom_base, NPU2_REG_OFFSET(stack, 0, offset), addr, @@ -783,7 +783,7 @@ static void setup_afu_mmio_bars(uint32_t gcid, uint32_t scom_base, static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); + uint64_t stack = index_to_stack(dev->brick_index); int stack_num = stack - NPU2_STACK_STCK_0; uint64_t addr, size; @@ -799,8 +799,8 @@ static void setup_afu_config_bars(uint32_t gcid, uint32_t scom_base, static void otl_enabletx(uint32_t gcid, uint32_t scom_base, struct npu2_dev *dev) { - uint64_t stack = index_to_stack(dev->index); - uint64_t block = index_to_block(dev->index); + uint64_t stack = index_to_stack(dev->brick_index); + uint64_t block = index_to_block(dev->brick_index); uint64_t reg; /* OTL Config 2 Register */ @@ -822,7 +822,7 @@ static void assert_reset(struct npu2_dev *dev) uint8_t pin, data; int rc; - switch (dev->index) { + switch (dev->brick_index) { case 2: case 4: pin = platform.ocapi->i2c_reset_odl0; @@ -910,7 +910,7 @@ static bool i2c_presence_detect(struct npu2_dev *dev) OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); - switch (dev->index) { + switch (dev->brick_index) { // TODO(ajd): Link or brick index? case 2: data = platform.ocapi->i2c_presence_odl0; break; @@ -929,7 +929,7 @@ static void reset_odl(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -965,7 +965,7 @@ static void set_init_pattern(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -992,7 +992,7 @@ static void start_training(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; - switch (dev->index) { + switch (dev->brick_index) { case 2: config_xscom = OB0_ODL0_CONFIG; break; @@ -1035,7 +1035,7 @@ static int64_t npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) uint64_t reg; int64_t link_width, training_status, rc = OPAL_SUCCESS; - reg = get_odl_status(dev->npu->chip_id, dev->index); + reg = get_odl_status(dev->npu->chip_id, dev->brick_index); link_width = GETFIELD(OB_ODL_STATUS_TRAINED_MODE, reg); training_status = GETFIELD(OB_ODL_STATUS_TRAINING_STATE_MACHINE, reg); @@ -1071,7 +1071,7 @@ static int64_t npu2_opencapi_retry_state(struct pci_slot *slot) */ OCAPIERR(dev, "Link failed to train, final link status: %016llx\n", - get_odl_status(chip_id, dev->index)); + get_odl_status(chip_id, dev->brick_index)); return OPAL_HARDWARE; } @@ -1093,7 +1093,7 @@ static int64_t npu2_opencapi_poll_link(struct pci_slot *slot) pci_slot_set_state(slot, OCAPI_SLOT_LINK_WAIT); /* fall-through */ case OCAPI_SLOT_LINK_WAIT: - reg = get_odl_status(chip_id, dev->index); + reg = get_odl_status(chip_id, dev->brick_index); if (GETFIELD(OB_ODL_STATUS_TRAINING_STATE_MACHINE, reg) == OCAPI_LINK_STATE_TRAINED) { OCAPIINF(dev, "link trained in %lld ms\n", @@ -1153,9 +1153,9 @@ static int64_t npu2_opencapi_freset(struct pci_slot *slot) if (dev->train_need_fence) { OCAPIDBG(dev, "Fencing OTL during reset\n"); set_fence_control(chip_id, dev->npu->xscom_base, - dev->index, 0b11); + dev->brick_index, 0b11); npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, - PPC_BIT(dev->index + 6)); + PPC_BIT(dev->brick_index + 6)); dev->train_fenced = true; } dev->train_need_fence = true; @@ -1180,9 +1180,10 @@ static int64_t npu2_opencapi_freset(struct pci_slot *slot) case OCAPI_SLOT_FRESET_DEASSERT_DELAY: if (dev->train_fenced) { OCAPIDBG(dev, "Unfencing OTL after reset\n"); - npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, PPC_BIT(dev->index)); + npu2_write(dev->npu, NPU2_MISC_FENCE_STATE, + PPC_BIT(dev->brick_index)); set_fence_control(chip_id, dev->npu->xscom_base, - dev->index, 0b00); + dev->brick_index, 0b00); dev->train_fenced = false; } @@ -1263,7 +1264,7 @@ static int64_t npu2_opencapi_pcicfg_read(struct phb *phb, uint32_t bdfn, return rc; genid_base = dev->bars[1].npu2_bar.base + - (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | @@ -1321,7 +1322,7 @@ static int64_t npu2_opencapi_pcicfg_write(struct phb *phb, uint32_t bdfn, return rc; genid_base = dev->bars[1].npu2_bar.base + - (index_to_block(dev->index) == NPU2_BLOCK_OTL1 ? 256 : 0); + (index_to_block(dev->brick_index) == NPU2_BLOCK_OTL1 ? 256 : 0); cfg_addr = NPU2_CQ_CTL_CONFIG_ADDR_ENABLE; cfg_addr = SETFIELD(NPU2_CQ_CTL_CONFIG_ADDR_BUS_NUMBER | @@ -1409,13 +1410,14 @@ static int64_t npu2_opencapi_set_pe(struct phb *phb, p = dev->npu; pe_bdfn = dev->bdfn; - + val = NPU2_MISC_BRICK_BDF2PE_MAP_ENABLE; val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, pe_bdfn); reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); - p->bdf2pe_cache[dev->index] = val; + NPU2_MISC_BRICK0_BDF2PE_MAP0 + + (dev->brick_index * 0x18)); + p->bdf2pe_cache[dev->brick_index] = val; npu2_write(p, reg, val); return OPAL_SUCCESS; @@ -1426,8 +1428,8 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, { uint32_t irq; struct npu2_dev *dev = phb_to_npu2_dev_ocapi(phb); - uint64_t block = index_to_block(dev->index); - uint64_t stacku = index_to_stacku(dev->index); + uint64_t block = index_to_block(dev->brick_index); + uint64_t stacku = index_to_stacku(dev->brick_index); uint64_t dsisr, dar, tfc, handle; /* @@ -1631,7 +1633,8 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, dev->phb_ocapi.ops = &npu2_opencapi_ops; dev->phb_ocapi.phb_type = phb_type_npu_v2_opencapi; dev->phb_ocapi.scan_map = 0; - dev->index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + dev->link_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); + dev->brick_index = dev->link_index; dev->pl_xscom_base = dt_prop_get_u64(dn_link, "ibm,npu-phy"); dev->lane_mask = dt_prop_get_u32(dn_link, "ibm,npu-lane-mask"); dev->link_speed = dt_prop_get_u64(dn_link, "ibm,link-speed"); @@ -1664,7 +1667,7 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, /* Procedure 13.1.3.9 - AFU Config BARs */ setup_afu_config_bars(n->chip_id, n->xscom_base, dev); - set_fence_control(n->chip_id, n->xscom_base, dev->index, 0b00); + set_fence_control(n->chip_id, n->xscom_base, dev->brick_index, 0b00); if (npu2_ocapi_training_state != NPU2_TRAIN_DEFAULT) { setup_debug_training_state(dev); @@ -1847,8 +1850,8 @@ static int64_t opal_npu_spa_setup(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); if (block == NPU2_BLOCK_OTL1) offset = NPU2_XSL_PSL_SPAP_A1; else @@ -1912,8 +1915,8 @@ static int64_t opal_npu_spa_clear_cache(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); cc_inv = NPU2_REG_OFFSET(stack, NPU2_BLOCK_XSL, NPU2_XSL_PSL_LLCMD_A0); lock(&dev->npu->lock); @@ -1987,8 +1990,8 @@ static int64_t opal_npu_tl_set(uint64_t phb_id, uint32_t __unused bdfn, if (!dev) return OPAL_PARAMETER; - block = index_to_block(dev->index); - stack = index_to_stack(dev->index); + block = index_to_block(dev->brick_index); + stack = index_to_stack(dev->brick_index); /* * The 'capabilities' argument defines what TL template the * device can receive. OpenCAPI 3.0 and 4.0 define 64 templates, so diff --git a/hw/npu2.c b/hw/npu2.c index 2b3540583ac1..be1c304420fc 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1075,10 +1075,10 @@ static int64_t npu2_set_pe(struct phb *phb, val = SETFIELD(NPU2_CQ_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); if (!NPU2DEV_BRICK(dev)) - reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->index/2, + reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->brick_index/2, NPU2_BLOCK_CTL, NPU2_CQ_BRICK0_BDF2PE_MAP0); else - reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->index/2, + reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + dev->brick_index/2, NPU2_BLOCK_CTL, NPU2_CQ_BRICK1_BDF2PE_MAP0); npu2_write(p, reg, val); @@ -1086,8 +1086,8 @@ static int64_t npu2_set_pe(struct phb *phb, val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_PE, val, pe_num); val = SETFIELD(NPU2_MISC_BRICK_BDF2PE_MAP_BDF, val, dev->nvlink.gpu_bdfn); reg = NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_MISC, - NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->index * 0x18)); - p->bdf2pe_cache[dev->index] = val; + NPU2_MISC_BRICK0_BDF2PE_MAP0 + (dev->brick_index * 0x18)); + p->bdf2pe_cache[dev->brick_index] = val; npu2_write(p, reg, val); return OPAL_SUCCESS; @@ -1601,7 +1601,7 @@ static uint32_t npu2_populate_vendor_cap(struct npu2_dev *dev, NULL); /* Link index */ - PCI_VIRT_CFG_INIT_RO(pvd, start + 0xc, 1, dev->index); + PCI_VIRT_CFG_INIT_RO(pvd, start + 0xc, 1, dev->link_index); return start + VENDOR_CAP_LEN; } @@ -1725,7 +1725,8 @@ static void npu2_populate_devices(struct npu2 *p, dev->type = NPU2_DEV_TYPE_NVLINK; dev->npu = p; dev->dt_node = link; - dev->index = dt_prop_get_u32(link, "ibm,npu-link-index"); + dev->link_index = dt_prop_get_u32(link, "ibm,npu-link-index"); + dev->brick_index = dev->link_index; group_id = dt_prop_get_u32(link, "ibm,npu-group-id"); dev->bdfn = npu_allocate_bdfn(p, group_id); @@ -1742,7 +1743,7 @@ static void npu2_populate_devices(struct npu2 *p, stack = NPU2_STACK_STCK_0 + NPU2DEV_STACK(dev); npu2_bar = &dev->bars[0].npu2_bar; npu2_bar->type = NPU_NTL; - npu2_bar->index = dev->index; + npu2_bar->index = dev->brick_index; npu2_bar->reg = NPU2_REG_OFFSET(stack, 0, NPU2DEV_BRICK(dev) == 0 ? NPU2_NTL0_BAR : NPU2_NTL1_BAR); npu2_get_bar(p->chip_id, npu2_bar); @@ -2248,8 +2249,10 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, goto out; } - xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_STACK, xts_bdf_lpar, 0x4 >> (ndev->index / 2)); - xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BRICK, xts_bdf_lpar, (ndev->index % 2)); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_STACK, xts_bdf_lpar, + 0x4 >> (ndev->brick_index / 2)); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BRICK, xts_bdf_lpar, + (ndev->brick_index % 2)); NPU2DBG(p, "XTS_BDF_MAP[%03d] = 0x%08llx\n", id, xts_bdf_lpar); npu2_write(p, NPU2_XTS_BDF_MAP + id*8, xts_bdf_lpar); diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 6bd77e48bf4f..8c1ba5fff37a 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -44,19 +44,19 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, (((stack) << 20) | ((block) << 16) | (offset)) #define NPU2_NTL_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->index >> 1), \ - NPU2_BLOCK_NTL0 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->brick_index >> 1), \ + NPU2_BLOCK_NTL0 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_NTLU_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0U + ((ndev)->index >> 1), \ - NPU2_BLOCK_NTL0 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(NPU2_STACK_STCK_0U + ((ndev)->brick_index >> 1), \ + NPU2_BLOCK_NTL0 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_DL_REG_OFFSET(ndev, offset) \ - NPU2_REG_OFFSET(((ndev)->index >> 1), \ - 8 + ((ndev)->index % 2)*2, offset) + NPU2_REG_OFFSET(((ndev)->brick_index >> 1), \ + 8 + ((ndev)->brick_index % 2)*2, offset) #define NPU2_SM_REG_OFFSET(ndev, sm, offset) \ - NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->index >> 1), \ + NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + ((ndev)->brick_index >> 1), \ NPU2_BLOCK_SM_0 + (sm), offset) /* Get the offset for this register */ diff --git a/include/npu2.h b/include/npu2.h index 4c2e20e0e2f7..10742031ec0f 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -47,10 +47,10 @@ #define NPU2_DEV_DL_RESET 0x2 /* Return the stack (0-2) of a device */ -#define NPU2DEV_STACK(ndev) ((ndev)->index / 2) +#define NPU2DEV_STACK(ndev) ((ndev)->brick_index / 2) /* Return the brick number (0-1) within a stack */ -#define NPU2DEV_BRICK(ndev) ((ndev)->index % 2) +#define NPU2DEV_BRICK(ndev) ((ndev)->brick_index % 2) /* This represents the state of the actual hardware BARs not the * emulated PCIe BARs. The is a subtle difference between the two as @@ -111,7 +111,8 @@ struct npu2_dev_nvlink { struct npu2_dev { enum npu2_dev_type type; - uint32_t index; + uint32_t link_index; + uint32_t brick_index; uint64_t pl_xscom_base; struct dt_node *dt_node; struct npu2_pcie_bar bars[2]; From patchwork Mon Sep 3 04:53:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965231 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423d0y0mJhz9s2P for ; Mon, 3 Sep 2018 14:54:26 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423d0x5zBQzF39R for ; Mon, 3 Sep 2018 14:54:25 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czs3lbRzF38W for ; Mon, 3 Sep 2018 14:53:29 +1000 (AEST) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834mwao112130 for ; Mon, 3 Sep 2018 00:53:27 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m8vpptwcg-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:26 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:25 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:22 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rLDZ36307156 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:21 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 80131AE056; Mon, 3 Sep 2018 07:52:46 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 96368AE051; Mon, 3 Sep 2018 07:52:45 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 3 Sep 2018 07:52:45 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 0669EA027C; Mon, 3 Sep 2018 14:53:18 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:53:01 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-4275-0000-0000-000002B42ED1 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-4276-0000-0000-000037BD3B26 Message-Id: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 3/6] hw/npu2: Common NPU2 init routine between NVLink and OpenCAPI X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Replace probe_npu2() and probe_npu2_opencapi() with a new shared probe_npu2(). Refactor some of the common NPU setup code into shared code. No functional change. This patch does not implement support for using both types of devices simultaneously on the same NPU - we expect to add this sometime in the future. Signed-off-by: Andrew Donnellan Acked-by: Reza Arbab Reviewed-by: Alistair Popple Reviewed-by: Frederic Barrat --- v1->v2: - cleaned up unneeded null check (Alistair) v2->v3: - add check for npu->devices overflow (Fred) - get rid of ocapi_global_mmio_base, which I'd separated out for... some presumably very good reason that I can't remember (Fred) --- core/init.c | 2 +- hw/npu2-common.c | 119 +++++++++++++++++++++++- hw/npu2-opencapi.c | 158 ++++++++++--------------------- hw/npu2.c | 236 ++++++++++++++-------------------------------- include/npu2.h | 5 +- include/skiboot.h | 1 +- 6 files changed, 250 insertions(+), 271 deletions(-) diff --git a/core/init.c b/core/init.c index ca6c468c3941..9095981e67e0 100644 --- a/core/init.c +++ b/core/init.c @@ -1126,8 +1126,6 @@ void __noreturn __nomcount main_cpu_entry(const void *fdt) /* Probe NPUs */ probe_npu(); probe_npu2(); - /* TODO: Eventually, we'll do NVLink and OpenCAPI together */ - probe_npu2_opencapi(); /* Initialize PCI */ pci_init_slots(); diff --git a/hw/npu2-common.c b/hw/npu2-common.c index d076b4906fcc..476fc0683d56 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -20,6 +20,7 @@ #include #include #include +#include enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link) { @@ -107,3 +108,121 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas npu2_scom_write(p->chip_id, p->xscom_base, reg, NPU2_MISC_DA_LEN_4B, (uint64_t)new_val << 32); } + +static struct npu2 *setup_npu(struct dt_node *dn) +{ + struct npu2 *npu; + struct npu2_dev *dev; + struct dt_node *np; + uint32_t num_links; + void *npumem; + char *path; + int gcid; + struct proc_chip *chip; + int i = 0; + + /* Retrieve chip ID */ + path = dt_get_path(dn); + gcid = dt_get_chip_id(dn); + chip = get_chip(gcid); + assert(chip); + + num_links = dt_prop_get_u32(dn, "ibm,npu-links"); + npumem = zalloc(sizeof(struct npu2) + num_links * + sizeof(struct npu2_dev)); + assert(npumem); + npu = npumem; + + npu->dt_node = dn; + npu->index = dt_prop_get_u32(dn, "ibm,npu-index"); + npu->chip_id = gcid; + npu->xscom_base = dt_get_address(dn, 0, NULL); + npu->phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); + npu->devices = npumem + sizeof(struct npu2); + + dt_for_each_compatible(dn, np, "ibm,npu-link") { + assert(i < num_links); + dev = &npu->devices[i]; + dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); + /* May be overridden by platform presence detection */ + dev->brick_index = dev->link_index; + dev->type = npu2_dt_link_dev_type(np); + dev->npu = npu; + dev->dt_node = np; + dev->pl_xscom_base = dt_prop_get_u64(np, "ibm,npu-phy"); + dev->lane_mask = dt_prop_get_u32(np, "ibm,npu-lane-mask"); + dev->link_speed = dt_prop_get_u64(np, "ibm,link-speed"); + i++; + }; + npu->total_devices = i; + + prlog(PR_INFO, "NPU: Chip %d Found NPU2#%d (%d links) at %s\n", + npu->chip_id, npu->index, npu->total_devices, path); + prlog(PR_INFO, " SCOM Base: %08llx\n", npu->xscom_base); + free(path); + return npu; +} + +static void setup_devices(struct npu2 *npu) +{ + bool nvlink_detected = false, ocapi_detected = false; + struct npu2_dev *dev; + + /* + * TODO: In future, we'll do brick configuration here to support mixed + * setups. + */ + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + switch (dev->type) { + case NPU2_DEV_TYPE_NVLINK: + nvlink_detected = true; + break; + case NPU2_DEV_TYPE_OPENCAPI: + ocapi_detected = true; + break; + default: + prlog(PR_INFO, "NPU: Link %d device not present\n", + npu->devices[i].link_index); + } + } + + if (nvlink_detected && ocapi_detected) { + prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported, aborting NPU init\n"); + return; + } + + if (nvlink_detected) + npu2_nvlink_init_npu(npu); + else if (ocapi_detected) + npu2_opencapi_init_npu(npu); +} + +void probe_npu2(void) +{ + struct proc_chip *chip = next_chip(NULL); + struct npu2 *npu; + struct dt_node *np; + const char *zcal; + + /* Abort if we're running on DD1 */ + if (chip && + (chip->type == PROC_CHIP_P9_NIMBUS || + chip->type == PROC_CHIP_P9_CUMULUS) && + (chip->ec_level & 0xf0) == 0x10) { + prlog(PR_INFO, "NPU2: DD1 not supported\n"); + return; + } + + /* Check for a zcal override */ + zcal = nvram_query("nv_zcal_override"); + if (zcal) { + nv_zcal_nominal = atoi(zcal); + prlog(PR_WARNING, "NPU2: Using ZCAL impedance override = %d\n", nv_zcal_nominal); + } + + dt_for_each_compatible(dt_root, np, "ibm,power9-npu") { + npu = setup_npu(np); + setup_devices(npu); + } +} diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 68ae93a2f836..270876f78aa3 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1450,7 +1450,7 @@ static int npu2_add_mmio_regs(struct phb *phb, struct pci_device *pd, dar = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_DAR(stacku, block); tfc = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_TFC(stacku, block); handle = (uint64_t) dev->npu->regs + NPU2_OTL_OSL_PEHANDLE(stacku, - block); + block); dt_add_property_cells(pd->dn, "ibm,opal-xsl-irq", irq); dt_add_property_cells(pd->dn, "ibm,opal-xsl-mmio", hi32(dsisr), lo32(dsisr), @@ -1576,20 +1576,15 @@ static void setup_debug_training_state(struct npu2_dev *dev) } } -static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, - struct npu2_dev *dev) +static void setup_device(struct npu2_dev *dev) { - uint32_t dev_index, npu_index; struct dt_node *dn_phb, *dn; struct pci_slot *slot; char port_name[17]; uint64_t mm_win[2]; - dev_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); - npu_index = dt_prop_get_u32(n->dt_node, "ibm,npu-index"); - /* Populate PHB device node */ - phys_map_get(n->chip_id, NPU_OCAPI_MMIO, dev_index, &mm_win[0], + phys_map_get(dev->npu->chip_id, NPU_OCAPI_MMIO, dev->brick_index, &mm_win[0], &mm_win[1]); prlog(PR_DEBUG, "OCAPI: Setting MMIO window to %016llx + %016llx\n", mm_win[0], mm_win[1]); @@ -1609,40 +1604,28 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, dt_add_property_strings(dn_phb, "device_type", "pciex"); dt_add_property(dn_phb, "reg", mm_win, sizeof(mm_win)); - dt_add_property_cells(dn_phb, "ibm,npu-index", npu_index); - dt_add_property_cells(dn_phb, "ibm,chip-id", n->chip_id); - dt_add_property_cells(dn_phb, "ibm,xscom-base", n->xscom_base); - dt_add_property_cells(dn_phb, "ibm,npcq", n->dt_node->phandle); + dt_add_property_cells(dn_phb, "ibm,npu-index", dev->npu->index); + dt_add_property_cells(dn_phb, "ibm,chip-id", dev->npu->chip_id); + dt_add_property_cells(dn_phb, "ibm,xscom-base", dev->npu->xscom_base); + dt_add_property_cells(dn_phb, "ibm,npcq", dev->npu->dt_node->phandle); dt_add_property_cells(dn_phb, "ibm,links", 1); dt_add_property(dn_phb, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(dn_phb, "ibm,phb-diag-data-size", 0); dt_add_property_cells(dn_phb, "ibm,opal-num-pes", NPU2_MAX_PE_NUM); - n->mm_base = mm_win[0]; - n->mm_size = mm_win[1]; - dt_add_property_cells(dn_phb, "ranges", 0x02000000, - hi32(n->mm_base), lo32(n->mm_base), - hi32(n->mm_base), lo32(n->mm_base), - hi32(n->mm_size), lo32(n->mm_size)); + hi32(mm_win[0]), lo32(mm_win[0]), + hi32(mm_win[0]), lo32(mm_win[0]), + hi32(mm_win[1]), lo32(mm_win[1])); - dev->type = NPU2_DEV_TYPE_OPENCAPI; - dev->npu = n; - dev->dt_node = dn_link; dev->phb_ocapi.dt_node = dn_phb; dev->phb_ocapi.ops = &npu2_opencapi_ops; dev->phb_ocapi.phb_type = phb_type_npu_v2_opencapi; dev->phb_ocapi.scan_map = 0; - dev->link_index = dt_prop_get_u32(dn_link, "ibm,npu-link-index"); - dev->brick_index = dev->link_index; - dev->pl_xscom_base = dt_prop_get_u64(dn_link, "ibm,npu-phy"); - dev->lane_mask = dt_prop_get_u32(dn_link, "ibm,npu-lane-mask"); - dev->link_speed = dt_prop_get_u64(dn_link, "ibm,link-speed"); + dev->bdfn = 0; dev->train_need_fence = false; dev->train_fenced = false; - n->total_devices++; - /* Find I2C port for handling device reset */ snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", dev->npu->chip_id, platform.ocapi->i2c_engine, @@ -1663,11 +1646,11 @@ static void npu2_opencapi_setup_device(struct dt_node *dn_link, struct npu2 *n, /* TODO: Procedure 13.1.3.7 - AFU Memory Range BARs */ /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ - setup_afu_mmio_bars(n->chip_id, n->xscom_base, dev); + setup_afu_mmio_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); /* Procedure 13.1.3.9 - AFU Config BARs */ - setup_afu_config_bars(n->chip_id, n->xscom_base, dev); + setup_afu_config_bars(dev->npu->chip_id, dev->npu->xscom_base, dev); - set_fence_control(n->chip_id, n->xscom_base, dev->brick_index, 0b00); + set_fence_control(dev->npu->chip_id, dev->npu->xscom_base, dev->brick_index, 0b00); if (npu2_ocapi_training_state != NPU2_TRAIN_DEFAULT) { setup_debug_training_state(dev); @@ -1689,107 +1672,72 @@ failed: return; } -static void npu2_opencapi_probe(struct dt_node *dn) +static void read_nvram_training_state(void) { - struct dt_node *link; - char *path; - uint32_t gcid, index, links, scom_base; - uint64_t reg[2]; - uint64_t dev_index; - struct npu2 *n; - int rc, i = 0; - - gcid = dt_get_chip_id(dn); - index = dt_prop_get_u32(dn, "ibm,npu-index"); - links = dt_prop_get_u32(dn, "ibm,npu-links"); - - /* Don't try to init when we have an NVLink link */ - dt_for_each_compatible(dn, link, "ibm,npu-link") { - if (npu2_dt_link_dev_type(link) != NPU2_DEV_TYPE_OPENCAPI) { - prlog(PR_DEBUG, - "OCAPI: NPU%d: Non-OpenCAPI link found, skipping OpenCAPI init\n", - index); - return; - } + const char *state; + + state = nvram_query("opencapi-link-training"); + if (state) { + if (!strcmp(state, "prbs31")) + npu2_ocapi_training_state = NPU2_TRAIN_PRBS31; + else if (!strcmp(state, "none")) + npu2_ocapi_training_state = NPU2_TRAIN_NONE; + else + prlog(PR_WARNING, + "OCAPI: invalid training state in NVRAM: %s\n", + state); } +} - path = dt_get_path(dn); - prlog(PR_INFO, "OCAPI: Chip %d Found OpenCAPI NPU%d (%d links) at %s\n", - gcid, index, links, path); - free(path); +int npu2_opencapi_init_npu(struct npu2 *npu) +{ + struct npu2_dev *dev; + uint64_t reg[2]; + int rc; assert(platform.ocapi); + read_nvram_training_state(); /* TODO: Test OpenCAPI with fast reboot and make it work */ disable_fast_reboot("OpenCAPI device enabled"); - scom_base = dt_get_address(dn, 0, NULL); - prlog(PR_INFO, "OCAPI: SCOM Base: %08x\n", scom_base); + setup_global_mmio_bar(npu->chip_id, npu->xscom_base, reg); - setup_global_mmio_bar(gcid, scom_base, reg); + npu->regs = (void *)reg[0]; - n = zalloc(sizeof(struct npu2) + links * sizeof(struct npu2_dev)); - n->devices = (struct npu2_dev *)(n + 1); - n->chip_id = gcid; - n->xscom_base = scom_base; - n->regs = (void *)reg[0]; - n->dt_node = dn; + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; - dt_for_each_compatible(dn, link, "ibm,npu-link") { - dev_index = dt_prop_get_u32(link, "ibm,npu-link-index"); - prlog(PR_INFO, "OCAPI: Configuring link index %lld\n", - dev_index); + prlog(PR_INFO, "OCAPI: Configuring link index %d, brick %d\n", + dev->link_index, dev->brick_index); /* Procedure 13.1.3.1 - Select OCAPI vs NVLink */ - brick_config(gcid, scom_base, dev_index); + brick_config(npu->chip_id, npu->xscom_base, dev->brick_index); /* Procedure 13.1.3.5 - Transaction Layer Configuration */ - tl_config(gcid, scom_base, dev_index); + tl_config(npu->chip_id, npu->xscom_base, dev->brick_index); /* Procedure 13.1.3.6 - Address Translation Configuration */ - address_translation_config(gcid, scom_base, dev_index); + address_translation_config(npu->chip_id, npu->xscom_base, dev->brick_index); } /* Procedure 13.1.3.10 - Interrupt Configuration */ - rc = setup_irq(n); + rc = setup_irq(npu); if (rc) goto failed; - dt_for_each_compatible(dn, link, "ibm,npu-link") { - npu2_opencapi_setup_device(link, n, &n->devices[i]); - i++; + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (dev->type != NPU2_DEV_TYPE_OPENCAPI) + continue; + setup_device(dev); } - return; + return 0; failed: - free(n); -} - -static void read_nvram_training_state(void) -{ - const char *state; - - state = nvram_query("opencapi-link-training"); - if (state) { - if (!strcmp(state, "prbs31")) - npu2_ocapi_training_state = NPU2_TRAIN_PRBS31; - else if (!strcmp(state, "none")) - npu2_ocapi_training_state = NPU2_TRAIN_NONE; - else - prlog(PR_WARNING, - "OCAPI: invalid training state in NVRAM: %s\n", - state); - } -} - -void probe_npu2_opencapi(void) -{ - struct dt_node *np_npu; - - read_nvram_training_state(); - - dt_for_each_compatible(dt_root, np_npu, "ibm,power9-npu") - npu2_opencapi_probe(np_npu); + return -1; } static const struct phb_ops npu2_opencapi_ops = { diff --git a/hw/npu2.c b/hw/npu2.c index be1c304420fc..419dd37a24b1 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -1357,44 +1357,13 @@ static void assign_mmio_bars(uint64_t gcid, uint32_t scom, uint64_t reg[2], uint } /* - * Probe NPU2 device node and create PCI root device node - * accordingly. The NPU2 device node should specify number - * of links and xscom base address to access links. + * Set up NPU for NVLink and create PCI root device node + * accordingly. */ -static void npu2_probe_phb(struct dt_node *dn) +int npu2_nvlink_init_npu(struct npu2 *npu) { - struct proc_chip *proc_chip; - struct dt_node *np, *link; - bool ocapi_detected = false, nvlink_detected = false; - uint32_t gcid, scom, index, phb_index, links; + struct dt_node *np; uint64_t reg[2], mm_win[2], val; - char *path; - - /* Abort if any OpenCAPI links detected */ - dt_for_each_compatible(dn, link, "ibm,npu-link") { - if (npu2_dt_link_dev_type(link) == NPU2_DEV_TYPE_OPENCAPI) - ocapi_detected = true; - else - nvlink_detected = true; - } - - if (ocapi_detected && nvlink_detected) { - prlog(PR_ERR, "NPU: NVLink and OpenCAPI devices on same chip not supported\n"); - return; - } else if (ocapi_detected) { - prlog(PR_INFO, "NPU: OpenCAPI link configuration detected, not initialising NVLink\n"); - return; - } - - /* Retrieve chip id */ - path = dt_get_path(dn); - gcid = dt_get_chip_id(dn); - proc_chip = get_chip(gcid); - assert(proc_chip); - if ((proc_chip->ec_level & 0xf0) > 0x20) { - prerror("NPU: unsupported ec level on Chip 0x%x!\n", gcid); - return; - } /* TODO: Clean this up with register names, etc. when we get * time. This just turns NVLink mode on in each brick and should @@ -1403,64 +1372,56 @@ static void npu2_probe_phb(struct dt_node *dn) * * Obviously if the year is now 2020 that didn't happen and you * should fix this :-) */ - xscom_write_mask(gcid, 0x5011000, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011030, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011060, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011090, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011200, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011230, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011260, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011290, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011400, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011430, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011460, PPC_BIT(58), PPC_BIT(58)); - xscom_write_mask(gcid, 0x5011490, PPC_BIT(58), PPC_BIT(58)); - - xscom_write_mask(gcid, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); - xscom_write_mask(gcid, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(gcid, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); - xscom_write_mask(gcid, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x5011000, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011030, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011060, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011090, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011200, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011230, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011260, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011290, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011400, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011430, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011460, PPC_BIT(58), PPC_BIT(58)); + xscom_write_mask(npu->chip_id, 0x5011490, PPC_BIT(58), PPC_BIT(58)); + + xscom_write_mask(npu->chip_id, 0x50110c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50112c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50114c0, PPC_BIT(53), PPC_BIT(53)); + xscom_write_mask(npu->chip_id, 0x50110f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x50112f1, PPC_BIT(41), PPC_BIT(41)); + xscom_write_mask(npu->chip_id, 0x50114f1, PPC_BIT(41), PPC_BIT(41)); val = NPU2_NTL_MISC_CFG2_BRICK_ENABLE | NPU2_NTL_MISC_CFG2_NDL_TX_PARITY_ENA | NPU2_NTL_MISC_CFG2_NDL_PRI_PARITY_ENA | NPU2_NTL_MISC_CFG2_RCV_CREDIT_OVERFLOW_ENA; - xscom_write_mask(gcid, 0x5011110, val, val); - xscom_write_mask(gcid, 0x5011130, val, val); - xscom_write_mask(gcid, 0x5011310, val, val); - xscom_write_mask(gcid, 0x5011330, val, val); - xscom_write_mask(gcid, 0x5011510, val, val); - xscom_write_mask(gcid, 0x5011530, val, val); + xscom_write_mask(npu->chip_id, 0x5011110, val, val); + xscom_write_mask(npu->chip_id, 0x5011130, val, val); + xscom_write_mask(npu->chip_id, 0x5011310, val, val); + xscom_write_mask(npu->chip_id, 0x5011330, val, val); + xscom_write_mask(npu->chip_id, 0x5011510, val, val); + xscom_write_mask(npu->chip_id, 0x5011530, val, val); val = PPC_BIT(6) | PPC_BIT(7) | PPC_BIT(11); - xscom_write_mask(gcid, 0x5011009, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011039, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011069, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011099, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011209, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011239, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011269, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011299, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011409, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011439, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011469, val, PPC_BITMASK(6,11)); - xscom_write_mask(gcid, 0x5011499, val, PPC_BITMASK(6,11)); - - index = dt_prop_get_u32(dn, "ibm,npu-index"); - phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); - links = dt_prop_get_u32(dn, "ibm,npu-links"); - prlog(PR_INFO, "NPU: Chip %d Found NPU2#%d (%d links) at %s\n", - gcid, index, links, path); - free(path); - - /* Retrieve scom base address */ - scom = dt_get_address(dn, 0, NULL); - prlog(PR_INFO, " SCOM Base: %08x\n", scom); + xscom_write_mask(npu->chip_id, 0x5011009, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011039, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011069, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011099, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011209, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011239, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011269, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011299, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011409, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011439, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011469, val, PPC_BITMASK(6,11)); + xscom_write_mask(npu->chip_id, 0x5011499, val, PPC_BITMASK(6,11)); /* Reassign the BARs */ - assign_mmio_bars(gcid, scom, reg, mm_win); + assign_mmio_bars(npu->chip_id, npu->xscom_base, reg, mm_win); + npu->regs = (void *)reg[0]; + npu->mm_base = mm_win[0]; + npu->mm_size = mm_win[1]; if (reg[0] && reg[1]) prlog(PR_INFO, " Global MMIO BAR: %016llx (%lldMB)\n", @@ -1477,17 +1438,21 @@ static void npu2_probe_phb(struct dt_node *dn) "ibm,ioda2-npu2-phb"); dt_add_property_strings(np, "device_type", "pciex"); dt_add_property(np, "reg", reg, sizeof(reg)); - dt_add_property_cells(np, "ibm,phb-index", phb_index); - dt_add_property_cells(np, "ibm,npu-index", index); - dt_add_property_cells(np, "ibm,chip-id", gcid); - dt_add_property_cells(np, "ibm,xscom-base", scom); - dt_add_property_cells(np, "ibm,npcq", dn->phandle); - dt_add_property_cells(np, "ibm,links", links); + dt_add_property_cells(np, "ibm,phb-index", npu->phb_index); + dt_add_property_cells(np, "ibm,npu-index", npu->index); + dt_add_property_cells(np, "ibm,chip-id", npu->chip_id); + dt_add_property_cells(np, "ibm,xscom-base", npu->xscom_base); + dt_add_property_cells(np, "ibm,npcq", npu->dt_node->phandle); + dt_add_property_cells(np, "ibm,links", npu->total_devices); dt_add_property(np, "ibm,mmio-window", mm_win, sizeof(mm_win)); dt_add_property_cells(np, "ibm,phb-diag-data-size", 0); /* Disable fast reboot - not currently supported */ disable_fast_reboot("NVLink device enabled"); + + npu2_nvlink_create_phb(npu, np); + + return 0; } static uint32_t npu2_populate_pcie_cap(struct npu2_dev *dev, @@ -1949,49 +1914,25 @@ static void npu2_setup_irqs(struct npu2 *p) npu2_write(p, reg, val); } -static void npu2_create_phb(struct dt_node *dn) +void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn) { - const struct dt_property *prop; - struct npu2 *p; struct pci_slot *slot; - uint32_t links; - void *pmem; - - /* Retrieve number of devices */ - links = dt_prop_get_u32(dn, "ibm,links"); - pmem = zalloc(sizeof(struct npu2) + links * sizeof(struct npu2_dev)); - assert(pmem); - - /* Populate PHB */ - p = pmem; - p->index = dt_prop_get_u32(dn, "ibm,phb-index"); - p->chip_id = dt_prop_get_u32(dn, "ibm,chip-id"); - p->xscom_base = dt_prop_get_u32(dn, "ibm,xscom-base"); - p->total_devices = links; - p->regs = (void *)dt_get_address(dn, 0, NULL); - - prop = dt_require_property(dn, "ibm,mmio-window", -1); - assert(prop->len >= (2 * sizeof(uint64_t))); - p->mm_base = ((const uint64_t *)prop->prop)[0]; - p->mm_size = ((const uint64_t *)prop->prop)[1]; - - p->devices = pmem + sizeof(struct npu2); /* Generic PHB */ - p->phb_nvlink.dt_node = dn; - p->phb_nvlink.ops = &npu_ops; - p->phb_nvlink.phb_type = phb_type_npu_v2; - init_lock(&p->lock); - init_lock(&p->phb_nvlink.lock); - list_head_init(&p->phb_nvlink.devices); - list_head_init(&p->phb_nvlink.virt_devices); - - npu2_setup_irqs(p); - npu2_populate_devices(p, dn); - npu2_add_interrupt_map(p, dn); - npu2_add_phb_properties(p); - - slot = npu2_slot_create(&p->phb_nvlink); + npu->phb_nvlink.dt_node = dn; + npu->phb_nvlink.ops = &npu_ops; + npu->phb_nvlink.phb_type = phb_type_npu_v2; + init_lock(&npu->lock); + init_lock(&npu->phb_nvlink.lock); + list_head_init(&npu->phb_nvlink.devices); + list_head_init(&npu->phb_nvlink.virt_devices); + + npu2_setup_irqs(npu); + npu2_populate_devices(npu, dn); + npu2_add_interrupt_map(npu, dn); + npu2_add_phb_properties(npu); + + slot = npu2_slot_create(&npu->phb_nvlink); if (!slot) { /** @@ -2002,41 +1943,10 @@ static void npu2_create_phb(struct dt_node *dn) prlog(PR_ERR, "NPU: Cannot create PHB slot\n"); } - pci_register_phb(&p->phb_nvlink, OPAL_DYNAMIC_PHB_ID); - - npu2_init_ioda_cache(p); - npu2_hw_init(p); -} - -void probe_npu2(void) -{ - struct proc_chip *chip = next_chip(NULL); - struct dt_node *np; - const char *zcal; - - /* Abort if we're running on DD1 */ - if (chip && - (chip->type == PROC_CHIP_P9_NIMBUS || - chip->type == PROC_CHIP_P9_CUMULUS) && - (chip->ec_level & 0xf0) == 0x10) { - prlog(PR_INFO, "NPU: DD1 not supported\n"); - return; - } - - /* Check for a zcal override */ - zcal = nvram_query("nv_zcal_override"); - if (zcal) { - nv_zcal_nominal = atoi(zcal); - prlog(PR_WARNING, "NPU: Using ZCAL impedance override = %d\n", nv_zcal_nominal); - } - - /* Scan NPU2 XSCOM nodes */ - dt_for_each_compatible(dt_root, np, "ibm,power9-npu") - npu2_probe_phb(np); + pci_register_phb(&npu->phb_nvlink, OPAL_DYNAMIC_PHB_ID); - /* Scan newly created PHB nodes */ - dt_for_each_compatible(dt_root, np, "ibm,power9-npu-pciex") - npu2_create_phb(np); + npu2_init_ioda_cache(npu); + npu2_hw_init(npu); } /* diff --git a/include/npu2.h b/include/npu2.h index 10742031ec0f..0e07d77b62c5 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -167,6 +167,7 @@ struct npu2 { /* NVLink */ struct phb phb_nvlink; + uint32_t phb_index; }; static inline struct npu2 *phb_to_npu2_nvlink(struct phb *phb) @@ -193,6 +194,10 @@ static inline struct phb *npu2_dev_to_phb(struct npu2_dev *ndev) } } +int npu2_opencapi_init_npu(struct npu2 *npu); +int npu2_nvlink_init_npu(struct npu2 *npu); +void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn); + enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link); void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val); uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg); diff --git a/include/skiboot.h b/include/skiboot.h index bba76c12c39e..f2818b1a2326 100644 --- a/include/skiboot.h +++ b/include/skiboot.h @@ -201,7 +201,6 @@ extern int preload_capp_ucode(void); extern void preload_io_vpd(void); extern void probe_npu(void); extern void probe_npu2(void); -extern void probe_npu2_opencapi(void); extern void uart_init(void); extern void mbox_init(void); extern void early_uart_init(void); From patchwork Mon Sep 3 04:53:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965232 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423d1G1ZJhz9s2P for ; Mon, 3 Sep 2018 14:54:42 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423d1G0Bb8zF38m for ; Mon, 3 Sep 2018 14:54:42 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czs6YmDzF38Q for ; Mon, 3 Sep 2018 14:53:29 +1000 (AEST) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834n2YP085974 for ; Mon, 3 Sep 2018 00:53:27 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m8vjkuac7-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:27 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:24 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:22 +0100 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rLPC39845894 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:21 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 445A64C040; Mon, 3 Sep 2018 07:53:17 +0100 (BST) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5A9F94C046; Mon, 3 Sep 2018 07:53:16 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 3 Sep 2018 07:53:16 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 178F6A037F; Mon, 3 Sep 2018 14:53:18 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:53:02 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-0028-0000-0000-000002F331DA X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-0029-0000-0000-000023ACA2A6 Message-Id: <2adf22c30d6b9eded8a6cb2cdbf3302d63c8d9e4.1535950380.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 4/6] hw/npu2, platform: Add NPU2 platform device detection callback X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" There is no standardised way to determine the presence and type of devices connected to an NPU on POWER9. Currently, we hardcode device types based on platform type (as no platform currently supports both OpenCAPI and NVLink), and for OpenCAPI platforms we use I2C to detect presence. Witherspoon (and potentially other platforms later on) supports both NVLink and OpenCAPI, and additionally uses SXM2 connectors which can carry more than one link, rather than the SlimSAS connectors used for OpenCAPI on Zaius and ZZ. This necessitates some special handling. Add a platform callback for NPU device detection. In a later patch, we will use this to implement Witherspoon-specific device detection. For now, add a Witherspoon stub that sets all links to NVLink (i.e. current behaviour). Move the existing I2C-based presence detection for OpenCAPI devices on Zaius/ZZ into common code, which we use by default for platforms which do not define a callback. Clean up the use of the ibm,npu-link-type property, which will now be exposed solely for debugging and not consumed internally. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat --- v1->v2: - turn the "generic" i2c detection code into a callback (Alistair) v2->v3: - Abort NPU init if we can't find the device presence I2C port (Fred) - define device detection callback for ZZ (Fred) - add comment to explain OpenCAPI slot presence function (Fred) --- core/platform.c | 2 +- hdata/spira.c | 1 +- hw/npu2-common.c | 112 +++++++++++++++++++++++++++++----- hw/npu2-opencapi.c | 93 ++++------------------------ include/npu2.h | 14 +++- include/platform.h | 4 +- platforms/astbmc/witherspoon.c | 10 +++- platforms/astbmc/zaius.c | 3 +- platforms/ibm-fsp/zz.c | 2 +- 9 files changed, 144 insertions(+), 97 deletions(-) diff --git a/core/platform.c b/core/platform.c index b32cbf5ce9d1..4b3eaa4bd1d1 100644 --- a/core/platform.c +++ b/core/platform.c @@ -25,6 +25,7 @@ #include #include #include +#include #include bool manufacturing_mode = false; @@ -204,6 +205,7 @@ static struct platform generic_platform = { .start_preload_resource = generic_start_preload_resource, .resource_loaded = generic_resource_loaded, .ocapi = &generic_ocapi, + .npu2_device_detect = npu2_i2c_presence_detect, /* Assumes ZZ */ }; const struct bmc_platform *bmc_platform = &generic_bmc; diff --git a/hdata/spira.c b/hdata/spira.c index c820c4daf26e..189584d5ddb6 100644 --- a/hdata/spira.c +++ b/hdata/spira.c @@ -1490,7 +1490,6 @@ static void add_npu(struct dt_node *xscom, const struct HDIF_array_hdr *links, } dt_add_property_string(node, "compatible", "ibm,npu-link"); - dt_add_property_string(node, "ibm,npu-link-type", "nvlink"); dt_add_property_cells(node, "reg", link_count); dt_add_property_cells(node, "ibm,npu-link-index", link_count); dt_add_property_cells(node, "ibm,workbook-link-id", link_id); diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 476fc0683d56..15da3d316c24 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -21,19 +21,7 @@ #include #include #include - -enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link) -{ - const char *link_type = dt_prop_get(link, "ibm,npu-link-type") ?: - "unknown"; - if (streq(link_type, "nvlink")) { - return NPU2_DEV_TYPE_NVLINK; - } else if (streq(link_type, "opencapi")) { - return NPU2_DEV_TYPE_OPENCAPI; - } else { - return NPU2_DEV_TYPE_UNKNOWN; - } -} +#include /* * We use the indirect method because it uses the same addresses as @@ -109,12 +97,62 @@ void npu2_write_mask_4b(struct npu2 *p, uint64_t reg, uint32_t val, uint32_t mas (uint64_t)new_val << 32); } +static bool _i2c_presence_detect(struct npu2_dev *dev) +{ + uint8_t state, data; + int rc; + + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, + platform.ocapi->i2c_presence_addr, + SMBUS_READ, 0, 1, + &state, 1, 120); + if (rc) { + OCAPIERR(dev, "error detecting link presence: %d\n", rc); + return true; /* assume link exists */ + } + + OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); + + switch (dev->link_index) { + case 2: + data = platform.ocapi->i2c_presence_odl0; + break; + case 3: + data = platform.ocapi->i2c_presence_odl1; + break; + default: + OCAPIERR(dev, "presence detection on invalid link\n"); + return true; + } + /* Presence detect bits are active low */ + return !(state & data); +} + +/* + * A default presence detection implementation for platforms like ZZ and Zaius + * that don't implement their own. Assumes all devices found will be OpenCAPI. + */ +void npu2_i2c_presence_detect(struct npu2 *npu) +{ + struct npu2_dev *dev; + assert(platform.ocapi); + for (int i = 0; i < npu->total_devices; i++) { + dev = &npu->devices[i]; + if (platform.ocapi->force_presence || + _i2c_presence_detect(dev)) + dev->type = NPU2_DEV_TYPE_OPENCAPI; + else + dev->type = NPU2_DEV_TYPE_UNKNOWN; + } +} + static struct npu2 *setup_npu(struct dt_node *dn) { struct npu2 *npu; struct npu2_dev *dev; struct dt_node *np; uint32_t num_links; + char port_name[17]; void *npumem; char *path; int gcid; @@ -138,6 +176,28 @@ static struct npu2 *setup_npu(struct dt_node *dn) npu->chip_id = gcid; npu->xscom_base = dt_get_address(dn, 0, NULL); npu->phb_index = dt_prop_get_u32(dn, "ibm,phb-index"); + + if (platform.ocapi) { + /* Find I2C port for handling device presence/reset */ + snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", + gcid, platform.ocapi->i2c_engine, + platform.ocapi->i2c_port); + prlog(PR_DEBUG, "NPU: Looking for I2C port %s\n", port_name); + + dt_for_each_compatible(dt_root, np, "ibm,power9-i2c-port") { + if (streq(port_name, dt_prop_get(np, "ibm,port-name"))) { + npu->i2c_port_id_ocapi = dt_prop_get_u32(np, "ibm,opal-id"); + break; + } + } + + if (!npu->i2c_port_id_ocapi) { + prlog(PR_ERR, "NPU: Couldn't find I2C port %s\n", + port_name); + goto failed; + } + } + npu->devices = npumem + sizeof(struct npu2); dt_for_each_compatible(dn, np, "ibm,npu-link") { @@ -146,7 +206,8 @@ static struct npu2 *setup_npu(struct dt_node *dn) dev->link_index = dt_prop_get_u32(np, "ibm,npu-link-index"); /* May be overridden by platform presence detection */ dev->brick_index = dev->link_index; - dev->type = npu2_dt_link_dev_type(np); + /* Will be overridden by presence detection */ + dev->type = NPU2_DEV_TYPE_UNKNOWN; dev->npu = npu; dev->dt_node = np; dev->pl_xscom_base = dt_prop_get_u64(np, "ibm,npu-phy"); @@ -161,6 +222,12 @@ static struct npu2 *setup_npu(struct dt_node *dn) prlog(PR_INFO, " SCOM Base: %08llx\n", npu->xscom_base); free(path); return npu; + +failed: + prlog(PR_ERR, "NPU: Chip %d NPU setup failed\n", gcid); + free(path); + free(npu); + return NULL; } static void setup_devices(struct npu2 *npu) @@ -177,13 +244,22 @@ static void setup_devices(struct npu2 *npu) switch (dev->type) { case NPU2_DEV_TYPE_NVLINK: nvlink_detected = true; + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "nvlink"); break; case NPU2_DEV_TYPE_OPENCAPI: ocapi_detected = true; + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "opencapi"); break; default: prlog(PR_INFO, "NPU: Link %d device not present\n", npu->devices[i].link_index); + dt_add_property_strings(dev->dt_node, + "ibm,npu-link-type", + "unknown"); } } @@ -221,8 +297,16 @@ void probe_npu2(void) prlog(PR_WARNING, "NPU2: Using ZCAL impedance override = %d\n", nv_zcal_nominal); } + if (!platform.npu2_device_detect) { + prlog(PR_INFO, "NPU: Platform does not support NPU\n"); + return; + } + dt_for_each_compatible(dt_root, np, "ibm,power9-npu") { npu = setup_npu(np); + if (!npu) + continue; + platform.npu2_device_detect(npu); setup_devices(npu); } } diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index 270876f78aa3..b5a32dd32244 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -52,14 +52,6 @@ #include #include -#define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) -#define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) -#define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ - dev->npu->chip_id, dev->brick_index, ## a) - - #define NPU_IRQ_LEVELS 35 #define NPU_IRQ_LEVELS_XSL 23 #define MAX_PE_HANDLE ((1 << 15) - 1) @@ -842,7 +834,7 @@ static void assert_reset(struct npu2_dev *dev) * register and a pin is in output mode if its value is 0 */ data = ~pin; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x3, 1, &data, sizeof(data), 120); @@ -851,7 +843,7 @@ static void assert_reset(struct npu2_dev *dev) /* register 1 controls the signal, reset is active low */ data = ~pin; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x1, 1, &data, sizeof(data), 120); @@ -874,7 +866,7 @@ static void deassert_reset(struct npu2_dev *dev) int rc; data = 0xFF; - rc = i2c_request_send(dev->i2c_port_id_ocapi, + rc = i2c_request_send(dev->npu->i2c_port_id_ocapi, platform.ocapi->i2c_reset_addr, SMBUS_WRITE, 0x1, 1, &data, sizeof(data), 120); @@ -888,43 +880,6 @@ static void deassert_reset(struct npu2_dev *dev) } } -static bool i2c_presence_detect(struct npu2_dev *dev) -{ - uint8_t state, data; - int rc; - - /* - * Opencapi presence detection is done through i2c - * - * Lagrange platforms (ZZ, Zaius) use the same default mechanism. - * Witherspoon will need a specific implementation, TBD. - */ - rc = i2c_request_send(dev->i2c_port_id_ocapi, - platform.ocapi->i2c_presence_addr, - SMBUS_READ, 0, 1, - &state, 1, 120); - if (rc) { - OCAPIERR(dev, "error detecting link presence: %d\n", rc); - return true; /* assume link exists */ - } - - OCAPIDBG(dev, "I2C presence detect: 0x%x\n", state); - - switch (dev->brick_index) { // TODO(ajd): Link or brick index? - case 2: - data = platform.ocapi->i2c_presence_odl0; - break; - case 3: - data = platform.ocapi->i2c_presence_odl1; - break; - default: - OCAPIERR(dev, "presence detection on invalid link\n"); - return true; - } - /* Presence detect bits are active low */ - return !(state & data); -} - static void reset_odl(uint32_t gcid, struct npu2_dev *dev) { uint64_t reg, config_xscom; @@ -1015,17 +970,18 @@ static void start_training(uint32_t gcid, struct npu2_dev *dev) xscom_write(gcid, config_xscom, reg); } -static int64_t npu2_opencapi_get_presence_state(struct pci_slot *slot, +static int64_t npu2_opencapi_get_presence_state(struct pci_slot __unused *slot, uint8_t *val) { - bool present; - struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); - - if (platform.ocapi->force_presence) - present = true; - else - present = i2c_presence_detect(dev); - *val = present; + /* + * Presence detection for OpenCAPI is currently done at the start of + * NPU initialisation, and we only create slots if a device is present. + * As such we will never be asked to get the presence of a slot that's + * empty. + * + * This may change if we ever support hotplug down the track. + */ + *val = true; return OPAL_SUCCESS; } @@ -1578,9 +1534,8 @@ static void setup_debug_training_state(struct npu2_dev *dev) static void setup_device(struct npu2_dev *dev) { - struct dt_node *dn_phb, *dn; + struct dt_node *dn_phb; struct pci_slot *slot; - char port_name[17]; uint64_t mm_win[2]; /* Populate PHB device node */ @@ -1626,23 +1581,6 @@ static void setup_device(struct npu2_dev *dev) dev->bdfn = 0; dev->train_need_fence = false; dev->train_fenced = false; - /* Find I2C port for handling device reset */ - snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", - dev->npu->chip_id, platform.ocapi->i2c_engine, - platform.ocapi->i2c_port); - prlog(PR_DEBUG, "OCAPI: Looking for I2C port %s\n", port_name); - - dt_for_each_compatible(dt_root, dn, "ibm,power9-i2c-port") { - if (streq(port_name, dt_prop_get(dn, "ibm,port-name"))) { - dev->i2c_port_id_ocapi = dt_prop_get_u32(dn, "ibm,opal-id"); - break; - } - } - - if (!dev->i2c_port_id_ocapi) { - prlog(PR_ERR, "OCAPI: Couldn't find I2C port %s\n", port_name); - goto failed; - } /* TODO: Procedure 13.1.3.7 - AFU Memory Range BARs */ /* Procedure 13.1.3.8 - AFU MMIO Range BARs */ @@ -1667,9 +1605,6 @@ static void setup_device(struct npu2_dev *dev) } pci_register_phb(&dev->phb_ocapi, OPAL_DYNAMIC_PHB_ID); return; -failed: - dt_add_property_string(dn_phb, "status", "error"); - return; } static void read_nvram_training_state(void) diff --git a/include/npu2.h b/include/npu2.h index 0e07d77b62c5..1de963d19928 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -17,6 +17,7 @@ #ifndef __NPU2_H #define __NPU2_H +#include #include /* Debugging options */ @@ -36,6 +37,14 @@ #define NPU2DEVINF(p, fmt, a...) NPU2DEVLOG(PR_INFO, p, fmt, ##a) #define NPU2DEVERR(p, fmt, a...) NPU2DEVLOG(PR_ERR, p, fmt, ##a) +#define OCAPIDBG(dev, fmt, a...) prlog(PR_DEBUG, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) +#define OCAPIINF(dev, fmt, a...) prlog(PR_INFO, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) +#define OCAPIERR(dev, fmt, a...) prlog(PR_ERR, "OCAPI[%d:%d]: " fmt, \ + dev->npu->chip_id, dev->brick_index, ## a) + + /* Number of PEs supported */ #define NPU2_MAX_PE_NUM 16 #define NPU2_RESERVED_PE_NUM 15 @@ -136,7 +145,6 @@ struct npu2_dev { /* OpenCAPI */ struct phb phb_ocapi; - uint64_t i2c_port_id_ocapi; bool train_need_fence; bool train_fenced; }; @@ -168,6 +176,8 @@ struct npu2 { /* NVLink */ struct phb phb_nvlink; uint32_t phb_index; + + uint64_t i2c_port_id_ocapi; }; static inline struct npu2 *phb_to_npu2_nvlink(struct phb *phb) @@ -194,11 +204,11 @@ static inline struct phb *npu2_dev_to_phb(struct npu2_dev *ndev) } } +void npu2_i2c_presence_detect(struct npu2 *npu); int npu2_opencapi_init_npu(struct npu2 *npu); int npu2_nvlink_init_npu(struct npu2 *npu); void npu2_nvlink_create_phb(struct npu2 *npu, struct dt_node *dn); -enum npu2_dev_type npu2_dt_link_dev_type(struct dt_node *link); void npu2_write_4b(struct npu2 *p, uint64_t reg, uint32_t val); uint32_t npu2_read_4b(struct npu2 *p, uint64_t reg); void npu2_write(struct npu2 *p, uint64_t reg, uint64_t val); diff --git a/include/platform.h b/include/platform.h index 1a35a86a6f8b..efafbd2239b9 100644 --- a/include/platform.h +++ b/include/platform.h @@ -22,6 +22,7 @@ struct phb; struct pci_device; struct pci_slot; struct errorlog; +struct npu2; enum resource_id { RESOURCE_ID_KERNEL, @@ -76,6 +77,9 @@ struct platform { /* OpenCAPI platform-specific I2C information */ const struct platform_ocapi *ocapi; + /* NPU2 device detection */ + void (*npu2_device_detect)(struct npu2 *npu); + /* * Probe platform, return true on a match, called before * any allocation has been performed outside of the heap diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index d663709f8766..ce83ff9701d3 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "astbmc.h" #include "ast.h" @@ -153,6 +154,14 @@ static void witherspoon_pre_pci_fixup(void) phb4_pre_pci_fixup_witherspoon(); } +static void witherspoon_npu2_device_detect(struct npu2 *npu) +{ + /* Stub until we implement real device detection */ + for (int i = 0; i < npu->total_devices; i++) { + npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + } +} + /* The only difference between these is the PCI slot handling */ DECLARE_PLATFORM(witherspoon) = { @@ -170,4 +179,5 @@ DECLARE_PLATFORM(witherspoon) = { .terminate = ipmi_terminate, .pci_get_slot_info = dt_slot_get_slot_info, + .npu2_device_detect = witherspoon_npu2_device_detect, }; diff --git a/platforms/astbmc/zaius.c b/platforms/astbmc/zaius.c index 2f296c3679ac..267e6578bbe7 100644 --- a/platforms/astbmc/zaius.c +++ b/platforms/astbmc/zaius.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "astbmc.h" @@ -51,7 +52,6 @@ static void create_link(struct dt_node *npu, int group, int index) link = dt_new(npu, namebuf); dt_add_property_string(link, "compatible", "ibm,npu-link"); - dt_add_property_string(link, "ibm,npu-link-type", "opencapi"); dt_add_property_cells(link, "ibm,npu-link-index", index); switch (index) { @@ -153,4 +153,5 @@ DECLARE_PLATFORM(zaius) = { .exit = ipmi_wdt_final_reset, .terminate = ipmi_terminate, .ocapi = &zaius_ocapi, + .npu2_device_detect = npu2_i2c_presence_detect, }; diff --git a/platforms/ibm-fsp/zz.c b/platforms/ibm-fsp/zz.c index dc48768b8a96..7c717d3c7087 100644 --- a/platforms/ibm-fsp/zz.c +++ b/platforms/ibm-fsp/zz.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "ibm-fsp.h" #include "lxvpd.h" @@ -90,4 +91,5 @@ DECLARE_PLATFORM(zz) = { .sensor_read = ibm_fsp_sensor_read, .terminate = ibm_fsp_terminate, .ocapi = &zz_ocapi, + .npu2_device_detect = npu2_i2c_presence_detect, }; From patchwork Mon Sep 3 04:53:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965233 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423d1X6hD0z9s2P for ; Mon, 3 Sep 2018 14:54:56 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423d1X5GKbzF39Y for ; Mon, 3 Sep 2018 14:54:56 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czt14YFzF38k for ; Mon, 3 Sep 2018 14:53:29 +1000 (AEST) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834n2a9085981 for ; Mon, 3 Sep 2018 00:53:28 -0400 Received: from e06smtp04.uk.ibm.com (e06smtp04.uk.ibm.com [195.75.94.100]) by mx0a-001b2d01.pphosted.com with ESMTP id 2m8vjkuacm-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:27 -0400 Received: from localhost by e06smtp04.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:25 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp04.uk.ibm.com (192.168.101.134) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:23 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rMA341287768 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:22 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1388D11C050; Mon, 3 Sep 2018 07:53:16 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70E2511C058; Mon, 3 Sep 2018 07:53:15 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 3 Sep 2018 07:53:15 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 29C1EA03A3; Mon, 3 Sep 2018 14:53:18 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:53:03 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-0016-0000-0000-000001FF3615 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-0017-0000-0000-00003255C6CF Message-Id: <42ef573e4158dfe789ec9b70f5848e171f698924.1535950380.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 5/6] hw/npu2, platform: Restructure OpenCAPI i2c reset/presence pins X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" In platform_ocapi, we define i2c_{reset,presence}_odl{0,1} to specify the appropriate reset/presence GPIO pins for devices connected to ODL0 and ODL1 respectively. This is obviously wrong, because a device connected to brick 2 and a device connected to brick 4 are going to be different devices connected to different I2C pins, but rather conveniently we haven't had to deal with systems that can use the full 4 bricks as yet. Now that we're adding OpenCAPI support for Witherspoon, we should change this to specify pins separately for all 4 bricks. Replace i2c_{reset,presence}_odl{0,1} with i2c_{reset,presence}_brick{2,3,4,5} and update the presence detection code, device reset code, and existing platforms accordingly. Signed-off-by: Andrew Donnellan Reviewed-by: Frederic Barrat Reviewed-by: Oliver O'Halloran --- v1->v2: - rename the i2c fields from pin* to brick* to be clearer (Oliver) v2->v3: - rename the i2c fields in the commit message... (Oliver) v3->v4: - add missing break (Fred) --- core/platform.c | 24 ++++++++++++++---------- hw/npu2-common.c | 10 ++++++++-- hw/npu2-opencapi.c | 10 +++++++--- include/platform.h | 12 ++++++++---- platforms/astbmc/zaius.c | 22 +++++++++++++--------- platforms/ibm-fsp/zz.c | 24 ++++++++++++++---------- 6 files changed, 64 insertions(+), 38 deletions(-) diff --git a/core/platform.c b/core/platform.c index 4b3eaa4bd1d1..7985ce564509 100644 --- a/core/platform.c +++ b/core/platform.c @@ -172,21 +172,25 @@ static int generic_start_preload_resource(enum resource_id id, uint32_t subid, /* These values will work for a ZZ booted using BML */ const struct platform_ocapi generic_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ /* * The ZZs we typically use for BML/generic platform tend to * have old planars and presence detection is broken there, so * force presence. */ - .force_presence = true, - .odl_phy_swap = true, + .force_presence = true, + .odl_phy_swap = true, }; static struct bmc_platform generic_bmc = { diff --git a/hw/npu2-common.c b/hw/npu2-common.c index 15da3d316c24..9e392b879803 100644 --- a/hw/npu2-common.c +++ b/hw/npu2-common.c @@ -115,10 +115,16 @@ static bool _i2c_presence_detect(struct npu2_dev *dev) switch (dev->link_index) { case 2: - data = platform.ocapi->i2c_presence_odl0; + data = platform.ocapi->i2c_presence_brick2; break; case 3: - data = platform.ocapi->i2c_presence_odl1; + data = platform.ocapi->i2c_presence_brick3; + break; + case 4: + data = platform.ocapi->i2c_presence_brick4; + break; + case 5: + data = platform.ocapi->i2c_presence_brick5; break; default: OCAPIERR(dev, "presence detection on invalid link\n"); diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index b5a32dd32244..5e258e1aff07 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -816,12 +816,16 @@ static void assert_reset(struct npu2_dev *dev) switch (dev->brick_index) { case 2: - case 4: - pin = platform.ocapi->i2c_reset_odl0; + pin = platform.ocapi->i2c_reset_brick2; break; case 3: + pin = platform.ocapi->i2c_reset_brick3; + break; + case 4: + pin = platform.ocapi->i2c_reset_brick4; + break; case 5: - pin = platform.ocapi->i2c_reset_odl1; + pin = platform.ocapi->i2c_reset_brick5; break; default: assert(false); diff --git a/include/platform.h b/include/platform.h index efafbd2239b9..fee5a76cf9e8 100644 --- a/include/platform.h +++ b/include/platform.h @@ -50,11 +50,15 @@ struct platform_ocapi { uint8_t i2c_engine; /* I2C engine number */ uint8_t i2c_port; /* I2C port number */ uint8_t i2c_reset_addr; /* I2C address for reset */ - uint8_t i2c_reset_odl0; /* I2C pin to write to reset ODL0 */ - uint8_t i2c_reset_odl1; /* I2C pin to write to reset ODL1 */ + uint8_t i2c_reset_brick2; /* I2C pin to write to reset brick 2 */ + uint8_t i2c_reset_brick3; /* I2C pin to write to reset brick 3 */ + uint8_t i2c_reset_brick4; /* I2C pin to write to reset brick 4 */ + uint8_t i2c_reset_brick5; /* I2C pin to write to reset brick 5 */ uint8_t i2c_presence_addr; /* I2C address for presence detection */ - uint8_t i2c_presence_odl0; /* I2C mask for detection on ODL0 */ - uint8_t i2c_presence_odl1; /* I2C mask for detection on ODL1 */ + uint8_t i2c_presence_brick2; /* I2C pin to read for presence on brick 2 */ + uint8_t i2c_presence_brick3; /* I2C pin to read for presence on brick 3 */ + uint8_t i2c_presence_brick4; /* I2C pin to read for presence on brick 4 */ + uint8_t i2c_presence_brick5; /* I2C pin to read for presence on brick 5 */ bool force_presence; /* don't use i2c detection */ bool odl_phy_swap; /* Swap ODL1 to use brick 2 rather than * brick 1 lanes */ diff --git a/platforms/astbmc/zaius.c b/platforms/astbmc/zaius.c index 267e6578bbe7..0eed10c706d0 100644 --- a/platforms/astbmc/zaius.c +++ b/platforms/astbmc/zaius.c @@ -26,15 +26,19 @@ #include "astbmc.h" const struct platform_ocapi zaius_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ - .odl_phy_swap = true, + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ + .odl_phy_swap = true, }; #define NPU_BASE 0x5011000 diff --git a/platforms/ibm-fsp/zz.c b/platforms/ibm-fsp/zz.c index 7c717d3c7087..e54472699c59 100644 --- a/platforms/ibm-fsp/zz.c +++ b/platforms/ibm-fsp/zz.c @@ -30,20 +30,24 @@ /* We don't yet create NPU device nodes on ZZ, but these values are correct */ const struct platform_ocapi zz_ocapi = { - .i2c_engine = 1, - .i2c_port = 4, - .i2c_reset_addr = 0x20, - .i2c_reset_odl0 = (1 << 1), - .i2c_reset_odl1 = (1 << 6), - .i2c_presence_addr = 0x20, - .i2c_presence_odl0 = (1 << 2), /* bottom connector */ - .i2c_presence_odl1 = (1 << 7), /* top connector */ + .i2c_engine = 1, + .i2c_port = 4, + .i2c_reset_addr = 0x20, + .i2c_reset_brick2 = (1 << 1), + .i2c_reset_brick3 = (1 << 6), + .i2c_reset_brick4 = 0, /* unused */ + .i2c_reset_brick5 = 0, /* unused */ + .i2c_presence_addr = 0x20, + .i2c_presence_brick2 = (1 << 2), /* bottom connector */ + .i2c_presence_brick3 = (1 << 7), /* top connector */ + .i2c_presence_brick4 = 0, /* unused */ + .i2c_presence_brick5 = 0, /* unused */ /* * i2c presence detection is broken on ZZ planar < v4 so we * force the presence until all our systems are upgraded */ - .force_presence = true, - .odl_phy_swap = true, + .force_presence = true, + .odl_phy_swap = true, }; static bool zz_probe(void) From patchwork Mon Sep 3 04:53:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 965234 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 423d1t71kNz9s2P for ; Mon, 3 Sep 2018 14:55:14 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 423d1t4smVzF13P for ; Mon, 3 Sep 2018 14:55:14 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=au1.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 423czt2q5mzF38W for ; Mon, 3 Sep 2018 14:53:29 +1000 (AEST) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w834n0Dh029968 for ; Mon, 3 Sep 2018 00:53:28 -0400 Received: from e06smtp05.uk.ibm.com (e06smtp05.uk.ibm.com [195.75.94.101]) by mx0b-001b2d01.pphosted.com with ESMTP id 2m8u1snq7a-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 03 Sep 2018 00:53:27 -0400 Received: from localhost by e06smtp05.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 3 Sep 2018 05:53:26 +0100 Received: from b06cxnps3075.portsmouth.uk.ibm.com (9.149.109.195) by e06smtp05.uk.ibm.com (192.168.101.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Mon, 3 Sep 2018 05:53:23 +0100 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w834rMhH42598604 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 3 Sep 2018 04:53:22 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 30711A404D; Mon, 3 Sep 2018 07:53:16 +0100 (BST) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8EFEFA4053; Mon, 3 Sep 2018 07:53:15 +0100 (BST) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 3 Sep 2018 07:53:15 +0100 (BST) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 3D36EA03A5; Mon, 3 Sep 2018 14:53:18 +1000 (AEST) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Mon, 3 Sep 2018 14:53:04 +1000 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18090304-0020-0000-0000-000002C02614 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18090304-0021-0000-0000-0000210D4208 Message-Id: <29f72c30ea43f52c1cdd63e9882e017111b91548.1535950380.git-series.andrew.donnellan@au1.ibm.com> MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-09-03_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809030053 Subject: [Skiboot] [PATCH v4 6/6] platforms/astbmc/witherspoon: Implement OpenCAPI support X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: arbab@linux.ibm.com, fbarrat@linux.vnet.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" OpenCAPI on Witherspoon is slightly more involved than on Zaius and ZZ, due to the OpenCAPI links using the SXM2 connectors that are used for NVLink GPUs. This patch adds the regular OpenCAPI platform information, and also a Witherspoon-specific presence detection callback that uses the previously added OCC GPU presence detection to figure out the device types plugged into each SXM2 socket. The SXM2 connectors are capable of carrying 2 OpenCAPI links, and future OpenCAPI devices are expected to make use of this. However, we don't yet support ganged links and the various implications that has for handling things like device reset, so for now, we only enable 1 brick per device. Signed-off-by: Andrew Donnellan Acked-by: Reza Arbab Reviewed-by: Alistair Popple Reviewed-by: Frederic Barrat --- v1->v2: - update platform field names v2->v3: - refactor link type/index setting (Fred/Alistair) - add explanatory comment for i2c reset settings (Fred) v3->v4: - break out of loop once device found in set_link_details() (Fred) --- platforms/astbmc/witherspoon.c | 205 +++++++++++++++++++++++++++++++++- 1 file changed, 202 insertions(+), 3 deletions(-) diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index ce83ff9701d3..77a01d2d979d 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -28,10 +28,18 @@ #include #include #include +#include +#include #include "astbmc.h" #include "ast.h" +static enum { + WITHERSPOON_TYPE_UNKNOWN, + WITHERSPOON_TYPE_SEQUOIA, + WITHERSPOON_TYPE_REDBUD +} witherspoon_type; + /* * HACK: Hostboot doesn't export the correct data for the system VPD EEPROM * for this system. So we need to work around it here. @@ -50,8 +58,35 @@ static void vpd_dt_fixup(void) } } +static void witherspoon_create_ocapi_i2c_bus(void) +{ + struct dt_node *xscom, *i2cm, *i2c_bus; + prlog(PR_DEBUG, "OCAPI: Adding I2C bus device node for OCAPI reset\n"); + dt_for_each_compatible(dt_root, xscom, "ibm,xscom") { + i2cm = dt_find_by_name(xscom, "i2cm@a1000"); + if (!i2cm) { + prlog(PR_ERR, "OCAPI: Failed to add I2C bus device node\n"); + continue; + } + + if (dt_find_by_name(i2cm, "i2c-bus@4")) + continue; + + i2c_bus = dt_new_addr(i2cm, "i2c-bus", 4); + dt_add_property_cells(i2c_bus, "reg", 4); + dt_add_property_cells(i2c_bus, "bus-frequency", 0x61a80); + dt_add_property_strings(i2c_bus, "compatible", + "ibm,opal-i2c", "ibm,power8-i2c-port", + "ibm,power9-i2c-port"); + } +} + static bool witherspoon_probe(void) { + struct dt_node *np; + int highest_gpu_group_id = 0; + int gpu_group_id; + if (!dt_node_is_compatible(dt_root, "ibm,witherspoon")) return false; @@ -63,6 +98,26 @@ static bool witherspoon_probe(void) vpd_dt_fixup(); + witherspoon_create_ocapi_i2c_bus(); + + dt_for_each_compatible(dt_root, np, "ibm,npu-link") { + gpu_group_id = dt_prop_get_u32(np, "ibm,npu-group-id"); + if (gpu_group_id > highest_gpu_group_id) + highest_gpu_group_id = gpu_group_id; + }; + + switch (highest_gpu_group_id) { + case 1: + witherspoon_type = WITHERSPOON_TYPE_REDBUD; + break; + case 2: + witherspoon_type = WITHERSPOON_TYPE_SEQUOIA; + break; + default: + witherspoon_type = WITHERSPOON_TYPE_UNKNOWN; + prlog(PR_NOTICE, "PLAT: Unknown Witherspoon variant detected\n"); + } + return true; } @@ -154,14 +209,157 @@ static void witherspoon_pre_pci_fixup(void) phb4_pre_pci_fixup_witherspoon(); } -static void witherspoon_npu2_device_detect(struct npu2 *npu) +static void set_link_details(struct npu2 *npu, uint32_t link_index, + uint32_t brick_index, enum npu2_dev_type type) { - /* Stub until we implement real device detection */ + struct npu2_dev *dev = NULL; for (int i = 0; i < npu->total_devices; i++) { - npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + if (npu->devices[i].link_index == link_index) { + dev = &npu->devices[i]; + break; + } + } + if (!dev) { + prlog(PR_ERR, "PLAT: Could not find NPU link index %d\n", + link_index); + return; } + dev->brick_index = brick_index; + dev->type = type; } +static void witherspoon_npu2_device_detect(struct npu2 *npu) +{ + struct proc_chip *chip; + uint8_t state; + uint64_t i2c_port_id = 0; + char port_name[17]; + struct dt_node *dn; + int rc; + + bool gpu0_present, gpu1_present; + + if (witherspoon_type != WITHERSPOON_TYPE_REDBUD) { + prlog(PR_DEBUG, "PLAT: Setting all NPU links to NVLink, OpenCAPI only supported on Redbud\n"); + for (int i = 0; i < npu->total_devices; i++) { + npu->devices[i].type = NPU2_DEV_TYPE_NVLINK; + } + return; + } + assert(npu->total_devices == 6); + + chip = get_chip(npu->chip_id); + + /* Find I2C port */ + snprintf(port_name, sizeof(port_name), "p8_%08x_e%dp%d", + chip->id, platform.ocapi->i2c_engine, + platform.ocapi->i2c_port); + dt_for_each_compatible(dt_root, dn, "ibm,power9-i2c-port") { + if (streq(port_name, dt_prop_get(dn, "ibm,port-name"))) { + i2c_port_id = dt_prop_get_u32(dn, "ibm,opal-id"); + break; + } + } + + if (!i2c_port_id) { + prlog(PR_ERR, "PLAT: Could not find NPU presence I2C port\n"); + return; + } + + gpu0_present = occ_get_gpu_presence(chip, 0); + if (gpu0_present) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 slot present\n", chip->id); + } + + gpu1_present = occ_get_gpu_presence(chip, 1); + if (gpu1_present) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 slot present\n", chip->id); + } + + /* Set pins to input */ + state = 0xff; + rc = i2c_request_send(i2c_port_id, + platform.ocapi->i2c_presence_addr, SMBUS_WRITE, 3, + 1, &state, 1, 120); + if (rc) + goto i2c_failed; + + /* Read the presence value */ + state = 0x00; + rc = i2c_request_send(i2c_port_id, + platform.ocapi->i2c_presence_addr, SMBUS_READ, 0, + 1, &state, 1, 120); + if (rc) + goto i2c_failed; + + if (gpu0_present) { + if (state & (1 << 0)) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is OpenCAPI\n", + chip->id); + /* + * On witherspoon, bricks 2 and 3 are connected to + * the lanes matching links 1 and 0 in OpenCAPI mode. + */ + set_link_details(npu, 0, 3, NPU2_DEV_TYPE_OPENCAPI); + /* We current don't support using the second link */ + set_link_details(npu, 1, 2, NPU2_DEV_TYPE_UNKNOWN); + } else { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#0 is NVLink\n", + chip->id); + set_link_details(npu, 0, 0, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 1, 1, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 2, 2, NPU2_DEV_TYPE_NVLINK); + } + } + + if (gpu1_present) { + if (state & (1 << 1)) { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is OpenCAPI\n", + chip->id); + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_OPENCAPI); + /* We current don't support using the second link */ + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_UNKNOWN); + } else { + prlog(PR_DEBUG, "PLAT: Chip %d GPU#1 is NVLink\n", + chip->id); + set_link_details(npu, 3, 3, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 4, 4, NPU2_DEV_TYPE_NVLINK); + set_link_details(npu, 5, 5, NPU2_DEV_TYPE_NVLINK); + } + } + + return; + +i2c_failed: + prlog(PR_ERR, "PLAT: NPU device type detection failed, rc=%d\n", rc); + return; +} + +const struct platform_ocapi witherspoon_ocapi = { + .i2c_engine = 1, + .i2c_port = 4, + .odl_phy_swap = false, + .i2c_reset_addr = 0x20, + /* + * Witherspoon uses SXM2 connectors, carrying 2 OCAPI links + * over a single connector - hence each pair of bricks shares + * the same pin for resets. We currently only support using + * bricks 3 and 4, among other reasons because we can't handle + * a reset on one link causing the other link to reset as + * well. + */ + .i2c_reset_brick2 = 1 << 0, + .i2c_reset_brick3 = 1 << 0, + .i2c_reset_brick4 = 1 << 1, + .i2c_reset_brick5 = 1 << 1, + .i2c_presence_addr = 0x20, + /* unused, we do this in custom presence detect */ + .i2c_presence_brick2 = 0, + .i2c_presence_brick3 = 0, + .i2c_presence_brick4 = 0, + .i2c_presence_brick5 = 0, +}; + /* The only difference between these is the PCI slot handling */ DECLARE_PLATFORM(witherspoon) = { @@ -179,5 +377,6 @@ DECLARE_PLATFORM(witherspoon) = { .terminate = ipmi_terminate, .pci_get_slot_info = dt_slot_get_slot_info, + .ocapi = &witherspoon_ocapi, .npu2_device_detect = witherspoon_npu2_device_detect, };