From patchwork Wed Feb 21 05:59:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Donnellan X-Patchwork-Id: 875952 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmRjp4nysz9ryl for ; Wed, 21 Feb 2018 17:03:10 +1100 (AEDT) Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmRjp24L2zF0sV for ; Wed, 21 Feb 2018 17:03:10 +1100 (AEDT) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=au1.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=andrew.donnellan@au1.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmRf56wY8zF0mC for ; Wed, 21 Feb 2018 16:59:57 +1100 (AEDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w1L5xrTG107630 for ; Wed, 21 Feb 2018 00:59:55 -0500 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0a-001b2d01.pphosted.com with ESMTP id 2g8x03gqtt-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 21 Feb 2018 00:59:54 -0500 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 21 Feb 2018 05:59:45 -0000 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 21 Feb 2018 05:59:42 -0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w1L5xfOc48234654; Wed, 21 Feb 2018 05:59:41 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2D10DA4148; Wed, 21 Feb 2018 05:52:53 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8C4CEA414F; Wed, 21 Feb 2018 05:52:52 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 21 Feb 2018 05:52:52 +0000 (GMT) Received: from intelligence.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 27411A03A3; Wed, 21 Feb 2018 16:59:38 +1100 (AEDT) From: Andrew Donnellan To: skiboot@lists.ozlabs.org Date: Wed, 21 Feb 2018 16:59:18 +1100 X-Mailer: git-send-email 2.11.0 In-Reply-To: References: In-Reply-To: References: X-TM-AS-GCONF: 00 x-cbid: 18022105-0020-0000-0000-000003F9E6CE X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18022105-0021-0000-0000-0000428DF670 Message-Id: <28531d9c36b0cb7c2b76bc725433a637906b61ae.1519192720.git-series.andrew.donnellan@au1.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-02-21_02:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1802210074 Subject: [Skiboot] [PATCH 06/10] npu2-hw-procedures: Add support for OpenCAPI PHY link training X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alistair@popple.id.au MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Unlike NVLink, which uses the pci-virt framework to fake a PCI configuration space for NVLink devices, the OpenCAPI device model presents us with a real configuration space handled by the device over the OpenCAPI link. As a result, we have to train the OpenCAPI link in skiboot before we do PCI probing, so that config space can be accessed, rather than having link training being triggered by the Linux driver. Add some helper functions to wrap the existing NVLink PHY training sequence so we can easily run it within skiboot. Additionally, we add OpenCAPI-specific lane settings, and a function to "bump" lanes that haven't trained properly (this process isn't documented in the workbook, but the hardware experts assure us that this improves link training reliability...) Signed-off-by: Andrew Donnellan Signed-off-by: Frederic Barrat Reviewed-by: Frederic Barrat Acked-by: Reza Arbab --- v5->v6: - Skip phy_rx_clock_sel subprocedure for OpenCAPI, as it triggers a FIR bit unnecessarily (Fred) - Fix brick<->ODL mapping on OBUS3 (Fred) --- hw/npu2-hw-procedures.c | 108 ++++++++++++++++++++++++++++++++++++----- include/npu2-regs.h | 6 ++- include/npu2.h | 3 +- 3 files changed, 105 insertions(+), 12 deletions(-) diff --git a/hw/npu2-hw-procedures.c b/hw/npu2-hw-procedures.c index eb087bc17f3a..cd64b9ee2634 100644 --- a/hw/npu2-hw-procedures.c +++ b/hw/npu2-hw-procedures.c @@ -60,6 +60,7 @@ struct npu2_phy_reg NPU2_PHY_TX_FIFO_INIT = {0x105, 53, 1}; struct npu2_phy_reg NPU2_PHY_TX_RXCAL = {0x103, 57, 1}; struct npu2_phy_reg NPU2_PHY_RX_INIT_DONE = {0x0ca, 48, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_EDGE_TRACK_CNTL = {0x092, 48, 2}; +struct npu2_phy_reg NPU2_PHY_RX_PR_BUMP_SL_1UI = {0x092, 57, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_FW_OFF = {0x08a, 56, 1}; struct npu2_phy_reg NPU2_PHY_RX_PR_FW_INERTIA_AMT = {0x08a, 57, 3}; struct npu2_phy_reg NPU2_PHY_RX_CFG_LTE_MC = {0x000, 60, 4}; @@ -68,6 +69,8 @@ struct npu2_phy_reg NPU2_PHY_RX_B_INTEG_COARSE_GAIN = {0x026, 48, 4}; struct npu2_phy_reg NPU2_PHY_RX_E_INTEG_COARSE_GAIN = {0x030, 48, 4}; /* These registers are per-PHY, not per lane */ +struct npu2_phy_reg NPU2_PHY_RX_SPEED_SELECT = {0x262, 51, 2}; +struct npu2_phy_reg NPU2_PHY_RX_AC_COUPLED = {0x262, 53, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_SWO_EN = {0x3c9, 48, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_REQ = {0x3c1, 49, 1}; struct npu2_phy_reg NPU2_PHY_TX_ZCAL_DONE = {0x3c1, 50, 1}; @@ -378,6 +381,11 @@ static uint32_t phy_reset_complete(struct npu2_dev *ndev) { int lane; + if (ndev->type == NPU2_DEV_TYPE_OPENCAPI) { + phy_write(ndev, &NPU2_PHY_RX_AC_COUPLED, 1); + phy_write(ndev, &NPU2_PHY_RX_SPEED_SELECT, 1); + } + FOR_EACH_LANE(ndev, lane) { phy_write_lane(ndev, &NPU2_PHY_RX_LANE_ANA_PDWN, lane, 0); phy_write_lane(ndev, &NPU2_PHY_RX_LANE_DIG_PDWN, lane, 0); @@ -640,18 +648,25 @@ static uint32_t phy_rx_dccal_complete(struct npu2_dev *ndev) static uint32_t phy_rx_clock_sel(struct npu2_dev *ndev) { - /* - * Change the RX clk mux control to be done by software instead of HW. This - * avoids glitches caused by changing the mux setting. - * - * Work around a known DL bug by doing these writes twice. - */ - npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), 0x80000002, 0x80000003); - npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), 0x80000002, 0x80000003); - - npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), 0x80000000, 0x80000003); - npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), 0x80000000, 0x80000003); - + if (ndev->type != NPU2_DEV_TYPE_OPENCAPI) { + /* + * Change the RX clk mux control to be done by + * software instead of HW. This avoids glitches caused + * by changing the mux setting. + * + * Work around a known DL bug by doing these writes + * twice. + */ + npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), + 0x80000002, 0x80000003); + npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), + 0x80000002, 0x80000003); + + npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), + 0x80000000, 0x80000003); + npu2_write_mask_4b(ndev->npu, NPU2_NTL_DL_CLK_CTRL(ndev), + 0x80000000, 0x80000003); + } return PROCEDURE_NEXT; } @@ -887,3 +902,72 @@ void npu2_dev_procedure_reset(struct npu2_dev *dev) { npu2_clear_link_flag(dev, NPU2_DEV_DL_RESET); } + +static uint32_t run_procedure(struct npu2_dev *dev, uint16_t procedure_number) +{ + struct procedure *proc; + const char *name; + uint32_t result; + + assert(procedure_number <= ARRAY_SIZE(npu_procedures)); + proc = npu_procedures[procedure_number]; + assert(proc); + + name = proc->name; + NPU2DEVINF(dev, "Running procedure %s\n", name); + dev->procedure_status = PROCEDURE_INPROGRESS; + dev->procedure_number = procedure_number; + dev->procedure_step = 0; + dev->procedure_data = 0; + dev->procedure_tb = mftb(); + + result = get_procedure_status(dev); + while (!(result & PROCEDURE_COMPLETE)) { + time_wait_ms(1); + result = get_procedure_status(dev); + } + return result; +} + +void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev) +{ + uint64_t reg; + uint64_t status_xscom; + int lane, bit = 7; + + switch (dev->index) { + case 2: + status_xscom = OB0_ODL0_TRAINING_STATUS; + break; + case 3: + status_xscom = OB0_ODL1_TRAINING_STATUS; + break; + case 4: + status_xscom = OB3_ODL1_TRAINING_STATUS; + break; + case 5: + status_xscom = OB3_ODL0_TRAINING_STATUS; + break; + default: + assert(false); + } + xscom_read(dev->npu->chip_id, status_xscom, ®); + reg = GETFIELD(OB_ODL_TRAINING_STATUS_STS_RX_PATTERN_B, reg); + + FOR_EACH_LANE(dev, lane) { + if (reg & (1 << bit--)) + continue; + prlog(PR_TRACE, "OCAPI: bumpui bumping lane %d\n", lane); + for (int i = 0; i < 4; i++) { + phy_write_lane(dev, &NPU2_PHY_RX_PR_BUMP_SL_1UI, lane, 1); + phy_write_lane(dev, &NPU2_PHY_RX_PR_BUMP_SL_1UI, lane, 0); + } + } +} + +void npu2_opencapi_phy_setup(struct npu2_dev *dev) +{ + run_procedure(dev, 4); /* procedure_phy_reset */ + run_procedure(dev, 5); /* procedure_phy_tx_zcal */ + run_procedure(dev, 6); /* procedure_phy_rx_dccal */ +} diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 86d5f7c2bb11..62d405ad5c2d 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -661,4 +661,10 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define PU_IOE_PB_FP_CFG_FP1_FMR_DISABLE PPC_BIT(52) #define PU_IOE_PB_FP_CFG_FP1_PRS_DISABLE PPC_BIT(57) +#define OB0_ODL0_TRAINING_STATUS 0x901082E +#define OB0_ODL1_TRAINING_STATUS 0x901082F +#define OB3_ODL0_TRAINING_STATUS 0xC01082E +#define OB3_ODL1_TRAINING_STATUS 0xC01082F +#define OB_ODL_TRAINING_STATUS_STS_RX_PATTERN_B PPC_BITMASK(8, 15) + #endif /* __NPU2_REGS_H */ diff --git a/include/npu2.h b/include/npu2.h index 155b8fbe48e2..969057700b40 100644 --- a/include/npu2.h +++ b/include/npu2.h @@ -200,10 +200,13 @@ int64_t npu2_dev_procedure(void *dev, struct pci_cfg_reg_filter *pcrf, uint32_t offset, uint32_t len, uint32_t *data, bool write); void npu2_dev_procedure_reset(struct npu2_dev *dev); + void npu2_set_link_flag(struct npu2_dev *ndev, uint8_t flag); void npu2_clear_link_flag(struct npu2_dev *ndev, uint8_t flag); uint32_t reset_ntl(struct npu2_dev *ndev); extern int nv_zcal_nominal; bool is_p9dd1(void); +void npu2_opencapi_phy_setup(struct npu2_dev *dev); +void npu2_opencapi_bump_ui_lane(struct npu2_dev *dev); #endif /* __NPU2_H */