From patchwork Wed May 25 07:40:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Roese X-Patchwork-Id: 1635303 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=denx.de header.i=@denx.de header.a=rsa-sha256 header.s=phobos-20191101 header.b=MXI1qQId; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4L7NLg0D3rz9sBB for ; Wed, 25 May 2022 17:42:02 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 0B4BF842C4; Wed, 25 May 2022 09:41:31 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=denx.de Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=denx.de; s=phobos-20191101; t=1653464491; bh=dtpBOJnKziS7+jCM7RBkkn+Z5TAlJKCpuunnLZXCmdQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From; b=MXI1qQIdOJAhfnWEs6oYi0INftgV86Qnt+ywnAbHh4XGr8u/yar+j/dK9tkXKmdiC mYjwTIFlrqrjTGXcxxukP1ixokvIQL8MDl3vGIQNAZOjPEIAlhZMkyQOTe+/hcRHzV RRDu+ckcG1wMkxCEOLaD4bhGaAw/R4T94Ty8HjyF/pPGwwq2dbDXaVbZnpLLTBsywb 31b/Tvbf8KvtEiWe1QYxad1UG2M3LXonChSsHnHJugIEJVVFyr6B0Sor6QEZ7ZQYua xVy4yXwcbGpWdsItnK/TqG4lIeFaU+rOeW5yErm4T/f22c9+ZLT3ZigCjVuL9DEquh sCcDCpcRCX01w== Received: by phobos.denx.de (Postfix, from userid 109) id C6898842C2; Wed, 25 May 2022 09:40:33 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_NEUTRAL,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.2 Received: from mout-u-107.mailbox.org (mout-u-107.mailbox.org [91.198.250.252]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 61C0C842D2 for ; Wed, 25 May 2022 09:40:25 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=none (p=none dis=none) header.from=denx.de Authentication-Results: phobos.denx.de; spf=fail smtp.mailfrom=sr@denx.de Received: from smtp1.mailbox.org (smtp1.mailbox.org [10.196.197.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-u-107.mailbox.org (Postfix) with ESMTPS id 4L7NJl2QXVz9sTm; Wed, 25 May 2022 09:40:23 +0200 (CEST) From: Stefan Roese To: u-boot@lists.denx.de Cc: trini@konsulko.com, sjg@chromium.org, cchavva@marvell.com, awilliams@marvell.com Subject: [PATCH 7/7] mips: octeon_nic23: Add PCIe FLR fixup via cyclic infrastructure Date: Wed, 25 May 2022 09:40:19 +0200 Message-Id: <20220525074019.946607-8-sr@denx.de> In-Reply-To: <20220525074019.946607-1-sr@denx.de> References: <20220525074019.946607-1-sr@denx.de> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean From: Aaron Williams This patch adds a fixup function related to a PCIe FLR (Function Level Reset) problem on the NIC23 PCIe board. This function is imported from the Marvell Octeon 2013 U-Boot version as a (nearly) verbatim copy. It uses the newly introduced cyclic infrastructure, so that this function gets called every 100us, which is needed to detect this FLR issue. Signed-off-by: Aaron Williams Signed-off-by: Stefan Roese --- board/Marvell/octeon_nic23/board.c | 197 +++++++++++++++++++++++++++++ configs/octeon_nic23_defconfig | 3 + 2 files changed, 200 insertions(+) diff --git a/board/Marvell/octeon_nic23/board.c b/board/Marvell/octeon_nic23/board.c index 3e2c54444397..446fa5a659a8 100644 --- a/board/Marvell/octeon_nic23/board.c +++ b/board/Marvell/octeon_nic23/board.c @@ -3,8 +3,10 @@ * Copyright (C) 2021-2022 Stefan Roese */ +#include #include #include +#include #include #include @@ -15,11 +17,90 @@ #include #include #include +#include #include "board_ddr.h" +/** + * cvmx_spem#_cfg_rd + * + * This register allows read access to the configuration in the PCIe core. + * + */ +union cvmx_spemx_cfg_rd { + u64 u64; + struct cvmx_spemx_cfg_rd_s { + u64 data : 32; + u64 addr : 32; + } s; + struct cvmx_spemx_cfg_rd_s cn73xx; +}; + +/** + * cvmx_spem#_cfg_wr + * + * This register allows write access to the configuration in the PCIe core. + * + */ +union cvmx_spemx_cfg_wr { + u64 u64; + struct cvmx_spemx_cfg_wr_s { + u64 data : 32; + u64 addr : 32; + } s; + struct cvmx_spemx_cfg_wr_s cn73xx; +}; + +/** + * cvmx_spem#_flr_pf_stopreq + * + * PF function level reset stop outbound requests register. + * Hardware automatically sets the STOPREQ bit for the PF when it enters a + * function level reset (FLR). Software is responsible for clearing the STOPREQ + * bit but must not do so prior to hardware taking down the FLR, which could be + * as long as 100ms. It may be appropriate for software to wait longer before clearing + * STOPREQ, software may need to drain deep DPI queues for example. + * Whenever SPEM receives a PF or child VF request mastered by CNXXXX over S2M (i.e. P or NP), + * when STOPREQ is set for the function, SPEM will discard the outgoing request + * before sending it to the PCIe core. If a NP, SPEM will schedule an immediate + * SWI_RSP_ERROR completion for the request - no timeout is required. + * In both cases, SPEM()_DBG_PF()_INFO[P()_BMD_E] will be set and a error + * interrupt is generated. + * + * STOPREQ mimics the behavior of PCIEEP()_CFG001[ME] for outbound requests that will + * master the PCIe bus (P and NP). + * + * STOPREQ will have no effect on completions returned by CNXXXX over the S2M, + * nor on M2S traffic. + * + * When a PF()_STOPREQ is set, none of the associated + * PEM()_FLR_PF()_VF_STOPREQ[VF_STOPREQ] will be set. + * + * STOPREQ is reset when the MAC is reset, and is not reset after a chip soft reset. + */ +union cvmx_spemx_flr_pf_stopreq { + u64 u64; + struct cvmx_spemx_flr_pf_stopreq_s { + u64 reserved_3_63 : 61; + u64 pf2_stopreq : 1; + u64 pf1_stopreq : 1; + u64 pf0_stopreq : 1; + } s; + struct cvmx_spemx_flr_pf_stopreq_s cn73xx; +}; + +#define CVMX_SPEMX_CFG_WR(offset) 0x00011800C0000028ull +#define CVMX_SPEMX_CFG_RD(offset) 0x00011800C0000030ull +#define CVMX_SPEMX_FLR_PF_STOPREQ(offset) 0x00011800C0000218ull + +#define DTX_SELECT_LTSSM 0x0 +#define DTX_SELECT_LTSSM_ENA 0x3ff +#define LTSSM_L0 0x11 + #define NIC23_DEF_DRAM_FREQ 800 +static u32 pci_cfgspace_reg0[2] = { 0, 0 }; + static u8 octeon_nic23_cfg0_spd_values[512] = { OCTEON_NIC23_CFG0_SPD_VALUES }; @@ -145,8 +226,118 @@ void board_configure_qlms(void) cvmx_qlm_measure_clock(4), cvmx_qlm_measure_clock(5)); } +/** + * If there is a PF FLR then the PCI EEPROM is not re-read. In this case + * we need to re-program the vendor and device ID immediately after hardware + * completes FLR. + * + * PCI spec requires FLR to be completed within 100ms. The user who triggered + * FLR expects hardware to finish FLR within 100ms, otherwise the user will + * end up reading DEVICE_ID incorrectly from the reset value. + * CN23XX exits FLR at any point between 66 and 99ms, so U-Boot has to wait + * 99ms to let hardware finish its part, then finish reprogramming the + * correct device ID before the end of 100ms. + * + * Note: this solution only works properly when there is no other activity + * within U-Boot for 100ms from the time FLR is triggered. + * + * This function gets called every 100usec. If FLR happens during any + * other activity like bootloader/image update then it is possible that + * this function does not get called for more than the FLR period which will + * cause the PF device ID restore to happen after whoever initiated the FLR to + * read the incorrect device ID 0x9700 (reset value) instead of 0x9702 + * (restored value). + */ +static void octeon_board_restore_pf(void *ctx) +{ + union cvmx_spemx_flr_pf_stopreq stopreq; + static bool start_initialized[2] = {false, false}; + bool pf0_flag, pf1_flag; + u64 ltssm_bits; + const u64 pf_flr_wait_usecs = 99700; + u64 elapsed_usecs; + union cvmx_spemx_cfg_wr cfg_wr; + union cvmx_spemx_cfg_rd cfg_rd; + static u64 start_us[2]; + int pf_num; + + csr_wr(CVMX_DTX_SPEM_SELX(0), DTX_SELECT_LTSSM); + csr_rd(CVMX_DTX_SPEM_SELX(0)); + csr_wr(CVMX_DTX_SPEM_ENAX(0), DTX_SELECT_LTSSM_ENA); + csr_rd(CVMX_DTX_SPEM_ENAX(0)); + ltssm_bits = csr_rd(CVMX_DTX_SPEM_DATX(0)); + if (((ltssm_bits >> 3) & 0x3f) != LTSSM_L0) + return; + + stopreq.u64 = csr_rd(CVMX_SPEMX_FLR_PF_STOPREQ(0)); + pf0_flag = stopreq.s.pf0_stopreq; + pf1_flag = stopreq.s.pf1_stopreq; + /* See if PF interrupt happened */ + if (!(pf0_flag || pf1_flag)) + return; + + if (pf0_flag && !start_initialized[0]) { + start_initialized[0] = true; + start_us[0] = get_timer_us(0); + } + + /* Store programmed PCIe DevID SPEM0 PF0 */ + if (pf0_flag && !pci_cfgspace_reg0[0]) { + cfg_rd.s.addr = (0 << 24) | 0x0; + csr_wr(CVMX_SPEMX_CFG_RD(0), cfg_rd.u64); + cfg_rd.u64 = csr_rd(CVMX_SPEMX_CFG_RD(0)); + pci_cfgspace_reg0[0] = cfg_rd.s.data; + } + + if (pf1_flag && !start_initialized[1]) { + start_initialized[1] = true; + start_us[1] = get_timer_us(0); + } + + /* Store programmed PCIe DevID SPEM0 PF1 */ + if (pf1_flag && !pci_cfgspace_reg0[1]) { + cfg_rd.s.addr = (1 << 24) | 0x0; + csr_wr(CVMX_SPEMX_CFG_RD(0), cfg_rd.u64); + cfg_rd.u64 = csr_rd(CVMX_SPEMX_CFG_RD(0)); + pci_cfgspace_reg0[1] = cfg_rd.s.data; + } + + /* For PF, rewrite pci config space reg 0 */ + for (pf_num = 0; pf_num < 2; pf_num++) { + if (!start_initialized[pf_num]) + continue; + + elapsed_usecs = get_timer_us(0) - start_us[pf_num]; + + if (elapsed_usecs > pf_flr_wait_usecs) { + /* Here, our measured FLR duration has passed; + * check if device ID has been reset, + * which indicates FLR completion (per MA team). + */ + cfg_rd.s.addr = (pf_num << 24) | 0x0; + csr_wr(CVMX_SPEMX_CFG_RD(0), cfg_rd.u64); + cfg_rd.u64 = csr_rd(CVMX_SPEMX_CFG_RD(0)); + /* if DevID has NOT been reset, FLR is not yet + * complete + */ + if (cfg_rd.s.data != pci_cfgspace_reg0[pf_num]) { + stopreq.s.pf0_stopreq = (pf_num == 0) ? 1 : 0; + stopreq.s.pf1_stopreq = (pf_num == 1) ? 1 : 0; + csr_wr(CVMX_SPEMX_FLR_PF_STOPREQ(0), stopreq.u64); + + cfg_wr.u64 = 0; + cfg_wr.s.addr = (pf_num << 24) | 0; + cfg_wr.s.data = pci_cfgspace_reg0[pf_num]; + csr_wr(CVMX_SPEMX_CFG_WR(0), cfg_wr.u64); + start_initialized[pf_num] = false; + } + } + } +} + int board_late_init(void) { + struct cyclic_struct *cyclic; struct gpio_desc gpio = {}; ofnode node; @@ -164,6 +355,12 @@ int board_late_init(void) board_configure_qlms(); + /* Register cyclic function for PCIe FLR fixup */ + cyclic = cyclic_register(octeon_board_restore_pf, 100, + "pcie_flr_fix", NULL); + if (!cyclic) + printf("Registering of cyclic function failed\n"); + return 0; } diff --git a/configs/octeon_nic23_defconfig b/configs/octeon_nic23_defconfig index 3ab5838f03e5..d9ccf079cbd9 100644 --- a/configs/octeon_nic23_defconfig +++ b/configs/octeon_nic23_defconfig @@ -20,6 +20,8 @@ CONFIG_AHCI=y CONFIG_OF_BOARD_FIXUP=y CONFIG_SYS_CONSOLE_ENV_OVERWRITE=y # CONFIG_SYS_DEVICE_NULLDEV is not set +CONFIG_CYCLIC=y +CONFIG_CYCLIC_MAX_CPU_TIME_US=5000 CONFIG_ARCH_MISC_INIT=y CONFIG_BOARD_EARLY_INIT_F=y CONFIG_BOARD_LATE_INIT=y @@ -38,6 +40,7 @@ CONFIG_CMD_TIME=y CONFIG_CMD_EXT4=y CONFIG_CMD_FAT=y CONFIG_CMD_FS_GENERIC=y +CONFIG_CMD_CYCLIC=y CONFIG_EFI_PARTITION=y CONFIG_ENV_IS_IN_SPI_FLASH=y CONFIG_TFTP_TSIZE=y