From patchwork Mon Jul 19 13:20:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vasant Hegde X-Patchwork-Id: 1507070 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=gWKOGZHr; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GT2dm4LqXz9sXJ for ; Mon, 19 Jul 2021 23:25:16 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4GT2dm3NXNz3cgW for ; Mon, 19 Jul 2021 23:25:16 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=gWKOGZHr; dkim-atps=neutral X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=hegdevasant@linux.vnet.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=gWKOGZHr; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4GT2ZT6Z7dz3bjl for ; Mon, 19 Jul 2021 23:22:25 +1000 (AEST) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16JD6J26073851 for ; Mon, 19 Jul 2021 09:22:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=UFYwIYtnoiVFZWq5M80u/Eyza83886Bor9XcfAJqbQI=; b=gWKOGZHrf9Gz+TZcTAHsRC8erct+G18mwyiA8c4BUkpjXUpAn9bkVupZPXENH00BTJ9j mSpEQNQJ/pKmm9jQZ12ctEk/QlZLPgX9tLpOfxqCSVVsrm+2cWuEs9OhkhNNPfJ7Aj+u 1gge101oco6mdCVkpmL9Al3hk7Hh4jb4e18kiykaoEfEDDD7uQEe9dJ3oYSa1Jps4d7J RvEbLLFeH8BrO71k1PY7o7uJ1Q5nYy3CrzvbTLmVT2FKDGbnBOA9Y+xmP6HkvOWgrShC 95gtVjklLAuJMeTIEt474okZvVInNssZ3iTCjkmfPbHMGO2DrLlonN2jhq+yYt7s1T+w aQ== Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 39w9h19huy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 19 Jul 2021 09:22:23 -0400 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 16JDEEve019358 for ; Mon, 19 Jul 2021 13:22:21 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma03ams.nl.ibm.com with ESMTP id 39upu88qkr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 19 Jul 2021 13:22:21 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 16JDJv5x19857672 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 13:19:57 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A8BEFA4051; Mon, 19 Jul 2021 13:22:18 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 49DCEA4053; Mon, 19 Jul 2021 13:22:17 +0000 (GMT) Received: from hegdevasant.in.ibm.com (unknown [9.85.86.230]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 19 Jul 2021 13:22:17 +0000 (GMT) From: Vasant Hegde To: skiboot@lists.ozlabs.org Date: Mon, 19 Jul 2021 18:50:06 +0530 Message-Id: <20210719132012.150948-56-hegdevasant@linux.vnet.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210719132012.150948-1-hegdevasant@linux.vnet.ibm.com> References: <20210719132012.150948-1-hegdevasant@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: UWT1OZqdQ0E1mciexorBtN29VUZc8WWE X-Proofpoint-GUID: UWT1OZqdQ0E1mciexorBtN29VUZc8WWE X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-19_05:2021-07-19, 2021-07-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 mlxscore=0 phishscore=0 malwarescore=0 lowpriorityscore=0 spamscore=0 suspectscore=0 mlxlogscore=999 clxscore=1015 adultscore=0 priorityscore=1501 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190076 Subject: [Skiboot] [PATCH 55/61] phb4/5: Fix PHB link width detection to avoid useless retrainings X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Frederic Barrat On P9 and P10, the PCI express controller (PEC) controls a set of 16 lanes, which can be grouped to form link(s) of various width (4, 8 or 16 lanes). A PCI host bridge (PHB) is handling each link. How many PHBs are active in each PEC is configurable per chip and vary between 2 chips in a system. Therefore PHBs have different link width. The link width of the PHB is used to check if the link is trained optimally and can cause link training retries if that's not the case. We were reading the max link width of a PHB from the link capability register of the PCI express capability of the root bridge. But that value is always an overshoot as it needs to accommodate any PEC configuration. It was hard to fault on P9, as a PEC needs to be trifurcated to start noticing a difference and the device-supported width can also mask it. But on P10, it's also noticeable on bifurcated configuration so it's a bit easier to spot. For example, on P10, PHB0 reports a supported width of 16 in its link capability register because that's what is needed in case of no furcation, but if the PEC is bifurcated or trifurcated, only 8 lanes are wired. So we won't be able to train at more than x8. If we believe the PHB is x16-capable, then we'll retrain the link, potentially several times, thinking it's not optimal, which is a waste of time. This patch finds out the real maximum link width of each PHB, which may require to go check the PEC configuration. The logic is the same on P9 and P10 though the hardware implementations differ slightly. Signed-off-by: Frederic Barrat Signed-off-by: Vasant Hegde --- hw/phb4.c | 91 ++++++++++++++++++++++++++++++++++++++++++++------ include/phb4.h | 1 + 2 files changed, 81 insertions(+), 11 deletions(-) diff --git a/hw/phb4.c b/hw/phb4.c index 274fa356a..3c316f41e 100644 --- a/hw/phb4.c +++ b/hw/phb4.c @@ -2726,18 +2726,14 @@ static bool phb4_link_optimal(struct pci_slot *slot, uint32_t *vdid) uint64_t reg; uint32_t id; uint16_t bdfn, lane_errs; - uint8_t trained_speed, phb_speed, dev_speed, target_speed, rx_errs; - uint8_t trained_width, phb_width, dev_width, target_width; + uint8_t trained_speed, dev_speed, target_speed, rx_errs; + uint8_t trained_width, dev_width, target_width; bool optimal_speed, optimal_width, optimal, retry_enabled, rx_err_ok; /* Current trained state */ phb4_get_link_info(slot, &trained_speed, &trained_width); - /* Get PHB capability */ - /* NOTE: phb_speed will account for the software speed limit */ - phb4_get_info(slot->phb, 0, &phb_speed, &phb_width); - /* Get device capability */ bdfn = 0x0100; /* bus=1 dev=0 device=0 */ /* Since this is the first access, we need to wait for CRS */ @@ -2746,9 +2742,9 @@ static bool phb4_link_optimal(struct pci_slot *slot, uint32_t *vdid) phb4_get_info(slot->phb, bdfn, &dev_speed, &dev_width); /* Work out if we are optimally trained */ - target_speed = MIN(phb_speed, dev_speed); + target_speed = MIN(p->max_link_speed, dev_speed); optimal_speed = (trained_speed >= target_speed); - target_width = MIN(phb_width, dev_width); + target_width = MIN(p->max_link_width, dev_width); optimal_width = (trained_width >= target_width); optimal = optimal_width && optimal_speed; retry_enabled = (phb4_chip_retry_workaround() && @@ -2764,9 +2760,11 @@ static bool phb4_link_optimal(struct pci_slot *slot, uint32_t *vdid) DEVICE(id), optimal ? "Optimal" : "Degraded", retry_enabled ? "enabled" : "disabled"); PHBDBG(p, "LINK: Speed Train:GEN%i PHB:GEN%i DEV:GEN%i%s\n", - trained_speed, phb_speed, dev_speed, optimal_speed ? "" : " *"); + trained_speed, p->max_link_speed, dev_speed, + optimal_speed ? "" : " *"); PHBDBG(p, "LINK: Width Train:x%02i PHB:x%02i DEV:x%02i%s\n", - trained_width, phb_width, dev_width, optimal_width ? "" : " *"); + trained_width, p->max_link_width, dev_width, + optimal_width ? "" : " *"); PHBDBG(p, "LINK: RX Errors Now:%i Max:%i Lane:0x%04x%s\n", rx_errs, rx_err_max, lane_errs, rx_err_ok ? "" : " *"); @@ -3043,6 +3041,75 @@ static unsigned int phb4_get_max_link_speed(struct phb4 *p, struct dt_node *np) return max_link_speed; } +static unsigned int __phb4_get_max_link_width(struct phb4 *p) +{ + uint64_t addr, reg; + unsigned int lane_config, width = 16; + + /* + * On P9, only PEC2 is configurable (no-/bi-/tri-furcation) + */ + switch (p->pec) { + case 0: + width = 16; + break; + case 1: + width = 8; + break; + case 2: + addr = XPEC_P9_PCI_CPLT_CONF1 + 2 * XPEC_PCI_CPLT_OFFSET; + xscom_read(p->chip_id, addr, ®); + lane_config = GETFIELD(XPEC_P9_PCI_LANE_CFG, reg); + + if (lane_config == 0b10 && p->index >= 4) + width = 4; + else + width = 8; + } + return width; +} + +static unsigned int __phb5_get_max_link_width(struct phb4 *p) +{ + uint64_t addr, reg; + unsigned int lane_config, width = 16; + + /* + * On P10, the 2 PECs are identical and each can have a + * different furcation, so we always need to check the PEC + * config + */ + addr = XPEC_P10_PCI_CPLT_CONF1 + p->pec * XPEC_PCI_CPLT_OFFSET; + xscom_read(p->chip_id, addr, ®); + lane_config = GETFIELD(XPEC_P10_PCI_LANE_CFG, reg); + + switch (lane_config) { + case 0b00: + width = 16; + break; + case 0b01: + width = 8; + break; + case 0b10: + if (p->index == 0 || p->index == 3) + width = 8; + else + width = 4; + break; + default: + PHBERR(p, "Unexpected PEC lane config value %#x\n", + lane_config); + } + return width; +} + +static unsigned int phb4_get_max_link_width(struct phb4 *p) +{ + if (is_phb5()) + return __phb5_get_max_link_width(p); + else + return __phb4_get_max_link_width(p); +} static void phb4_assert_perst(struct pci_slot *slot, bool assert) { @@ -5982,7 +6049,9 @@ static void phb4_create(struct dt_node *np) goto failed; p->max_link_speed = phb4_get_max_link_speed(p, np); - PHBINF(p, "Max link speed: GEN%i\n", p->max_link_speed); + p->max_link_width = phb4_get_max_link_width(p); + PHBINF(p, "Max link speed: GEN%i, max link width %i\n", + p->max_link_speed, p->max_link_width); /* Check for lane equalization values from HB or HDAT */ p->lane_eq_en = true; diff --git a/include/phb4.h b/include/phb4.h index 0bbfc926c..4f1fb31c5 100644 --- a/include/phb4.h +++ b/include/phb4.h @@ -197,6 +197,7 @@ struct phb4 { bool lane_eq_en; unsigned int max_link_speed; unsigned int dt_max_link_speed; + unsigned int max_link_width; uint64_t mrt_size; uint64_t mbt_size;