From patchwork Mon Jul 19 13:20:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vasant Hegde X-Patchwork-Id: 1507075 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=luqbr+x8; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4GT2fF45mbz9s5R for ; Mon, 19 Jul 2021 23:25:41 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4GT2fF3P3Mz3bwr for ; Mon, 19 Jul 2021 23:25:41 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=luqbr+x8; dkim-atps=neutral X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=hegdevasant@linux.vnet.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=luqbr+x8; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4GT2Zh267Bz3blk for ; Mon, 19 Jul 2021 23:22:36 +1000 (AEST) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16JDBfMu053487 for ; Mon, 19 Jul 2021 09:22:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=YNhDsdUXOZ/Y+MIZv/D3hyiEtg/d6FqEt9b7oZ4NPCg=; b=luqbr+x8emWve8QDrQQ25hPbgHSxNqHvjZ/I2zK+4SFc9RLxdKRCErEpSesyrfPmfngs 7JyUxH4RpgNvjKnAeQP7b/DTNRDrBWoNBFyUat05B+yJ7qnmZLoIF8DCFErEz44UtkwW VlEBypwtmw1WBcThqmw1YEKsVRjCpWC4HnHlY21Tr/wXEJ09OiH6r77oN6NHp5/Idhtx RLEnFfksEAJCnGCIYgzlrQ40GkvG3G8tJLBZF1wSFBaJ7CiCVW6e3OngplE/qmNURcCM a9la1Fv297i5zHXeQpaDGJYqX4Rho24T2pEZ6YOqWFa7CfviXOqJVYSLv+L+jO+I47qs jQ== Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 39w8w8jv4b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 19 Jul 2021 09:22:33 -0400 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 16JDEQc9019388 for ; Mon, 19 Jul 2021 13:22:32 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma03ams.nl.ibm.com with ESMTP id 39upu88qks-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 19 Jul 2021 13:22:31 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 16JDMTTo25231618 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 13:22:29 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0126AA4057; Mon, 19 Jul 2021 13:22:29 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D87A7A404D; Mon, 19 Jul 2021 13:22:27 +0000 (GMT) Received: from hegdevasant.in.ibm.com (unknown [9.85.86.230]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 19 Jul 2021 13:22:27 +0000 (GMT) From: Vasant Hegde To: skiboot@lists.ozlabs.org Date: Mon, 19 Jul 2021 18:50:11 +0530 Message-Id: <20210719132012.150948-61-hegdevasant@linux.vnet.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210719132012.150948-1-hegdevasant@linux.vnet.ibm.com> References: <20210719132012.150948-1-hegdevasant@linux.vnet.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: LukJpY9QFXofOvIfwC0Hx6IWueJbTOYj X-Proofpoint-GUID: LukJpY9QFXofOvIfwC0Hx6IWueJbTOYj X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-19_05:2021-07-19, 2021-07-19 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 mlxscore=0 priorityscore=1501 lowpriorityscore=0 phishscore=0 adultscore=0 suspectscore=0 clxscore=1015 impostorscore=0 spamscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190076 Subject: [Skiboot] [PATCH 60/61] hw/chiptod: Retry the sync procedure on failure X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ryan Grimm Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" From: Ryan Grimm The chiptod sync will sometimes fail and then sync successfully after a retry. So, try an arbitrary 10 numbers of times before we either abort() on main procedure fail or disable threads on secondary procedure fail. Also, put a message on the log if secondaries fail so we have evidence in the log when they aren't enabled. Signed-off-by: Ryan Grimm Signed-off-by: Vasant Hegde --- hw/chiptod.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/hw/chiptod.c b/hw/chiptod.c index 3b57f5f16..fd9414990 100644 --- a/hw/chiptod.c +++ b/hw/chiptod.c @@ -221,6 +221,8 @@ static uint64_t base_tfmr; static struct lock chiptod_lock = LOCK_UNLOCKED; static bool chiptod_unrecoverable; +#define NUM_SYNC_RETRIES 10 + static void _chiptod_cache_tod_regs(int32_t chip_id) { int i; @@ -892,7 +894,7 @@ static void chiptod_sync_master(void *data) *result = true; return; error: - prerror("Master sync failed! TFMR=0x%016lx\n", mfspr(SPR_TFMR)); + prerror("Master sync failed! TFMR=0x%016lx, retrying...\n", mfspr(SPR_TFMR)); *result = false; } @@ -962,7 +964,7 @@ static void chiptod_sync_slave(void *data) *result = true; return; error: - prerror("Slave sync failed ! TFMR=0x%016lx\n", mfspr(SPR_TFMR)); + prerror("Slave sync failed ! TFMR=0x%016lx, retrying...\n", mfspr(SPR_TFMR)); *result = false; } @@ -1818,6 +1820,7 @@ void chiptod_init(void) { struct cpu_thread *cpu0, *cpu; bool sres; + int i; /* Mambo and qemu doesn't simulate the chiptod */ if (chip_quirk(QUIRK_NO_CHIPTOD)) @@ -1841,10 +1844,14 @@ void chiptod_init(void) prlog(PR_DEBUG, "Base TFMR=0x%016llx\n", base_tfmr); - /* Schedule master sync */ - sres = false; - cpu_wait_job(cpu_queue_job(cpu0, "chiptod_sync_master", + i = NUM_SYNC_RETRIES; + do { + /* Schedule master sync */ + sres = false; + cpu_wait_job(cpu_queue_job(cpu0, "chiptod_sync_master", chiptod_sync_master, &sres), true); + } while (!sres && i--); + if (!sres) { op_display(OP_FATAL, OP_MOD_CHIPTOD, 2); abort(); @@ -1858,13 +1865,19 @@ void chiptod_init(void) if (cpu == cpu0) continue; - /* Queue job */ - sres = false; - cpu_wait_job(cpu_queue_job(cpu, "chiptod_sync_slave", - chiptod_sync_slave, &sres), - true); + i = NUM_SYNC_RETRIES; + do { + /* Queue job */ + sres = false; + cpu_wait_job(cpu_queue_job(cpu, "chiptod_sync_slave", + chiptod_sync_slave, &sres), + true); + } while (!sres && i--); + if (!sres) { op_display(OP_WARN, OP_MOD_CHIPTOD, 3|(cpu->pir << 8)); + prerror("CHIPTOD: Failed to sync PIR 0x%04x\n", + this_cpu()->pir); /* Disable threads */ cpu_disable_all_threads(cpu);