From patchwork Thu Sep 1 05:58:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1672675 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TS/ZiQZd; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MJBSK2ysdz1yhR for ; Thu, 1 Sep 2022 16:47:45 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MJBSK257Hz3dq3 for ; Thu, 1 Sep 2022 16:47:45 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TS/ZiQZd; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TS/ZiQZd; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MJ9Mm28wWz2y6K for ; Thu, 1 Sep 2022 15:58:44 +1000 (AEST) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2815mXA7003025; Thu, 1 Sep 2022 05:58:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=DG5VXmyFI+nUrD2Yxa98atltZBZjXu9BX4Io7GtKovs=; b=TS/ZiQZdfMrZmFQouNwhPCgJmQMdmi4aE9iYRdbS5r5azdICV7vPbbQQZl704BhlyH8M rEdpLjbogGKsM+sImEg55eftOw6HB9m1O77RDu3erviAwaLxBxtSNvu8fxCvftqtFwm+ SWoFAjr3zF5lswM0jQdxMOcIXahjMcPE51E0cBcZOva3iJcn8uCcIeVB4xz7E2V6KWZt IwvhlmYMlAPWXuqqawmHGRAfgmSD3O5KJnEAdwSbuxXfbWKA7ejIXxjyhUbNFC0jrVAc h56U8M30P+u8YdYRnLEvZTJ06BpzH8YZl/wI0soEVUTGxy4FpFv/RNZavihuTC/RrRqt Rw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japht8xfc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:37 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2815pfft018658; Thu, 1 Sep 2022 05:58:37 GMT Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japht8xbf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:37 +0000 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 2815o9EG018841; Thu, 1 Sep 2022 05:58:34 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma05fra.de.ibm.com with ESMTP id 3j7aw8vht9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:34 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 2815wV5G42467610 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 1 Sep 2022 05:58:32 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CDF9FA405F; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 304D4A4054; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id D083E604C3; Thu, 1 Sep 2022 15:58:28 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/4] powerpc/code-patching: add patch_memory() for writing RO text Date: Thu, 1 Sep 2022 15:58:20 +1000 Message-Id: <20220901055823.152983-2-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220901055823.152983-1-bgray@linux.ibm.com> References: <20220901055823.152983-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: fIXE6FHSbv8TMIKWMWnVhdl1upGacE9r X-Proofpoint-ORIG-GUID: oW2OmWvigu3spryu6NDO3VSclQqoxPQT X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-01_03,2022-08-31_03,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 bulkscore=0 clxscore=1015 adultscore=0 phishscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 lowpriorityscore=0 suspectscore=0 priorityscore=1501 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000 definitions=main-2209010023 X-Mailman-Approved-At: Thu, 01 Sep 2022 16:45:35 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, Benjamin Gray , ajd@linux.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" From: Russell Currey powerpc allocates a text poke area of one page that is used by patch_instruction() to modify read-only text when STRICT_KERNEL_RWX is enabled. patch_instruction() is only designed for instructions, so writing data using the text poke area can only happen 4 bytes at a time - each with a page map/unmap, pte flush and syncs. This patch introduces patch_memory(), implementing the same interface as memcpy(), similar to x86's text_poke() and s390's s390_kernel_write(). patch_memory() only needs to map the text poke area once, unless the write would cross a page boundary. Signed-off-by: Russell Currey Signed-off-by: Benjamin Gray --- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/lib/code-patching.c | 65 ++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 1c6316ec4b74..3de90748bce7 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -76,6 +76,7 @@ int create_cond_branch(ppc_inst_t *instr, const u32 *addr, int patch_branch(u32 *addr, unsigned long target, int flags); int patch_instruction(u32 *addr, ppc_inst_t instr); int raw_patch_instruction(u32 *addr, ppc_inst_t instr); +void *patch_memory(void *dest, const void *src, size_t size); static inline unsigned long patch_site_addr(s32 *site) { diff --git a/arch/powerpc/lib/code-patching.c b/arch/powerpc/lib/code-patching.c index 6edf0697a526..0cca39af44cb 100644 --- a/arch/powerpc/lib/code-patching.c +++ b/arch/powerpc/lib/code-patching.c @@ -14,6 +14,7 @@ #include #include #include +#include static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr) { @@ -183,6 +184,65 @@ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) return err; } + +static int do_patch_memory(void *dest, const void *src, size_t size) +{ + int err; + unsigned long text_poke_addr, patch_addr; + + text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr; + + err = map_patch_area(dest, text_poke_addr); + if (err) + return err; + + patch_addr = text_poke_addr + offset_in_page(dest); + copy_to_kernel_nofault((u8 *)patch_addr, src, size); + + flush_icache_range(patch_addr, size); + unmap_patch_area(text_poke_addr); + + return 0; +} + +/** + * patch_memory - write data using the text poke area + * + * @dest: destination address + * @src: source address + * @size: size in bytes + * + * like memcpy(), but using the text poke area. No atomicity guarantees. + * Do not use for instructions, use patch_instruction() instead. + * Handles crossing page boundaries, though you shouldn't need to. + * + * Return value: + * @dest + **/ +void *patch_memory(void *dest, const void *src, size_t size) +{ + int err; + unsigned long flags; + size_t written, write_size; + + // If the poke area isn't set up, it's early boot and we can just memcpy. + if (!this_cpu_read(text_poke_area)) + return memcpy(dest, src, size); + + for (written = 0; written < size; written += write_size) { + // Write as much as possible without crossing a page boundary. + write_size = min_t(size_t, size - written, + PAGE_SIZE - offset_in_page(dest + written)); + + local_irq_save(flags); + err = do_patch_memory(dest + written, src + written, write_size); + local_irq_restore(flags); + if (err) + return ERR_PTR(err); + } + + return dest; +} #else /* !CONFIG_STRICT_KERNEL_RWX */ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) @@ -190,6 +250,11 @@ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) return raw_patch_instruction(addr, instr); } +void *patch_memory(void *dest, const void *src, size_t size) +{ + return memcpy(dest, src, size); +} + #endif /* CONFIG_STRICT_KERNEL_RWX */ __ro_after_init DEFINE_STATIC_KEY_FALSE(init_mem_is_free); From patchwork Thu Sep 1 05:58:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1672674 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=i48Nyria; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MJBRZ2zqMz1yhR for ; Thu, 1 Sep 2022 16:47:06 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MJBRZ2RkDz3cBm for ; Thu, 1 Sep 2022 16:47:06 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=i48Nyria; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=i48Nyria; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MJ9Mk6scRz2y6K for ; Thu, 1 Sep 2022 15:58:42 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2815KkiG013254; Thu, 1 Sep 2022 05:58:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=owaUQuuOjkD4sGAwPAWhoYDbmGg7zVh4iCrv+cHZOd4=; b=i48Nyria+0lseNhk9quiKa2Tho5lbmX9xPgtzxb6ri+zPaZmp3Ym2v91RaA/jcwPy4f/ aHwcr9PCSR36B9GZeYS4iQEZ2D6ZkzsNn156P2PEY1cjDtndlYvciuRO3H+xTngzf5A0 aduIYkFLCn0aqIM/OdTs/jsbMTjyQUtoT5SzJDk3lpdt/FTmyyJay8x4oKLPRIJNxduY gTE7TgoAoaNiF6KdkLr/5pjg0PiEZT29TxH+1ytI4DPJkeLWAETLh3ZUHGqRzf3N6WDK Uv47zkbHXS3lyeZ0Zbf1BcU+Lnsd9PVLhQoJCA2Ca2HaE7U/+A38M6/ytYD/lfhDa0my Sg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japgu18bt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:37 +0000 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2815KuXl013965; Thu, 1 Sep 2022 05:58:36 GMT Received: from ppma03fra.de.ibm.com (6b.4a.5195.ip4.static.sl-reverse.com [149.81.74.107]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japgu18ab-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:36 +0000 Received: from pps.filterd (ppma03fra.de.ibm.com [127.0.0.1]) by ppma03fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 2815oq22001937; Thu, 1 Sep 2022 05:58:34 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma03fra.de.ibm.com with ESMTP id 3j7aw8vhbw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:33 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 2815wVH237617988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 1 Sep 2022 05:58:31 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7A43F52050; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 255F25204F; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id D36C460257; Thu, 1 Sep 2022 15:58:28 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/4] static_call: Move static call selftest to static_call.c Date: Thu, 1 Sep 2022 15:58:21 +1000 Message-Id: <20220901055823.152983-3-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220901055823.152983-1-bgray@linux.ibm.com> References: <20220901055823.152983-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 7ePwlezPZ90PNkbl5y2UQDcD901QiciE X-Proofpoint-ORIG-GUID: XNehHQZiANI7ULRyw7QiC_z4UToZJSka X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-01_03,2022-08-31_03,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 phishscore=0 adultscore=0 clxscore=1015 impostorscore=0 mlxlogscore=879 mlxscore=0 malwarescore=0 lowpriorityscore=0 suspectscore=0 priorityscore=1501 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000 definitions=main-2209010022 X-Mailman-Approved-At: Thu, 01 Sep 2022 16:45:35 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, Benjamin Gray , ajd@linux.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These tests are out-of-line only, so moving them to the out-of-line file allows them to be run when an arch does not implement inline static calls. Signed-off-by: Benjamin Gray --- kernel/static_call.c | 43 +++++++++++++++++++++++++++++++++++++ kernel/static_call_inline.c | 43 ------------------------------------- 2 files changed, 43 insertions(+), 43 deletions(-) diff --git a/kernel/static_call.c b/kernel/static_call.c index e9c3e69f3837..953ec46e5691 100644 --- a/kernel/static_call.c +++ b/kernel/static_call.c @@ -6,3 +6,46 @@ long __static_call_return0(void) return 0; } EXPORT_SYMBOL_GPL(__static_call_return0); + +#ifdef CONFIG_STATIC_CALL_SELFTEST + +static int func_a(int x) +{ + return x+1; +} + +static int func_b(int x) +{ + return x+2; +} + +DEFINE_STATIC_CALL(sc_selftest, func_a); + +static struct static_call_data { + int (*func)(int); + int val; + int expect; +} static_call_data [] __initdata = { + { NULL, 2, 3 }, + { func_b, 2, 4 }, + { func_a, 2, 3 } +}; + +static int __init test_static_call_init(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { + struct static_call_data *scd = &static_call_data[i]; + + if (scd->func) + static_call_update(sc_selftest, scd->func); + + WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); + } + + return 0; +} +early_initcall(test_static_call_init); + +#endif /* CONFIG_STATIC_CALL_SELFTEST */ diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c index dc5665b62814..64d04d054698 100644 --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -498,46 +498,3 @@ int __init static_call_init(void) return 0; } early_initcall(static_call_init); - -#ifdef CONFIG_STATIC_CALL_SELFTEST - -static int func_a(int x) -{ - return x+1; -} - -static int func_b(int x) -{ - return x+2; -} - -DEFINE_STATIC_CALL(sc_selftest, func_a); - -static struct static_call_data { - int (*func)(int); - int val; - int expect; -} static_call_data [] __initdata = { - { NULL, 2, 3 }, - { func_b, 2, 4 }, - { func_a, 2, 3 } -}; - -static int __init test_static_call_init(void) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { - struct static_call_data *scd = &static_call_data[i]; - - if (scd->func) - static_call_update(sc_selftest, scd->func); - - WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); - } - - return 0; -} -early_initcall(test_static_call_init); - -#endif /* CONFIG_STATIC_CALL_SELFTEST */ From patchwork Thu Sep 1 05:58:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1672673 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Fgd5Bkhm; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MJBQq2T49z1yhR for ; Thu, 1 Sep 2022 16:46:27 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MJBQq0mVPz3c3B for ; Thu, 1 Sep 2022 16:46:27 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Fgd5Bkhm; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Fgd5Bkhm; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MJ9Ml0n6bz3bZB for ; Thu, 1 Sep 2022 15:58:42 +1000 (AEST) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2815YtAm009081; Thu, 1 Sep 2022 05:58:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=2ceMbmuPz7+4lyZ0SYcNbtzSIgfdR2p6RPESoTtegSc=; b=Fgd5BkhmfbgAUL4Pu74lnMgtfTVFxwc4JjYdx1kbcgxQPgZ2qyAhVHLhF79Brp1Synhz 0SQJFac4gG+xB3liEVFdMUQKNGuegWZ+csVVLh78yGxa0YTVjUMb40EbOvu08ALwgRde omUyXvyfsIGo9ynxo91a+rZXW6ymLsDVeF6gUGCLRC0dbvj0sU/w2bMG+jqOVtddiOA6 r5j0ODZ3ZFnt78taJJ7AUhDr2oXh0LUg+9O+PyCUSa3TWbVFV1c3+ZAQ/ZXTW0wwvzMU DFM+c02Ko0Az/pS0Ydnr+tJum7q9pg6YQhrhsAN/Sa3/T2/yYN94sO0hHGRx88YXyfkE MQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japqf0sxw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:36 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2815a1O7015863; Thu, 1 Sep 2022 05:58:36 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3japqf0swq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:36 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 2815pqYO021978; Thu, 1 Sep 2022 05:58:34 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma06ams.nl.ibm.com with ESMTP id 3j7ahj68w8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:34 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 2815wVY435062240 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 1 Sep 2022 05:58:31 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CD02CAE051; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2F419AE055; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id D5A3A604D1; Thu, 1 Sep 2022 15:58:28 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/4] powerpc/64: Add support for out-of-line static calls Date: Thu, 1 Sep 2022 15:58:22 +1000 Message-Id: <20220901055823.152983-4-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220901055823.152983-1-bgray@linux.ibm.com> References: <20220901055823.152983-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: OICweyCOCxh8_wFUjuDB83OheXYOlIvT X-Proofpoint-ORIG-GUID: ajxkcKND9i2hRpvZEhmjjZl7_VH09PCG X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-01_03,2022-08-31_03,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=0 priorityscore=1501 mlxscore=0 malwarescore=0 clxscore=1011 phishscore=0 mlxlogscore=971 bulkscore=0 impostorscore=0 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000 definitions=main-2209010023 X-Mailman-Approved-At: Thu, 01 Sep 2022 16:45:35 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, Benjamin Gray , ajd@linux.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implement static call support for 64 bit V2 ABI. This requires making sure the TOC is kept correct across kernel-module boundaries. As a secondary concern, it tries to use the local entry point of a target wherever possible. It does so by checking if both tramp & target are kernel code, and falls back to detecting the common global entry point patterns if modules are involved. Detecting the global entry point is also required for setting the local entry point as the trampoline target: if we cannot detect the local entry point, then we need to convservatively initialise r12 and use the global entry point. The implementation is incorrect as-is, because a trampoline cannot use the stack. Doing so has the same issue described in 85baa095, where parameters passed relative to the stack pointer (large arg count or varargs) are broken. However the trampoline must guarantee the TOC be restored before the caller uses it again. Static call sites are themselves static, so it is feasible to handle this by patching the callsites. However the linker currently does not seem to realise that the trampoline treats r2 as volatile (even with an alternative trampoline that does not have a separate local entry point), and so does not insert the appropriate restoration at the call site. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 13 +++- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/include/asm/static_call.h | 45 ++++++++++++- arch/powerpc/kernel/Makefile | 3 +- arch/powerpc/kernel/static_call.c | 80 ++++++++++++++++++++++-- 5 files changed, 132 insertions(+), 10 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 4c466acdc70d..1d5abbeb2c40 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -248,7 +248,7 @@ config PPC select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2) select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13) - select HAVE_STATIC_CALL if PPC32 + select HAVE_STATIC_CALL if PPC_ENABLE_STATIC_CALL select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HUGETLB_PAGE_SIZE_VARIABLE if PPC_BOOK3S_64 && HUGETLB_PAGE @@ -1023,6 +1023,17 @@ config PPC_RTAS_FILTER Say Y unless you know what you are doing and the filter is causing problems for you. +config PPC_ENABLE_STATIC_CALL + bool "Enable static calls" + default y + depends on PPC32 || PPC64_ELF_ABI_V2 + help + PowerPC static calls with the ELF V2 ABI are not as straightforward + as checking if the target is in range of a branch or not. They must + also ensure the TOC remains consistent. This leads to complex + performance results, so it's useful to make them configurable to + allow testing with the generic implementation too. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 3de90748bce7..319cb1eef71c 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -126,6 +126,7 @@ int translate_branch(ppc_inst_t *instr, const u32 *dest, const u32 *src); bool is_conditional_branch(ppc_inst_t instr); #define OP_RT_RA_MASK 0xffff0000UL +#define OP_SI_MASK 0x0000ffffUL #define LIS_R2 (PPC_RAW_LIS(_R2, 0)) #define ADDIS_R2_R12 (PPC_RAW_ADDIS(_R2, _R12, 0)) #define ADDI_R2_R2 (PPC_RAW_ADDI(_R2, _R2, 0)) diff --git a/arch/powerpc/include/asm/static_call.h b/arch/powerpc/include/asm/static_call.h index de1018cc522b..d85ff3f88c8e 100644 --- a/arch/powerpc/include/asm/static_call.h +++ b/arch/powerpc/include/asm/static_call.h @@ -2,12 +2,49 @@ #ifndef _ASM_POWERPC_STATIC_CALL_H #define _ASM_POWERPC_STATIC_CALL_H +#if defined(CONFIG_PPC64_ELF_ABI_V2) + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 6 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " addis 2, 12, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@ha \n" \ + " addi 2, 2, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@l \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", .-" STATIC_CALL_TRAMP_STR(name) "\n" \ + " " inst " \n" \ + " mflr 0 \n" \ + " std 0, 16(1) \n" \ + " stdu 1, -32(1) \n" \ + " std 2, 24(1) \n" \ + " addis 12, 2, 2f@toc@ha \n" \ + " ld 12, 2f@toc@l(12) \n" \ + " mtctr 12 \n" \ + " bctrl \n" \ + " ld 2, 24(1) \n" \ + " addi 1, 1, 32 \n" \ + " ld 0, 16(1) \n" \ + " mtlr 0 \n" \ + " blr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#define PPC_SCT_RET0 64 /* Offset of label 1 */ +#define PPC_SCT_DATA 72 /* Offset of label 2 (aligned) */ + +#elif defined(PPC32) + #define __PPC_SCT(name, inst) \ asm(".pushsection .text, \"ax\" \n" \ ".align 5 \n" \ ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ STATIC_CALL_TRAMP_STR(name) ": \n" \ - inst " \n" \ + " " inst " \n" \ " lis 12,2f@ha \n" \ " lwz 12,2f@l(12) \n" \ " mtctr 12 \n" \ @@ -22,8 +59,12 @@ #define PPC_SCT_RET0 20 /* Offset of label 1 */ #define PPC_SCT_DATA 28 /* Offset of label 2 */ +#else /* !CONFIG_PPC64_ELF_ABI_V2 && !PPC32 */ +#error "Unsupported ABI" +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ + #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) __PPC_SCT(name, "b " #func) #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) __PPC_SCT(name, "blr") -#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b .+20") +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b 1f") #endif /* _ASM_POWERPC_STATIC_CALL_H */ diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index 06d2d1f78f71..a30d0d0f5499 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -128,8 +128,9 @@ extra-y += vmlinux.lds obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o -obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o static_call.o +obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index 863a7aa24650..ed3bc361fdb0 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -1,33 +1,101 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include #include #include +#include -#include +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +static void* ppc_function_toc(u32 *func) { + u32 insn1 = *func; + u32 insn2 = *(func+1); + u64 si1 = sign_extend64((insn1 & OP_SI_MASK) << 16, 31); + u64 si2 = sign_extend64(insn2 & OP_SI_MASK, 15); + u64 addr = ((u64) func + si1) + si2; + + if ((((insn1 & OP_RT_RA_MASK) == ADDIS_R2_R12) || + ((insn1 & OP_RT_RA_MASK) == LIS_R2)) && + ((insn2 & OP_RT_RA_MASK) == ADDI_R2_R2)) + return (void*) addr; + else + return NULL; +} + +static bool shares_toc(void *func1, void *func2) { + void* func1_toc; + void* func2_toc; + + if (func1 == NULL || func2 == NULL) + return false; + + /* Assume the kernel only uses a single TOC */ + if (core_kernel_text((unsigned long)func1) && + core_kernel_text((unsigned long)func2)) + return true; + + /* Fall back to calculating the TOC from common patterns + * if modules are involved + */ + func1_toc = ppc_function_toc(func1); + func2_toc = ppc_function_toc(func2); + return func1_toc != NULL && func2_toc != NULL && (func1_toc == func2_toc); +} + +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) { int err; bool is_ret0 = (func == __static_call_return0); unsigned long target = (unsigned long)(is_ret0 ? tramp + PPC_SCT_RET0 : func); - bool is_short = is_offset_in_branch_range((long)target - (long)tramp); + bool is_short; + void* tramp_entry; if (!tramp) return; + tramp_entry = (void*)ppc_function_entry(tramp); + +#ifdef CONFIG_PPC64_ELF_ABI_V2 + if (shares_toc(tramp, (void*)target)) { + /* Confirm that the local entry point is in range */ + is_short = is_offset_in_branch_range( + (long)ppc_function_entry((void*)target) - (long)tramp_entry); + } else { + /* Combine out-of-range with not sharing a TOC. Though it's possible an + * out-of-range target shares a TOC, handling this separately complicates + * the trampoline. It's simpler to always use the global entry point + * in this case. + */ + is_short = false; + } +#else /* !CONFIG_PPC64_ELF_ABI_V2 */ + is_short = is_offset_in_branch_range((long)target - (long)tramp); +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ + mutex_lock(&text_mutex); if (func && !is_short) { - err = patch_instruction(tramp + PPC_SCT_DATA, ppc_inst(target)); + /* This assumes that the update is atomic. The current implementation uses + * stw/std and the store location is aligned. A sync is issued by one of the + * patch_instruction/patch_branch functions below. + */ + err = PTR_ERR_OR_ZERO(patch_memory( + tramp + PPC_SCT_DATA, &target, sizeof(target))); if (err) goto out; } if (!func) - err = patch_instruction(tramp, ppc_inst(PPC_RAW_BLR())); + err = patch_instruction(tramp_entry, ppc_inst(PPC_RAW_BLR())); else if (is_short) - err = patch_branch(tramp, target, 0); + err = patch_branch(tramp_entry, ppc_function_entry((void*) target), 0); else - err = patch_instruction(tramp, ppc_inst(PPC_RAW_NOP())); + err = patch_instruction(tramp_entry, ppc_inst(PPC_RAW_NOP())); + out: mutex_unlock(&text_mutex); From patchwork Thu Sep 1 05:58:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1672676 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Pn+JBCQq; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MJBT44gdWz1yhR for ; Thu, 1 Sep 2022 16:48:24 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MJBT41DNKz3cdx for ; Thu, 1 Sep 2022 16:48:24 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Pn+JBCQq; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Pn+JBCQq; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MJ9TX5vg0z30D0 for ; Thu, 1 Sep 2022 16:03:44 +1000 (AEST) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2815u5pD018997; Thu, 1 Sep 2022 06:03:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=oSITJ2+OLn+7/Bi0+cW9B16ZzQjlvXdLOoyH2UmbEO4=; b=Pn+JBCQq+zuR8LGGUpT0K/P9SWpzJSz/xXhvcB/SADWmSfVBDtpsASC0CKCKveJdQ5TC j/Y4diEBJGQMSzUOOV22/tI3nhgweLl/bQcyiSJPP7GDHcOO+1fAg2u4vYoKarGqrpvN TjrGaXkuu8Z3iOZnOF5pU8Xff3wdF8E9ONDJiAdr7lAaTMms0Kp7QLZFbJoCmkBhFE7S pI1r1wQL8bRP2T1TwjHt5SNHtaBJdjDn9zrVirvk3IZemV4/4aCX4Y7kEsbCCUBlZsqi nAI5LYUlwgyK3JwkZ6eXT4Gc8tu6xNczTCHwkX1CDb3Zcuc+u9l79cysfpdpRg1VBy5F dw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jaq1ar6yv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 06:03:39 +0000 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 2815x9nM004844; Thu, 1 Sep 2022 06:03:36 GMT Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jaq1ar6x3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 06:03:36 +0000 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 2815o0r4013867; Thu, 1 Sep 2022 05:58:34 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma03ams.nl.ibm.com with ESMTP id 3j7aw967v0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 01 Sep 2022 05:58:33 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 2815wVc135652062 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 1 Sep 2022 05:58:31 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C2EB252050; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 2543B5204E; Thu, 1 Sep 2022 05:58:31 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id D7F04604D2; Thu, 1 Sep 2022 15:58:28 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/4] powerpc/64: Add tests for out-of-line static calls Date: Thu, 1 Sep 2022 15:58:23 +1000 Message-Id: <20220901055823.152983-5-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220901055823.152983-1-bgray@linux.ibm.com> References: <20220901055823.152983-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: pGYucRoVgabJ2DMaLnsmorlZw1NTypdL X-Proofpoint-ORIG-GUID: zk-rRZvvbsKwAg7K3aOICrlTcgFrlanZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-01_04,2022-08-31_03,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 impostorscore=0 priorityscore=1501 mlxscore=0 phishscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000 definitions=main-2209010025 X-Mailman-Approved-At: Thu, 01 Sep 2022 16:45:35 +1000 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, Benjamin Gray , ajd@linux.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" KUnit tests for the various combinations of caller/trampoline/target and kernel/module. They must be run from a module loaded at runtime to guarantee they have a different TOC to the kernel. The tests try to mitigate the chance of panicing by restoring the TOC after every static call. Not all possible errors can be caught by this, but it makes certain kinds of errors easier to debug. Currently the large arg count test is failing due to the trampoline implementation using the stack. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 10 + arch/powerpc/kernel/Makefile | 1 + arch/powerpc/kernel/static_call.c | 104 ++++++++++ arch/powerpc/kernel/static_call_test.c | 257 +++++++++++++++++++++++++ 4 files changed, 372 insertions(+) create mode 100644 arch/powerpc/kernel/static_call_test.c diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1d5abbeb2c40..39e17c35e885 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -1034,6 +1034,16 @@ config PPC_ENABLE_STATIC_CALL performance results, so it's useful to make them configurable to allow testing with the generic implementation too. +config PPC_STATIC_CALL_KUNIT_TEST + tristate "KUnit tests for PPC64 ELF ABI V2 static calls" + default KUNIT_ALL_TESTS + depends on HAVE_STATIC_CALL && PPC64_ELF_ABI_V2 && KUNIT && m + help + Tests that check the TOC is kept consistent across all combinations + of caller/trampoline/target being kernel/module. Must be built as a + module and loaded at runtime to ensure the module has a different + TOC to the kernel. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index a30d0d0f5499..22c07e3d34df 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -131,6 +131,7 @@ obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o +obj-$(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) += static_call_test.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index ed3bc361fdb0..93a4ab80b451 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -7,6 +7,10 @@ #include #include +#if IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) +#include +#endif + #ifdef CONFIG_PPC64_ELF_ABI_V2 static void* ppc_function_toc(u32 *func) { @@ -103,3 +107,103 @@ void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) panic("%s: patching failed %pS at %pS\n", __func__, func, tramp); } EXPORT_SYMBOL_GPL(arch_static_call_transform); + + +#if IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) + +/* The following are some kernel hooks for testing the static call + * implementation from the static_call_test module. The bulk of the + * assertions are run in that module, except for the TOC checks that + * must be done in the core kernel context. + */ + +/* Reserve these registers for testing (same registers as in static_call_test.c) */ +register void* current_toc asm ("r2"); +register void* module_toc asm ("r14"); +register void* actual_module_toc asm ("r15"); +register void* kernel_toc asm ("r16"); +register void* actual_toc asm ("r17"); + +static void* static_kernel_toc; +static void* static_actual_toc; + +#define fixup_toc(test) \ + actual_toc = current_toc; \ + current_toc = kernel_toc; \ + KUNIT_EXPECT_PTR_EQ(test, kernel_toc, actual_toc) + +void ppc_sc_kernel_toc_init(void) +{ + static_kernel_toc = kernel_toc; + static_actual_toc = actual_toc; + + kernel_toc = current_toc; +} + +void ppc_sc_kernel_toc_exit(void) +{ + kernel_toc = static_kernel_toc; + actual_toc = static_actual_toc; +} + +int ppc_sc_kernel_target_1(struct kunit* test) +{ + fixup_toc(test); + return 1; +} + +int ppc_sc_kernel_target_2(struct kunit* test) +{ + fixup_toc(test); + return 2; +} + +DEFINE_STATIC_CALL(ppc_sc_kernel, ppc_sc_kernel_target_1); + +int ppc_sc_kernel_call(struct kunit* test) +{ + int ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + return ret; +} + +int ppc_sc_kernel_call_indirect(struct kunit* test, int (*fn)(struct kunit*)) +{ + int ret = fn(test); + fixup_toc(test); + return ret; +} + +long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i) +{ + fixup_toc(test); + KUNIT_EXPECT_EQ(test, a, b); + KUNIT_EXPECT_EQ(test, a, c); + KUNIT_EXPECT_EQ(test, a, d); + KUNIT_EXPECT_EQ(test, a, e); + KUNIT_EXPECT_EQ(test, a, f); + KUNIT_EXPECT_EQ(test, a, g); + KUNIT_EXPECT_EQ(test, a, h); + KUNIT_EXPECT_EQ(test, a, i); + return ~a; +} + +EXPORT_SYMBOL_GPL(ppc_sc_kernel_toc_init); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_toc_exit); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_1); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_2); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_big); +EXPORT_STATIC_CALL_GPL(ppc_sc_kernel); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call_indirect); + +#endif /* IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) */ diff --git a/arch/powerpc/kernel/static_call_test.c b/arch/powerpc/kernel/static_call_test.c new file mode 100644 index 000000000000..136227a623b4 --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.c @@ -0,0 +1,257 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +/* + * Tests to ensure correctness in a variety of cases for static calls. + * + * The tests focus on ensuring the TOC is kept consistent across the + * module-kernel boundary, as compilers can't see that a trampoline + * defined locally in the kernel might be jumping to a function in a + * module. This makes it important that these tests are compiled as a + * module, so the TOC will be different to the kernel's. + * + * Register variables are used to allow easy position independent + * correction of a TOC before it is used for anything. This means + * a failing test doesn't always crash the whole kernel. The registers + * are initialised on entry and restored on exit of each test using + * KUnit's init and exit hooks. The tests only call internal and + * specially defined kernel functions, so the use of these registers + * will not clobber anything else. + */ + +extern void ppc_sc_kernel_toc_init(void); +extern void ppc_sc_kernel_toc_exit(void); +DECLARE_STATIC_CALL(ppc_sc_kernel, int(struct kunit*)); +extern int ppc_sc_kernel_target_1(struct kunit* test); +extern int ppc_sc_kernel_target_2(struct kunit* test); +extern long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i); +extern int ppc_sc_kernel_call(struct kunit* test); +extern int ppc_sc_kernel_call_indirect(struct kunit* test, int(*fn)(struct kunit*)); + +/* Registers we reserve for use while testing */ +register void* current_toc asm ("r2"); +register void* module_toc asm ("r14"); +register void* actual_toc asm ("r15"); +register void* kernel_toc asm ("r16"); +register void* actual_kernel_toc asm ("r17"); + +/* To hold a copy of the old register values while we test */ +static void* static_module_toc; +static void* static_actual_toc; + +/* Restore the TOC to a known correct value and check if the prior value was bad */ +#define fixup_toc(test) \ + actual_toc = current_toc; \ + current_toc = module_toc; \ + KUNIT_EXPECT_PTR_EQ(test, module_toc, actual_toc) + +static int module_target_11(struct kunit *test) +{ + fixup_toc(test); + return 11; +} + +static int module_target_12(struct kunit *test) +{ + fixup_toc(test); + return 12; +} + +DEFINE_STATIC_CALL(module_sc, module_target_11); + +DEFINE_STATIC_CALL_RET0(module_sc_ret0, long(void)); +DEFINE_STATIC_CALL_NULL(module_sc_null, long(long)); + +static long add_one(long *val) +{ + return (*val)++; +} + +static void null_function_test(struct kunit *test) +{ + long val = 0; + + /* Check argument unconditionally evaluated */ + static_call_cond(module_sc_null)(add_one(&val)); + fixup_toc(test); + KUNIT_ASSERT_EQ(test, 1, val); +} + +static void return_zero_test(struct kunit *test) +{ + long ret; + KUNIT_ASSERT_EQ(test, 0, static_call(module_sc_ret0)()); + + static_call_update(ppc_sc_kernel, (void*) __static_call_return0); + ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(module_sc, (void*) __static_call_return0); + ret = static_call(module_sc)(test); + fixup_toc(test); + KUNIT_ASSERT_EQ(test, 0, ret); +} + +static void kernel_kernel_kernel_test(struct kunit *test) +{ + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + KUNIT_EXPECT_EQ(test, 1, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + KUNIT_EXPECT_EQ(test, 2, ppc_sc_kernel_call(test)); +} + +static void kernel_kernel_module_test(struct kunit *test) +{ + static_call_update(ppc_sc_kernel, module_target_11); + KUNIT_EXPECT_EQ(test, 11, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, module_target_12); + KUNIT_EXPECT_EQ(test, 12, ppc_sc_kernel_call(test)); +} + +static void kernel_module_kernel_test(struct kunit *test) +{ + static_call_update(module_sc, ppc_sc_kernel_target_1); + KUNIT_EXPECT_EQ(test, 1, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + KUNIT_EXPECT_EQ(test, 2, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); +} + +static void kernel_module_module_test(struct kunit *test) +{ + static_call_update(module_sc, module_target_11); + KUNIT_EXPECT_EQ(test, 11, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, module_target_12); + KUNIT_EXPECT_EQ(test, 12, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); +} + +static void module_kernel_kernel_test(struct kunit *test) +{ + long ret; + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 1, ret); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 2, ret); +} + +static void module_kernel_module_test(struct kunit *test) +{ + long ret; + + static_call_update(ppc_sc_kernel, module_target_11); + ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 11, ret); + + static_call_update(ppc_sc_kernel, module_target_12); + ret = static_call(ppc_sc_kernel)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 12, ret); +} + +static void module_module_kernel_test(struct kunit *test) +{ + long ret; + + static_call_update(module_sc, ppc_sc_kernel_target_1); + ret = static_call(module_sc)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 1, ret); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + ret = static_call(module_sc)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 2,ret); +} + +static void module_module_module_test(struct kunit *test) +{ + long ret; + + static_call_update(module_sc, module_target_11); + ret = static_call(module_sc)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 11, ret); + + static_call_update(module_sc, module_target_12); + ret = static_call(module_sc)(test); + fixup_toc(test); + KUNIT_EXPECT_EQ(test, 12, ret); +} + +DEFINE_STATIC_CALL(module_sc_stack_params, ppc_sc_kernel_target_big); + +static void stack_parameters_test(struct kunit *test) +{ + long m = 0x1234567887654321; + long ret = static_call(module_sc_stack_params)(test, m, m, m, m, m, m, m, m, m); + fixup_toc(test); + KUNIT_ASSERT_EQ(test, ~m, ret); +} + +static struct kunit_case static_call_test_cases[] = { + KUNIT_CASE(null_function_test), + KUNIT_CASE(return_zero_test), + KUNIT_CASE(stack_parameters_test), + KUNIT_CASE(kernel_kernel_kernel_test), + KUNIT_CASE(kernel_kernel_module_test), + KUNIT_CASE(kernel_module_kernel_test), + KUNIT_CASE(kernel_module_module_test), + KUNIT_CASE(module_kernel_kernel_test), + KUNIT_CASE(module_kernel_module_test), + KUNIT_CASE(module_module_kernel_test), + KUNIT_CASE(module_module_module_test), + {} +}; + +static int ppc_static_call_test_init(struct kunit *test) +{ + static_module_toc = module_toc; + static_actual_toc = actual_toc; + module_toc = current_toc; + + ppc_sc_kernel_toc_init(); + + return 0; +} + +static void ppc_static_call_test_exit(struct kunit *test) +{ + module_toc = static_module_toc; + actual_toc = static_actual_toc; + + ppc_sc_kernel_toc_exit(); +} + +static struct kunit_suite ppc_static_call_test_suite = { + .name = "ppc-static-call", + .test_cases = static_call_test_cases, + .init = ppc_static_call_test_init, + .exit = ppc_static_call_test_exit, +}; +kunit_test_suite(ppc_static_call_test_suite); + +MODULE_AUTHOR("Benjamin Gray "); +MODULE_LICENSE("GPL");