From patchwork Fri Sep 16 06:23:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678541 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qX0cTvdD; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPFb0Hbyz1ynm for ; Fri, 16 Sep 2022 16:25:23 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPFZ6b8Mz3c7y for ; Fri, 16 Sep 2022 16:25:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qX0cTvdD; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qX0cTvdD; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDg5K8Tz2xrH for ; Fri, 16 Sep 2022 16:24:35 +1000 (AEST) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G5fuUu004728; Fri, 16 Sep 2022 06:24:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=cVIvcIO4FNiAS7+bhAOH9qB0SLMv2jv7w0+Z9XT28Xk=; b=qX0cTvdDs8ltUHb5XS9mV/VqSjXZvx71UQRhMxNt8D5LSvraiR+9FC0ZQPd8bw4Bxc/L sjQl31X65PcOGrS1ZtHMVF3xUW/5IZ6vDRi4FTkGWTaUCZ64DNWs+fiaHUsHxiaQonhL yM9IY5tS5fSKKcI1gxmc9qvb3OBcYQBPKV7A+OKAhJ94EXVos+jSA8F7SgX3GQi1lWHb AjnUvRzhjhwqMLu8FPzaf9553vu2WUF0UySL9leEb6w7AhkUjwoJiD3Qqt0tgGokx6T7 cOTC+IYkChqKmKacg5gjuwJmdbkxDpn55o85kseO29GNL4QncVyTuNvILnudTil17zlz BQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk7j14tg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:19 +0000 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G5gPR4005743; Fri, 16 Sep 2022 06:24:18 GMT Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk7j14su-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:18 +0000 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6LFCM030575; Fri, 16 Sep 2022 06:24:16 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma05fra.de.ibm.com with ESMTP id 3jm91crdwg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:16 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6ODj538797748 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:13 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AAB1111C050; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0ACFE11C04A; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id B93A8603C2; Fri, 16 Sep 2022 16:24:08 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 1/6] powerpc/code-patching: Implement generic text patching function Date: Fri, 16 Sep 2022 16:23:25 +1000 Message-Id: <20220916062330.430468-2-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 2UaEYHegxjgP1ZKN0lDp-UdhEEJbzLRD X-Proofpoint-ORIG-GUID: 8cypm5SNvo-teIfQQcDkvJCkbBx2KhcA X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 phishscore=0 impostorscore=0 adultscore=0 mlxlogscore=920 malwarescore=0 clxscore=1015 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Adds a generic text patching mechanism for patches of 1, 2, 4, or 8 bytes. The patcher conditionally syncs the icache depending on if the content will be executed (as opposed to, e.g., read-only data). The `patch_instruction` function is reimplemented in terms of this more generic function. This generic implementation allows patching of arbitrary 64-bit data, whereas the original `patch_instruction` decided the size based on the 'instruction' opcode, so was not suitable for arbitrary data. Signed-off-by: Benjamin Gray Reported-by: kernel test robot Reported-by: kernel test robot --- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/lib/code-patching.c | 135 +++++++++++++++-------- 2 files changed, 89 insertions(+), 47 deletions(-) diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 1c6316ec4b74..6a52c19dae46 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -76,6 +76,7 @@ int create_cond_branch(ppc_inst_t *instr, const u32 *addr, int patch_branch(u32 *addr, unsigned long target, int flags); int patch_instruction(u32 *addr, ppc_inst_t instr); int raw_patch_instruction(u32 *addr, ppc_inst_t instr); +int patch_text_data(void *dest, const void *src, size_t size); static inline unsigned long patch_site_addr(s32 *site) { diff --git a/arch/powerpc/lib/code-patching.c b/arch/powerpc/lib/code-patching.c index ad0cf3108dd0..a09a0898c2ce 100644 --- a/arch/powerpc/lib/code-patching.c +++ b/arch/powerpc/lib/code-patching.c @@ -3,6 +3,7 @@ * Copyright 2008 Michael Ellerman, IBM Corporation. */ +#include #include #include #include @@ -14,32 +15,7 @@ #include #include #include - -static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr) -{ - if (!ppc_inst_prefixed(instr)) { - u32 val = ppc_inst_val(instr); - - __put_kernel_nofault(patch_addr, &val, u32, failed); - } else { - u64 val = ppc_inst_as_ulong(instr); - - __put_kernel_nofault(patch_addr, &val, u64, failed); - } - - asm ("dcbst 0, %0; sync; icbi 0,%1; sync; isync" :: "r" (patch_addr), - "r" (exec_addr)); - - return 0; - -failed: - return -EPERM; -} - -int raw_patch_instruction(u32 *addr, ppc_inst_t instr) -{ - return __patch_instruction(addr, instr, addr); -} +#include #ifdef CONFIG_STRICT_KERNEL_RWX static DEFINE_PER_CPU(struct vm_struct *, text_poke_area); @@ -147,16 +123,44 @@ static void unmap_patch_area(unsigned long addr) flush_tlb_kernel_range(addr, addr + PAGE_SIZE); } -static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int __patch_text(void *dest, const void *src, size_t size, bool is_exec, void *exec_addr) +{ + if (virt_to_pfn(dest) != virt_to_pfn(dest + size - 1)) + return -EFAULT; + + switch (size) { + case 1: + __put_kernel_nofault(dest, src, u8, failed); + break; + case 2: + __put_kernel_nofault(dest, src, u16, failed); + break; + case 4: + __put_kernel_nofault(dest, src, u32, failed); + break; + case 8: + __put_kernel_nofault(dest, src, u64, failed); + break; + } + + asm ("dcbst 0, %0; sync" :: "r" (dest)); + + if (is_exec) + asm ("icbi 0,%0; sync; isync" :: "r" (exec_addr)); + + return 0; + +failed: + return -EPERM; +} + +static pte_t *start_text_patch(void* dest, u32 **patch_addr) { - int err; - u32 *patch_addr; - unsigned long text_poke_addr; pte_t *pte; - unsigned long pfn = get_patch_pfn(addr); + unsigned long text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK; + unsigned long pfn = get_patch_pfn(dest); - text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK; - patch_addr = (u32 *)(text_poke_addr + offset_in_page(addr)); + *patch_addr = (u32 *)(text_poke_addr + offset_in_page(dest)); pte = virt_to_kpte(text_poke_addr); __set_pte_at(&init_mm, text_poke_addr, pte, pfn_pte(pfn, PAGE_KERNEL), 0); @@ -164,33 +168,63 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) if (radix_enabled()) asm volatile("ptesync": : :"memory"); - err = __patch_instruction(addr, instr, patch_addr); + return pte; +} +static void finish_text_patch(pte_t *pte) +{ + unsigned long text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK; pte_clear(&init_mm, text_poke_addr, pte); flush_tlb_kernel_range(text_poke_addr, text_poke_addr + PAGE_SIZE); +} + +static int do_patch_text(void *dest, const void *src, size_t size, bool is_exec) +{ + int err; + pte_t *pte; + u32 *patch_addr; + + pte = start_text_patch(dest, &patch_addr); + err = __patch_text(patch_addr, src, size, is_exec, dest); + finish_text_patch(pte); return err; } -static int do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int patch_text(void *dest, const void *src, size_t size, bool is_exec) { int err; unsigned long flags; - /* - * During early early boot patch_instruction is called - * when text_poke_area is not ready, but we still need - * to allow patching. We just do the plain old patching - */ + /* Make sure we aren't patching a freed init section */ + if (static_branch_likely(&init_mem_is_free) && init_section_contains(dest, 4)) + return 0; + if (!static_branch_likely(&poking_init_done)) - return raw_patch_instruction(addr, instr); + return __patch_text(dest, src, size, is_exec, dest); local_irq_save(flags); - err = __do_patch_instruction(addr, instr); + err = do_patch_text(dest, src, size, is_exec); local_irq_restore(flags); return err; } + +int patch_text_data(void *dest, const void *src, size_t size) { + return patch_text(dest, src, size, false); +} + +int raw_patch_instruction(u32 *addr, ppc_inst_t instr) +{ + if (!ppc_inst_prefixed(instr)) { + u32 val = ppc_inst_val(instr); + return __patch_text(addr, &val, sizeof(val), true, addr); + } else { + u64 val = ppc_inst_as_ulong(instr); + return __patch_text(addr, &val, sizeof(val), true, addr); + } +} + #else /* !CONFIG_STRICT_KERNEL_RWX */ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) @@ -198,17 +232,24 @@ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) return raw_patch_instruction(addr, instr); } +void *patch_memory(void *dest, const void *src, size_t size) +{ + return memcpy(dest, src, size); +} + #endif /* CONFIG_STRICT_KERNEL_RWX */ __ro_after_init DEFINE_STATIC_KEY_FALSE(init_mem_is_free); int patch_instruction(u32 *addr, ppc_inst_t instr) { - /* Make sure we aren't patching a freed init section */ - if (static_branch_likely(&init_mem_is_free) && init_section_contains(addr, 4)) - return 0; - - return do_patch_instruction(addr, instr); + if (!ppc_inst_prefixed(instr)) { + u32 val = ppc_inst_val(instr); + return patch_text(addr, &val, sizeof(val), true); + } else { + u64 val = ppc_inst_as_ulong(instr); + return patch_text(addr, &val, sizeof(val), true); + } } NOKPROBE_SYMBOL(patch_instruction); From patchwork Fri Sep 16 06:23:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678540 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TjdnY0wh; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPDw64ZKz1yp0 for ; Fri, 16 Sep 2022 16:24:48 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPDt481cz3c18 for ; Fri, 16 Sep 2022 16:24:46 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TjdnY0wh; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=TjdnY0wh; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDg13Hqz2xjt for ; Fri, 16 Sep 2022 16:24:34 +1000 (AEST) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G5hlrs010067; Fri, 16 Sep 2022 06:24:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=h5UBtJeIO6ZtF/gYD5gMZ0KDP0Wzw4KIjF+qDCxbDJ4=; b=TjdnY0whPXvg8vDMPZWnR8Pc1Fy5I+Kc40k7bkN6XNr5ebypvS8++O0CJnvj3Cj4L+XX l8PUM0tR8CylYXu6LsKDy2DxzD+EWj+hrk3tiJl9ZK4biuCUkRXaWM2CRww142+hmBpH +E3XEDMYQTMixjd5DrK0tQ4E+NNqJWyXqP6gg86PDN4bSnsXWJ6wj8LGOlee3jksASIx mCgAB9vStp6D+JSEUQnnngXQ/v7GBTs9tRn8f7inujtgxygLJlOynh28CA9ZJNMpjMqr 1UOl/idJDlbgplJBZ+oXCKPFFtlwTWRcBwFHc3fop/zgYkSWV5ygsbM7+DdVKT3V70hs YQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk8eh1w1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:18 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G5jDjx015460; Fri, 16 Sep 2022 06:24:17 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk8eh1vg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:17 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6Ko8O003065; Fri, 16 Sep 2022 06:24:15 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma06ams.nl.ibm.com with ESMTP id 3jm9218jjv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:15 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6OD8e23200056 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:13 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 600B74C04A; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 08E6A4C040; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Sep 2022 06:24:12 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 9411F600EE; Fri, 16 Sep 2022 16:24:09 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 2/6] powerpc/module: Handle caller-saved TOC in module linker Date: Fri, 16 Sep 2022 16:23:26 +1000 Message-Id: <20220916062330.430468-3-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: amjqpL4TnH9VLa4tjBrc8-eM5bEBtPaK X-Proofpoint-ORIG-GUID: cOGlfX9_LetUUGVUQUgnkGBWkLYmAi_p X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=980 impostorscore=0 bulkscore=0 clxscore=1011 suspectscore=0 lowpriorityscore=0 malwarescore=0 priorityscore=1501 phishscore=0 spamscore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The callee may set a field in `st_other` to 1 to indicate r2 should be treated as caller-saved. This means a trampoline must be used to save the current TOC before calling it and restore it afterwards, much like external calls. This is necessary for supporting V2 ABI static calls that do not preserve the TOC. Signed-off-by: Benjamin Gray --- arch/powerpc/kernel/module_64.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 7e45dc98df8a..3656476097c2 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -55,6 +55,11 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) * of function and try to derive r2 from it). */ return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other); } + +static bool need_r2save_stub(unsigned char st_other) { + return ((st_other & STO_PPC64_LOCAL_MASK) >> STO_PPC64_LOCAL_BIT) == 1; +} + #else static func_desc_t func_desc(unsigned long addr) @@ -66,6 +71,10 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) return 0; } +static bool need_r2save_stub(unsigned char st_other) { + return false; +} + void *dereference_module_function_descriptor(struct module *mod, void *ptr) { if (ptr < (void *)mod->arch.start_opd || @@ -632,7 +641,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, case R_PPC_REL24: /* FIXME: Handle weak symbols here --RR */ if (sym->st_shndx == SHN_UNDEF || - sym->st_shndx == SHN_LIVEPATCH) { + sym->st_shndx == SHN_LIVEPATCH || + need_r2save_stub(sym->st_other)) { /* External: go via stub */ value = stub_for_addr(sechdrs, value, me, strtab + sym->st_name); From patchwork Fri Sep 16 06:23:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678543 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HGLKn7lL; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPH62Rrlz1ynm for ; Fri, 16 Sep 2022 16:26:42 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPH61vh9z3c6l for ; Fri, 16 Sep 2022 16:26:42 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HGLKn7lL; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=HGLKn7lL; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDg6VsLz2xjt for ; Fri, 16 Sep 2022 16:24:35 +1000 (AEST) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G6Ikxd028722; Fri, 16 Sep 2022 06:24:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=P8yNr1iwyy/anAp6THxhvTwfhUWUo9w7irp8Zx7MnfU=; b=HGLKn7lLyYl1Jbi67+XjTLysQj1r/3egpy+XKTrjimzCR7dPcgM4mpJ7tIFgWUT6D6Oz PY6DP3qGxEGz3j3srbxQp6x0MEujl2L3Mt9JpQR6+yCQiawrmeMLpyrp+f0SMf4Pm/hK 75pgsjJcqvAMNTITRFWTfAmZ1igqU5zMbswigme6DLsqRA/lC4+anchA4s6/4AHroTtX d4wP1Oqu0q2XPE3jh8VfBgkgVwyh0sXOST1QgHWccIZ1xmlZtGEHNV05TSOt9QhT40Nd ij3+7us7jopP3KMpEO/Y75mUdmAzS/UoEsdsNAWV+Ww+3P23j5HkVUkqKPZLlbeDLhPi ag== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmks084v1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:18 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G6K70c003153; Fri, 16 Sep 2022 06:24:18 GMT Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmks084u5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:18 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6KtWp004049; Fri, 16 Sep 2022 06:24:15 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma04ams.nl.ibm.com with ESMTP id 3jm921gjxd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:15 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6OD0v21496196 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:13 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6034F4C04E; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 094744C046; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Sep 2022 06:24:12 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 307A160418; Fri, 16 Sep 2022 16:24:10 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 3/6] powerpc/module: Optimise nearby branches in ELF V2 ABI stub Date: Fri, 16 Sep 2022 16:23:27 +1000 Message-Id: <20220916062330.430468-4-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: UkezhFd9la0-LXB0S7sEPs4f0art40Uo X-Proofpoint-ORIG-GUID: eUhwm14bBXhS2udk_WGM-GCgm_ornfVe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 phishscore=0 adultscore=0 clxscore=1015 bulkscore=0 suspectscore=0 malwarescore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 spamscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Inserts a direct branch to the stub target when possible, replacing the mtctr/btctr sequence. The load into r12 could potentially be skipped too, but that change would need to refactor the arguments to indicate that the address does not have a separate local entry point. This helps the static call implementation, where modules calling their own trampolines are called through this stub and the trampoline is easily within range of a direct branch. Signed-off-by: Benjamin Gray --- arch/powerpc/kernel/module_64.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 3656476097c2..03ab28d86008 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -432,8 +432,17 @@ static inline int create_stub(const Elf64_Shdr *sechdrs, return create_ftrace_stub(entry, addr, me); for (i = 0; i < ARRAY_SIZE(ppc64_stub_insns); i++) { - if (patch_instruction(&entry->jump[i], - ppc_inst(ppc64_stub_insns[i]))) + ppc_inst_t inst = ppc_inst(ppc64_stub_insns[i]); + + // Replace the indirect branch with a direct branch where possible + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) && i == 4) { + ppc_inst_t direct; + if (create_branch(&direct, (void*) entry + (i * 4), addr, 0) == 0) { + inst = direct; + } + } + + if (patch_instruction(&entry->jump[i], inst)) return 0; } From patchwork Fri Sep 16 06:23:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678546 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=thK5Qq8N; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPKP3Y5Sz1ynm for ; Fri, 16 Sep 2022 16:28:41 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPKP3Gdqz3f5s for ; Fri, 16 Sep 2022 16:28:41 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=thK5Qq8N; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=thK5Qq8N; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDk1jCxz2yp5 for ; Fri, 16 Sep 2022 16:24:37 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G5VJc3004109; Fri, 16 Sep 2022 06:24:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9zEaH0k5pQzIL+oDYeijzfgTk3gU9Gj6CrTIUIx03i4=; b=thK5Qq8NJBUJbkvuO5LgKERscX/v2bTVv7YketpE+yEejxlONd44NHS29T4FNYRRoFo8 qj8LGFVIobIByICozUUWwWYVTL6HPsvesJIr90qaf3/MlwL2VlkaR63nsh1pD2KR8mrs tl97z/jEldTGNLRzK1lC0WifwZJhPPInngavEIoH8fcwAp5vMumv+Cgi3g3KFbdRPjVz j0rL1HtRzU8ALcKCeLSBLcG33ps2bDQc5jBdoZ7QG5Bn0gUzE5BoRzv5Mh1wNSMI7e2P oNcT0/Uk92/DWzEorLASF5D1yrm+ZeD3TM2qv6LD66Yv/x4jz1JyQF5/LsKB055AScFP ZA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk2f9jw8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:19 +0000 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G5mBGQ021168; Fri, 16 Sep 2022 06:24:19 GMT Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmk2f9jvf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:18 +0000 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6KK4P032090; Fri, 16 Sep 2022 06:24:16 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma02fra.de.ibm.com with ESMTP id 3jm91t8dsw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:16 +0000 Received: from d06av22.portsmouth.uk.ibm.com (d06av22.portsmouth.uk.ibm.com [9.149.105.58]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6ODR723200062 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:13 GMT Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A907B4C040; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from d06av22.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0917A4C044; Fri, 16 Sep 2022 06:24:13 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av22.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Sep 2022 06:24:12 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 1EEE860412; Fri, 16 Sep 2022 16:24:11 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 4/6] static_call: Move static call selftest to static_call_selftest.c Date: Fri, 16 Sep 2022 16:23:28 +1000 Message-Id: <20220916062330.430468-5-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Dm1Lk27cOOuTENkGtSEAFSdd67C_eIsh X-Proofpoint-GUID: S-Tu-pTKyMPvr1QaFRdv1NaIoheqjjMO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 priorityscore=1501 phishscore=0 impostorscore=0 mlxlogscore=999 suspectscore=0 clxscore=1015 adultscore=0 spamscore=0 bulkscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These tests are out-of-line only, so moving them to the their own file allows them to be run when an arch does not implement inline static calls. Signed-off-by: Benjamin Gray Reviewed-by: Andrew Donnellan --- kernel/Makefile | 1 + kernel/static_call_inline.c | 43 ----------------------------------- kernel/static_call_selftest.c | 41 +++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 43 deletions(-) create mode 100644 kernel/static_call_selftest.c diff --git a/kernel/Makefile b/kernel/Makefile index 318789c728d3..8ce8beaa3cc0 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -113,6 +113,7 @@ obj-$(CONFIG_KCSAN) += kcsan/ obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_HAVE_STATIC_CALL_INLINE) += static_call_inline.o +obj-$(CONFIG_STATIC_CALL_SELFTEST) += static_call_selftest.o obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c index dc5665b62814..64d04d054698 100644 --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -498,46 +498,3 @@ int __init static_call_init(void) return 0; } early_initcall(static_call_init); - -#ifdef CONFIG_STATIC_CALL_SELFTEST - -static int func_a(int x) -{ - return x+1; -} - -static int func_b(int x) -{ - return x+2; -} - -DEFINE_STATIC_CALL(sc_selftest, func_a); - -static struct static_call_data { - int (*func)(int); - int val; - int expect; -} static_call_data [] __initdata = { - { NULL, 2, 3 }, - { func_b, 2, 4 }, - { func_a, 2, 3 } -}; - -static int __init test_static_call_init(void) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { - struct static_call_data *scd = &static_call_data[i]; - - if (scd->func) - static_call_update(sc_selftest, scd->func); - - WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); - } - - return 0; -} -early_initcall(test_static_call_init); - -#endif /* CONFIG_STATIC_CALL_SELFTEST */ diff --git a/kernel/static_call_selftest.c b/kernel/static_call_selftest.c new file mode 100644 index 000000000000..246ad89f64eb --- /dev/null +++ b/kernel/static_call_selftest.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static int func_a(int x) +{ + return x+1; +} + +static int func_b(int x) +{ + return x+2; +} + +DEFINE_STATIC_CALL(sc_selftest, func_a); + +static struct static_call_data { + int (*func)(int); + int val; + int expect; +} static_call_data [] __initdata = { + { NULL, 2, 3 }, + { func_b, 2, 4 }, + { func_a, 2, 3 } +}; + +static int __init test_static_call_init(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { + struct static_call_data *scd = &static_call_data[i]; + + if (scd->func) + static_call_update(sc_selftest, scd->func); + + WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); + } + + return 0; +} +early_initcall(test_static_call_init); From patchwork Fri Sep 16 06:23:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678542 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UpFExbTr; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPGL4CqSz1ynm for ; Fri, 16 Sep 2022 16:26:02 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPGL3hRZz3c8s for ; Fri, 16 Sep 2022 16:26:02 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UpFExbTr; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UpFExbTr; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDg5gM8z2xsD for ; Fri, 16 Sep 2022 16:24:35 +1000 (AEST) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G6IkJt028702; Fri, 16 Sep 2022 06:24:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=IehbMDFhHrgcLjzu9JloQqfQaTegPLgrgg7Ye63N7T4=; b=UpFExbTryhksradwSxJ5tmFk2BmfxYf+1DlkJ2Dj6XWnmTB4+kLnLT28jI1+bTy+dvhf lTYWGl7zlDLGoG/NpcWzJSZ5NyDcTwW6gRvRE1RP0h50wt/bWm2TU324lJzT3eNI62DS Tdv17LeQo93rpwF/RTEAw676B2wxFxaejDY8MOeFfbTZiMoB4LrAJEE6JpvD9CPuqHif mnGIMnEuThn5slrUtamvnNWg4BZVgeHUlCKB/zwMrlW2TouG7ID4xpI0LfBl5f20uXAB f3bwhLY6NPpbZPYTlFxzyHHTNvz9Ns11p2452MUY29C11FLL7rz1Aoov5HXjfBWaHFpO Ug== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmks084vv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:20 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G6J4KY029824; Fri, 16 Sep 2022 06:24:20 GMT Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmks084v0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:20 +0000 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6LFCN030575; Fri, 16 Sep 2022 06:24:17 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma05fra.de.ibm.com with ESMTP id 3jm91crdwh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:17 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6OFVA27066650 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:15 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 374935204F; Fri, 16 Sep 2022 06:24:15 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 89E3F52051; Fri, 16 Sep 2022 06:24:14 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id A7EE760434; Fri, 16 Sep 2022 16:24:11 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 5/6] powerpc/64: Add support for out-of-line static calls Date: Fri, 16 Sep 2022 16:23:29 +1000 Message-Id: <20220916062330.430468-6-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: rH9c1kkJANSfpbIqHVpVZyOlFxq0ccFy X-Proofpoint-ORIG-GUID: BWbsypgdwowzi1lW02jNLA-V6c7SD8n8 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 phishscore=0 adultscore=0 clxscore=1015 bulkscore=0 suspectscore=0 malwarescore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 spamscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implement static call support for 64 bit V2 ABI. This requires making sure the TOC is kept correct across kernel-module boundaries. As a secondary concern, it tries to use the local entry point of a target wherever possible. It does so by checking if both tramp & target are kernel code, and falls back to detecting the common global entry point patterns if modules are involved. Detecting the global entry point is also required for setting the local entry point as the trampoline target: if we cannot detect the local entry point, then we need to convservatively initialise r12 and use the global entry point. The trampolines are marked with `.localentry NAME, 1` to make the linker save and restore the TOC on each call to the trampoline. This allows the trampoline to safely target functions with different TOC values. However this directive also implies the TOC is not initialised on entry to the trampoline. The kernel TOC is easily found in the PACA, but not an arbitrary module TOC. Therefore the trampoline implementation depends on whether it's in the kernel or not. If in the kernel, we initialise the TOC using the PACA. If in a module, we have to initialise the TOC with zero context, so it's quite expensive. Signed-off-by: Benjamin Gray Reported-by: kernel test robot --- arch/powerpc/Kconfig | 12 +++- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/include/asm/static_call.h | 80 +++++++++++++++++++-- arch/powerpc/kernel/Makefile | 3 +- arch/powerpc/kernel/static_call.c | 92 ++++++++++++++++++++++-- 5 files changed, 175 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 4c466acdc70d..d3bb61e0de43 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -248,7 +248,7 @@ config PPC select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2) select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13) - select HAVE_STATIC_CALL if PPC32 + select HAVE_STATIC_CALL if PPC_ENABLE_STATIC_CALL select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HUGETLB_PAGE_SIZE_VARIABLE if PPC_BOOK3S_64 && HUGETLB_PAGE @@ -1023,6 +1023,16 @@ config PPC_RTAS_FILTER Say Y unless you know what you are doing and the filter is causing problems for you. +config PPC_ENABLE_STATIC_CALL + bool "Enable static calls" + default y + depends on PPC32 || PPC64_ELF_ABI_V2 + help + PowerPC static calls with the ELF V2 ABI must ensure the TOC remains + consistent. The benefit is only noticeable when indirect branch prediction + is disabled. Otherwise, the generic implementation is better as it is + inlined and effectively a direct branch with a hot predictor. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 6a52c19dae46..1ab7f3522da2 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -126,6 +126,7 @@ int translate_branch(ppc_inst_t *instr, const u32 *dest, const u32 *src); bool is_conditional_branch(ppc_inst_t instr); #define OP_RT_RA_MASK 0xffff0000UL +#define OP_SI_MASK 0x0000ffffUL #define LIS_R2 (PPC_RAW_LIS(_R2, 0)) #define ADDIS_R2_R12 (PPC_RAW_ADDIS(_R2, _R12, 0)) #define ADDI_R2_R2 (PPC_RAW_ADDI(_R2, _R2, 0)) diff --git a/arch/powerpc/include/asm/static_call.h b/arch/powerpc/include/asm/static_call.h index de1018cc522b..5e763cf16e9c 100644 --- a/arch/powerpc/include/asm/static_call.h +++ b/arch/powerpc/include/asm/static_call.h @@ -2,12 +2,75 @@ #ifndef _ASM_POWERPC_STATIC_CALL_H #define _ASM_POWERPC_STATIC_CALL_H +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +#ifdef MODULE + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 6 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " mflr 11 \n" \ + " bcl 20, 31, $+4 \n" \ + "0: mflr 12 \n" \ + " mtlr 11 \n" \ + " addi 12, 12, (" STATIC_CALL_TRAMP_STR(name) " - 0b) \n" \ + " addis 2, 12, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@ha \n" \ + " addi 2, 2, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@l \n" \ + " " inst " \n" \ + " ld 12, (2f - " STATIC_CALL_TRAMP_STR(name) ")(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#else /* KERNEL */ + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 5 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " ld 2, 16(13) \n" \ + " " inst " \n" \ + " addis 12, 2, 2f@toc@ha \n" \ + " ld 12, 2f@toc@l(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#endif /* MODULE */ + +#define PPC_SCT_INST_MODULE 28 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 44 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 56 /* Offset of label 2 (aligned) */ + +#define PPC_SCT_INST_KERNEL 4 /* Offset of instruction to update */ +#define PPC_SCT_RET0_KERNEL 24 /* Offset of label 1 */ +#define PPC_SCT_DATA_KERNEL 32 /* Offset of label 2 (aligned) */ + +#elif defined(PPC32) + #define __PPC_SCT(name, inst) \ asm(".pushsection .text, \"ax\" \n" \ ".align 5 \n" \ ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ STATIC_CALL_TRAMP_STR(name) ": \n" \ - inst " \n" \ + " " inst " \n" \ " lis 12,2f@ha \n" \ " lwz 12,2f@l(12) \n" \ " mtctr 12 \n" \ @@ -19,11 +82,20 @@ ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ ".popsection \n") -#define PPC_SCT_RET0 20 /* Offset of label 1 */ -#define PPC_SCT_DATA 28 /* Offset of label 2 */ +#define PPC_SCT_INST_MODULE 0 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 20 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 28 /* Offset of label 2 */ + +#define PPC_SCT_INST_KERNEL PPC_SCT_INST_MODULE +#define PPC_SCT_RET0_KERNEL PPC_SCT_RET0_MODULE +#define PPC_SCT_DATA_KERNEL PPC_SCT_DATA_MODULE + +#else /* !CONFIG_PPC64_ELF_ABI_V2 && !PPC32 */ +#error "Unsupported ABI" +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) __PPC_SCT(name, "b " #func) #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) __PPC_SCT(name, "blr") -#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b .+20") +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b 1f") #endif /* _ASM_POWERPC_STATIC_CALL_H */ diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index 06d2d1f78f71..a30d0d0f5499 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -128,8 +128,9 @@ extra-y += vmlinux.lds obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o -obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o static_call.o +obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index 863a7aa24650..d2d9b496381d 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -1,33 +1,111 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include #include #include +#include -#include +static void* ppc_function_toc(u32 *func) { + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)) { + u32 insn1 = *func; + u32 insn2 = *(func+1); + u64 si1 = sign_extend64((insn1 & OP_SI_MASK) << 16, 31); + u64 si2 = sign_extend64(insn2 & OP_SI_MASK, 15); + u64 addr = ((u64) func + si1) + si2; + + if ((((insn1 & OP_RT_RA_MASK) == ADDIS_R2_R12) || + ((insn1 & OP_RT_RA_MASK) == LIS_R2)) && + ((insn2 & OP_RT_RA_MASK) == ADDI_R2_R2)) + return (void*)addr; + } + + return NULL; +} + +static bool shares_toc(void *func1, void *func2) { + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)) { + void* func1_toc; + void* func2_toc; + + if (func1 == NULL || func2 == NULL) + return false; + + /* Assume the kernel only uses a single TOC */ + if (core_kernel_text((unsigned long)func1) && + core_kernel_text((unsigned long)func2)) + return true; + + /* Fall back to calculating the TOC from common patterns + * if modules are involved + */ + func1_toc = ppc_function_toc(func1); + func2_toc = ppc_function_toc(func2); + return func1_toc != NULL && func2_toc != NULL && (func1_toc == func2_toc); + } + + return true; +} + +static unsigned long get_inst_addr(unsigned long tramp) { + return tramp + (core_kernel_text(tramp) ? PPC_SCT_INST_KERNEL : PPC_SCT_INST_MODULE); +} + +static unsigned long get_ret0_addr(unsigned long tramp) { + return tramp + (core_kernel_text(tramp) ? PPC_SCT_RET0_KERNEL : PPC_SCT_RET0_MODULE); +} + +static unsigned long get_data_addr(unsigned long tramp) { + return tramp + (core_kernel_text(tramp) ? PPC_SCT_DATA_KERNEL : PPC_SCT_DATA_MODULE); +} void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) { int err; bool is_ret0 = (func == __static_call_return0); - unsigned long target = (unsigned long)(is_ret0 ? tramp + PPC_SCT_RET0 : func); - bool is_short = is_offset_in_branch_range((long)target - (long)tramp); + unsigned long target = is_ret0 + ? get_ret0_addr((unsigned long)tramp) + : (unsigned long)func; + bool is_short; + void* tramp_inst; if (!tramp) return; + tramp_inst = (void*)get_inst_addr((unsigned long)tramp); + + if (is_ret0) { + is_short = true; + } else if (shares_toc(tramp, (void*)target)) { + /* Confirm that the local entry point is in range */ + is_short = is_offset_in_branch_range( + (long)ppc_function_entry((void*)target) - (long)tramp_inst); + } else { + /* Combine out-of-range with not sharing a TOC. Though it's possible an + * out-of-range target shares a TOC, handling this separately complicates + * the trampoline. It's simpler to always use the global entry point + * in this case. + */ + is_short = false; + } + mutex_lock(&text_mutex); if (func && !is_short) { - err = patch_instruction(tramp + PPC_SCT_DATA, ppc_inst(target)); + err = patch_text_data( + (void*)get_data_addr((unsigned long)tramp), &target, sizeof(target)); if (err) goto out; } if (!func) - err = patch_instruction(tramp, ppc_inst(PPC_RAW_BLR())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_BLR())); else if (is_short) - err = patch_branch(tramp, target, 0); + err = patch_branch(tramp_inst, ppc_function_entry((void*)target), 0); else - err = patch_instruction(tramp, ppc_inst(PPC_RAW_NOP())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_NOP())); + out: mutex_unlock(&text_mutex); From patchwork Fri Sep 16 06:23:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1678545 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=e0yZ4xk+; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MTPJf2kHSz1ynm for ; Fri, 16 Sep 2022 16:28:02 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MTPJd0Q1tz3f36 for ; Fri, 16 Sep 2022 16:28:01 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=e0yZ4xk+; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=e0yZ4xk+; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MTPDk04dFz2yWK for ; Fri, 16 Sep 2022 16:24:37 +1000 (AEST) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28G6Heh8029931; Fri, 16 Sep 2022 06:24:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=sdncVRBDViRHdXFPa9Z9aSoWknzzf1hGhAqh7aJlS7g=; b=e0yZ4xk+a6LMf2L4mS0lHuOVC1+1FTcqG/Rsi0oAasLNv067bJBJicT4Sg9acJJsTQwe 0dTomywdwXcJ4paPOTbBAsyDcmUU6ZR9CI5X2FORirullNfNBX2tg/xIcVd68qYMzhHa UDAAYnX44SypvoQN0XwlMwNe2EH/p+1CZDZrd5Lb3ZhmmTShUsCMQhFzqwUyjPNLfC13 TwrD0CZDvFnpyAgAaAIWtWStkQNOD0du1gGyqhT4ABUhgiwVsPSZslC16Vcd9gREyQQH J9wDXQefvWAo2eqfMJNG7tm7vGYrwKS5vZniat3XNAFDwLG4Ep58g1qSHJJiU06TYp+v bw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmkrfg5nb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:20 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28G6IqpW003015; Fri, 16 Sep 2022 06:24:19 GMT Received: from ppma04fra.de.ibm.com (6a.4a.5195.ip4.static.sl-reverse.com [149.81.74.106]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jmkrfg5mq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:19 +0000 Received: from pps.filterd (ppma04fra.de.ibm.com [127.0.0.1]) by ppma04fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28G6LEeW013452; Fri, 16 Sep 2022 06:24:17 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma04fra.de.ibm.com with ESMTP id 3jm9198dv2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 16 Sep 2022 06:24:17 +0000 Received: from b06wcsmtp001.portsmouth.uk.ibm.com (b06wcsmtp001.portsmouth.uk.ibm.com [9.149.105.160]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28G6OFYT38207802 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 16 Sep 2022 06:24:15 GMT Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3D40FA4054; Fri, 16 Sep 2022 06:24:15 +0000 (GMT) Received: from b06wcsmtp001.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9219FA405B; Fri, 16 Sep 2022 06:24:14 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by b06wcsmtp001.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 16 Sep 2022 06:24:14 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 30B7B60425; Fri, 16 Sep 2022 16:24:12 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 6/6] powerpc/64: Add tests for out-of-line static calls Date: Fri, 16 Sep 2022 16:23:30 +1000 Message-Id: <20220916062330.430468-7-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220916062330.430468-1-bgray@linux.ibm.com> References: <20220916062330.430468-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: MFS0Mjnac-r5Y3XDgmOhSLBjhlajZCxB X-Proofpoint-GUID: sfTjQyh8SS823snXeALQynQJIX7weUon X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-16_02,2022-09-14_04,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 adultscore=0 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 lowpriorityscore=0 mlxlogscore=999 impostorscore=0 bulkscore=0 clxscore=1015 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209160045 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" KUnit tests for the various combinations of caller/trampoline/target and kernel/module. They must be run from a module loaded at runtime to guarantee they have a different TOC to the kernel. The tests try to mitigate the chance of panicing by restoring the TOC after every static call. Not all possible errors can be caught by this (we can't stop a trampoline from using a bad TOC itself), but it makes certain errors easier to debug. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 10 + arch/powerpc/kernel/Makefile | 1 + arch/powerpc/kernel/static_call.c | 111 +++++++++++ arch/powerpc/kernel/static_call_test.c | 263 +++++++++++++++++++++++++ 4 files changed, 385 insertions(+) create mode 100644 arch/powerpc/kernel/static_call_test.c diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index d3bb61e0de43..86fb7444dd85 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -1033,6 +1033,16 @@ config PPC_ENABLE_STATIC_CALL is disabled. Otherwise, the generic implementation is better as it is inlined and effectively a direct branch with a hot predictor. +config PPC_STATIC_CALL_KUNIT_TEST + tristate "KUnit tests for PPC64 ELF ABI V2 static calls" + default KUNIT_ALL_TESTS + depends on HAVE_STATIC_CALL && PPC64_ELF_ABI_V2 && KUNIT && m + help + Tests that check the TOC is kept consistent across all combinations + of caller/trampoline/target being kernel/module. Must be built as a + module and loaded at runtime to ensure the module has a different + TOC to the kernel. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index a30d0d0f5499..22c07e3d34df 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -131,6 +131,7 @@ obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o +obj-$(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) += static_call_test.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index d2d9b496381d..18c78f7b044a 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -113,3 +113,114 @@ void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) panic("%s: patching failed %pS at %pS\n", __func__, func, tramp); } EXPORT_SYMBOL_GPL(arch_static_call_transform); + + +#if IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) + +#include + +/* The following are some kernel hooks for testing the static call + * implementation from the static_call_test module. The bulk of the + * assertions are run in that module, except for the TOC checks that + * must be done in the core kernel context. + */ + +/* Reserve these registers for testing (same registers as in static_call_test.c) */ +register void* current_toc asm ("r2"); +register void* module_toc asm ("r14"); +register void* actual_module_toc asm ("r15"); +register void* kernel_toc asm ("r16"); +register void* actual_toc asm ("r17"); + +static void* static_kernel_toc; +static void* static_actual_toc; + +#define restore_toc(test) \ + actual_toc = current_toc; \ + current_toc = kernel_toc + +#define check_toc(test) KUNIT_EXPECT_PTR_EQ(test, kernel_toc, actual_toc) + +#define toc_fixup(test) \ + restore_toc(); \ + check_toc(test) + +#define PROTECTED_SC(test, call) \ +({ \ + long ret; \ + ret = call; \ + toc_fixup(test); \ + ret; \ +}) + +void ppc_sc_kernel_toc_init(void) +{ + static_kernel_toc = kernel_toc; + static_actual_toc = actual_toc; /* save so we can restore when the tests finish */ + + kernel_toc = current_toc; +} + +void ppc_sc_kernel_toc_exit(void) +{ + kernel_toc = static_kernel_toc; + actual_toc = static_actual_toc; +} + +int ppc_sc_kernel_target_1(struct kunit* test) +{ + toc_fixup(test); + return 1; +} + +int ppc_sc_kernel_target_2(struct kunit* test) +{ + toc_fixup(test); + return 2; +} + +DEFINE_STATIC_CALL(ppc_sc_kernel, ppc_sc_kernel_target_1); + +int ppc_sc_kernel_call(struct kunit* test) +{ + return PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); +} + +int ppc_sc_kernel_call_indirect(struct kunit* test, int (*fn)(struct kunit*)) +{ + return PROTECTED_SC(test, fn(test)); +} + +long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i) +{ + toc_fixup(test); + KUNIT_EXPECT_EQ(test, a, b); + KUNIT_EXPECT_EQ(test, a, c); + KUNIT_EXPECT_EQ(test, a, d); + KUNIT_EXPECT_EQ(test, a, e); + KUNIT_EXPECT_EQ(test, a, f); + KUNIT_EXPECT_EQ(test, a, g); + KUNIT_EXPECT_EQ(test, a, h); + KUNIT_EXPECT_EQ(test, a, i); + return ~a; +} + +EXPORT_SYMBOL_GPL(ppc_sc_kernel_toc_init); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_toc_exit); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_1); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_2); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_big); +EXPORT_STATIC_CALL_GPL(ppc_sc_kernel); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call_indirect); + +#endif /* IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) */ diff --git a/arch/powerpc/kernel/static_call_test.c b/arch/powerpc/kernel/static_call_test.c new file mode 100644 index 000000000000..057892d43a0a --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.c @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +/* + * Tests to ensure correctness in a variety of cases for static calls. + * + * The tests focus on ensuring the TOC is kept consistent across the + * module-kernel boundary, as compilers can't see that a trampoline + * defined locally in the kernel might be jumping to a function in a + * module. This makes it important that these tests are compiled as a + * module, so the TOC will be different to the kernel's. + * + * Register variables are used to allow easy position independent + * correction of a TOC before it is used for anything. This means + * a failing test doesn't always crash the whole kernel. The registers + * are initialised on entry and restored on exit of each test using + * KUnit's init and exit hooks. The tests only call internal and + * specially defined kernel functions, so the use of these registers + * will not clobber anything else. + */ + +extern void ppc_sc_kernel_toc_init(void); +extern void ppc_sc_kernel_toc_exit(void); +DECLARE_STATIC_CALL(ppc_sc_kernel, int(struct kunit*)); +extern int ppc_sc_kernel_target_1(struct kunit* test); +extern int ppc_sc_kernel_target_2(struct kunit* test); +extern long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i); +extern int ppc_sc_kernel_call(struct kunit* test); +extern int ppc_sc_kernel_call_indirect(struct kunit* test, int(*fn)(struct kunit*)); + +/* Registers we reserve for use while testing */ +register void* current_toc asm ("r2"); +register void* module_toc asm ("r14"); +register void* actual_toc asm ("r15"); +register void* kernel_toc asm ("r16"); +register void* actual_kernel_toc asm ("r17"); + +/* To hold a copy of the old register values while we test */ +static void* static_module_toc; +static void* static_actual_toc; + +#define restore_toc(test) \ + actual_toc = current_toc; \ + current_toc = module_toc + +#define check_toc(test) KUNIT_EXPECT_PTR_EQ(test, module_toc, actual_toc) + +/* Corrects, then asserts the original TOC was valid */ +#define toc_fixup(test) \ + restore_toc(); \ + check_toc(test) + +/* Wrapper around a static call to verify and correct the TOC + * before running further code that might depend on it's value. + */ +#define PROTECTED_SC(test, call) \ +({ \ + long ret; \ + ret = call; \ + toc_fixup(test); \ + ret; \ +}) + +static int module_target_11(struct kunit *test) +{ + toc_fixup(test); + return 11; +} + +static int module_target_12(struct kunit *test) +{ + toc_fixup(test); + return 12; +} + +DEFINE_STATIC_CALL(module_sc, module_target_11); + +DEFINE_STATIC_CALL_RET0(module_sc_ret0, long(void)); +DEFINE_STATIC_CALL_NULL(module_sc_null, long(long)); + +static long add_one(long *val) +{ + return (*val)++; +} + +static void null_function_test(struct kunit *test) +{ + long val = 0; + + /* Check argument unconditionally evaluated */ + static_call_cond(module_sc_null)(add_one(&val)); + KUNIT_ASSERT_EQ(test, 1, val); +} + +static void return_zero_test(struct kunit *test) +{ + long ret; + + ret = PROTECTED_SC(test, static_call(module_sc_ret0)()); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(ppc_sc_kernel, (void*)__static_call_return0); + ret = PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(module_sc, (void*)__static_call_return0); + ret = PROTECTED_SC(test, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); +} + +static void kernel_kernel_kernel_test(struct kunit *test) +{ + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call(test)); +} + +static void kernel_kernel_module_test(struct kunit *test) +{ + static_call_update(ppc_sc_kernel, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call(test)); +} + +static void kernel_module_kernel_test(struct kunit *test) +{ + static_call_update(module_sc, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); +} + +static void kernel_module_module_test(struct kunit *test) +{ + static_call_update(module_sc, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); +} + +static void module_kernel_kernel_test(struct kunit *test) +{ + long ret; + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); +} + +static void module_kernel_module_test(struct kunit *test) +{ + long ret; + + static_call_update(ppc_sc_kernel, module_target_11); + ret = PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(ppc_sc_kernel, module_target_12); + ret = PROTECTED_SC(test, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); +} + +static void module_module_kernel_test(struct kunit *test) +{ + long ret; + + static_call_update(module_sc, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); +} + +static void module_module_module_test(struct kunit *test) +{ + long ret; + + static_call_update(module_sc, module_target_11); + ret = PROTECTED_SC(test, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(module_sc, module_target_12); + ret = PROTECTED_SC(test, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); +} + +DEFINE_STATIC_CALL(module_sc_stack_params, ppc_sc_kernel_target_big); + +static void stack_parameters_test(struct kunit *test) +{ + long m = 0x1234567887654321; + long ret = PROTECTED_SC(test, static_call(module_sc_stack_params)(test, m, m, m, m, m, m, m, m, m)); + KUNIT_ASSERT_EQ(test, ~m, ret); +} + +static struct kunit_case static_call_test_cases[] = { + KUNIT_CASE(null_function_test), + KUNIT_CASE(return_zero_test), + KUNIT_CASE(stack_parameters_test), + KUNIT_CASE(kernel_kernel_kernel_test), + KUNIT_CASE(kernel_kernel_module_test), + KUNIT_CASE(kernel_module_kernel_test), + KUNIT_CASE(kernel_module_module_test), + KUNIT_CASE(module_kernel_kernel_test), + KUNIT_CASE(module_kernel_module_test), + KUNIT_CASE(module_module_kernel_test), + KUNIT_CASE(module_module_module_test), + {} +}; + +static int ppc_static_call_test_init(struct kunit *test) +{ + static_module_toc = module_toc; + static_actual_toc = actual_toc; + module_toc = current_toc; + + ppc_sc_kernel_toc_init(); + + return 0; +} + +static void ppc_static_call_test_exit(struct kunit *test) +{ + module_toc = static_module_toc; + actual_toc = static_actual_toc; + + ppc_sc_kernel_toc_exit(); +} + +static struct kunit_suite ppc_static_call_test_suite = { + .name = "ppc-static-call", + .test_cases = static_call_test_cases, + .init = ppc_static_call_test_init, + .exit = ppc_static_call_test_exit, +}; +kunit_test_suite(ppc_static_call_test_suite); + +MODULE_AUTHOR("Benjamin Gray "); +MODULE_LICENSE("GPL");