From patchwork Sat Nov 28 14:40:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407529 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvTD47NRz9s1l for ; Sun, 29 Nov 2020 01:46:40 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=WQ/EdZrx; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvTD3pqdzF0Ys for ; Sun, 29 Nov 2020 01:46:40 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=WQ/EdZrx; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMF2zVlzDsLx for ; Sun, 29 Nov 2020 01:41:28 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id l17so6634604pgk.1 for ; Sat, 28 Nov 2020 06:41:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JagQ1b/sT701EO4aUNeBmHErTTB9I9jvINIF0fiCY+Y=; b=WQ/EdZrxRFJs7OLucikhJCSmJxlGSayPYK3p2LU9UtHjrqFOjHydzcCrzqDxw6dHc0 VbWwxu1/N+/vG/gE/Edr60pkKZi7UkWB9XpltUQn3Ts/fzp+83sXmuC8h2xXR1LTHNDp CFbxtTZBnVEiLK18yMqwqvGNX/wxnStWob5o1erfAidh3INXgG/7/59wpSQQwaPmhQ2s VQTDBxHR3CxKz2wEtygVyl43WvFLpXThkqp5D1hZCszqABDKN369E/8rrNxpsxLxLNw3 CDf0R1EIQ2e1F5FEqLJ6YVrfPpVp4SWON2h9tje7eot7gIwNNPj6cRhg/dMGj8B33i8v e8tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JagQ1b/sT701EO4aUNeBmHErTTB9I9jvINIF0fiCY+Y=; b=byS5L8VM51mKtgfn/oxB+2msuBnc47A7dp4PSK6X676bMPar7idZnCmG8j8iosJIdO e6CEV2pZreadH3iYtU64s5MZPKZXavaymKXpL/INmx7e+m6pmQbRVrlU6XJPpzFhN2Hk Z/3jbep5jgCI/DNKopx+uPZ9b0If8Irmqpo0AZglJgODQxW+Fa8++HYhXtYH0iu9uBe5 ocgH2vFq/bLWuSG8aypgKUj4cTcf0phfOvylFsdxXy2MRzuVL2Tlz+XL6vM1jaZzrkUS 9GOaqAK+IcD8axNN+G4GZVXflWiOxGjmZ/gUktScJCg1+Sa+DX1rS0mWV7It1ocSzZqv VIoQ== X-Gm-Message-State: AOAM531N6xFaUTUiYUJHDUKzqNXY81MNXT62IrvXZg8rr6OirbebYiky kKKfm5PMqDpZs0QfybCo9MLwRU2N3K8= X-Google-Smtp-Source: ABdhPJwyTxsoYwtlnfcyJjoFHrqILFFK//6cH9JnH2qO8cGdMXvM7YbUDCpgA7mV391wdPF1HV3DDg== X-Received: by 2002:a17:90b:1007:: with SMTP id gm7mr17140089pjb.18.1606574483771; Sat, 28 Nov 2020 06:41:23 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:23 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 01/19] powerpc/64s: move the last of the page fault handling logic to C Date: Sun, 29 Nov 2020 00:40:56 +1000 Message-Id: <20201128144114.944000-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The page fault handling still has some complex logic particularly around hash table handling, in asm. Implement this in C instead. Signed-off-by: Nicholas Piggin Reviewed-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 1 + arch/powerpc/include/asm/bug.h | 1 + arch/powerpc/kernel/exceptions-64s.S | 131 +++--------------- arch/powerpc/mm/book3s64/hash_utils.c | 77 ++++++---- arch/powerpc/mm/fault.c | 57 +++++++- 5 files changed, 127 insertions(+), 140 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index 683a9c7d1b03..bc8c91f2d26f 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -453,6 +453,7 @@ static inline unsigned long hpt_hash(unsigned long vpn, #define HPTE_LOCAL_UPDATE 0x1 #define HPTE_NOHPTE_UPDATE 0x2 +int do_hash_fault(struct pt_regs *regs, unsigned long ea, unsigned long dsisr); extern int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, pte_t *ptep, unsigned long trap, unsigned long flags, int ssize, int subpage_prot); diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h index 338f36cd9934..c0e9b7a967a8 100644 --- a/arch/powerpc/include/asm/bug.h +++ b/arch/powerpc/include/asm/bug.h @@ -112,6 +112,7 @@ struct pt_regs; extern int do_page_fault(struct pt_regs *, unsigned long, unsigned long); +int hash__do_page_fault(struct pt_regs *, unsigned long, unsigned long); extern void bad_page_fault(struct pt_regs *, unsigned long, int); extern void _exception(int, struct pt_regs *, int, unsigned long); extern void _exception_pkey(struct pt_regs *, unsigned long, int); diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 4d01f09ecf80..336fa1fa39d1 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -1401,14 +1401,15 @@ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE) * * Handling: * - Hash MMU - * Go to do_hash_page first to see if the HPT can be filled from an entry in - * the Linux page table. Hash faults can hit in kernel mode in a fairly + * Go to do_hash_fault, which attempts to fill the HPT from an entry in the + * Linux page table. Hash faults can hit in kernel mode in a fairly * arbitrary state (e.g., interrupts disabled, locks held) when accessing * "non-bolted" regions, e.g., vmalloc space. However these should always be - * backed by Linux page tables. + * backed by Linux page table entries. * - * If none is found, do a Linux page fault. Linux page faults can happen in - * kernel mode due to user copy operations of course. + * If no entry is found the Linux page fault handler is invoked (by + * do_hash_fault). Linux page faults can happen in kernel mode due to user + * copy operations of course. * * KVM: The KVM HDSI handler may perform a load with MSR[DR]=1 in guest * MMU context, which may cause a DSI in the host, which must go to the @@ -1439,13 +1440,17 @@ EXC_COMMON_BEGIN(data_access_common) GEN_COMMON data_access ld r4,_DAR(r1) ld r5,_DSISR(r1) + addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION - ld r6,_MSR(r1) - li r3,0x300 - b do_hash_page /* Try to handle as hpte fault */ + bl do_hash_fault MMU_FTR_SECTION_ELSE - b handle_page_fault + bl do_page_fault ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) + cmpdi r3,0 + beq+ interrupt_return + /* We need to restore NVGPRS */ + REST_NVGPRS(r1) + b interrupt_return GEN_KVM data_access @@ -1540,13 +1545,17 @@ EXC_COMMON_BEGIN(instruction_access_common) GEN_COMMON instruction_access ld r4,_DAR(r1) ld r5,_DSISR(r1) + addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION - ld r6,_MSR(r1) - li r3,0x400 - b do_hash_page /* Try to handle as hpte fault */ + bl do_hash_fault MMU_FTR_SECTION_ELSE - b handle_page_fault + bl do_page_fault ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) + cmpdi r3,0 + beq+ interrupt_return + /* We need to restore NVGPRS */ + REST_NVGPRS(r1) + b interrupt_return GEN_KVM instruction_access @@ -3202,99 +3211,3 @@ disable_machine_check: RFI_TO_KERNEL 1: mtlr r0 blr - -/* - * Hash table stuff - */ - .balign IFETCH_ALIGN_BYTES -do_hash_page: -#ifdef CONFIG_PPC_BOOK3S_64 - lis r0,(DSISR_BAD_FAULT_64S | DSISR_DABRMATCH | DSISR_KEYFAULT)@h - ori r0,r0,DSISR_BAD_FAULT_64S@l - and. r0,r5,r0 /* weird error? */ - bne- handle_page_fault /* if not, try to insert a HPTE */ - - /* - * If we are in an "NMI" (e.g., an interrupt when soft-disabled), then - * don't call hash_page, just fail the fault. This is required to - * prevent re-entrancy problems in the hash code, namely perf - * interrupts hitting while something holds H_PAGE_BUSY, and taking a - * hash fault. See the comment in hash_preload(). - */ - ld r11, PACA_THREAD_INFO(r13) - lwz r0,TI_PREEMPT(r11) - andis. r0,r0,NMI_MASK@h - bne 77f - - /* - * r3 contains the trap number - * r4 contains the faulting address - * r5 contains dsisr - * r6 msr - * - * at return r3 = 0 for success, 1 for page fault, negative for error - */ - bl __hash_page /* build HPTE if possible */ - cmpdi r3,0 /* see if __hash_page succeeded */ - - /* Success */ - beq interrupt_return /* Return from exception on success */ - - /* Error */ - blt- 13f - - /* Reload DAR/DSISR into r4/r5 for the DABR check below */ - ld r4,_DAR(r1) - ld r5,_DSISR(r1) -#endif /* CONFIG_PPC_BOOK3S_64 */ - -/* Here we have a page fault that hash_page can't handle. */ -handle_page_fault: -11: andis. r0,r5,DSISR_DABRMATCH@h - bne- handle_dabr_fault - addi r3,r1,STACK_FRAME_OVERHEAD - bl do_page_fault - cmpdi r3,0 - beq+ interrupt_return - mr r5,r3 - addi r3,r1,STACK_FRAME_OVERHEAD - ld r4,_DAR(r1) - bl bad_page_fault - b interrupt_return - -/* We have a data breakpoint exception - handle it */ -handle_dabr_fault: - ld r4,_DAR(r1) - ld r5,_DSISR(r1) - addi r3,r1,STACK_FRAME_OVERHEAD - bl do_break - /* - * do_break() may have changed the NV GPRS while handling a breakpoint. - * If so, we need to restore them with their updated values. - */ - REST_NVGPRS(r1) - b interrupt_return - - -#ifdef CONFIG_PPC_BOOK3S_64 -/* We have a page fault that hash_page could handle but HV refused - * the PTE insertion - */ -13: mr r5,r3 - addi r3,r1,STACK_FRAME_OVERHEAD - ld r4,_DAR(r1) - bl low_hash_fault - b interrupt_return -#endif - -/* - * We come here as a result of a DSI at a point where we don't want - * to call hash_page, such as when we are accessing memory (possibly - * user memory) inside a PMU interrupt that occurred while interrupts - * were soft-disabled. We want to invoke the exception handler for - * the access, or panic if there isn't a handler. - */ -77: addi r3,r1,STACK_FRAME_OVERHEAD - li r5,SIGSEGV - bl bad_page_fault - b interrupt_return diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 24702c0a92e0..bfa1b1966218 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1510,16 +1510,40 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap, } EXPORT_SYMBOL_GPL(hash_page); -int __hash_page(unsigned long trap, unsigned long ea, unsigned long dsisr, - unsigned long msr) +int do_hash_fault(struct pt_regs *regs, unsigned long ea, unsigned long dsisr) { unsigned long access = _PAGE_PRESENT | _PAGE_READ; unsigned long flags = 0; - struct mm_struct *mm = current->mm; - unsigned int region_id = get_region_id(ea); + struct mm_struct *mm; + unsigned int region_id; + int err; + + if (unlikely(dsisr & (DSISR_BAD_FAULT_64S | DSISR_DABRMATCH | DSISR_KEYFAULT))) + goto _do_page_fault; + + /* + * If we are in an "NMI" (e.g., an interrupt when soft-disabled), then + * don't call hash_page, just fail the fault. This is required to + * prevent re-entrancy problems in the hash code, namely perf + * interrupts hitting while something holds H_PAGE_BUSY, and taking a + * hash fault. See the comment in hash_preload(). + * + * We come here as a result of a DSI at a point where we don't want + * to call hash_page, such as when we are accessing memory (possibly + * user memory) inside a PMU interrupt that occurred while interrupts + * were soft-disabled. We want to invoke the exception handler for + * the access, or panic if there isn't a handler. + */ + if (unlikely(in_nmi())) { + bad_page_fault(regs, ea, SIGSEGV); + return 0; + } + region_id = get_region_id(ea); if ((region_id == VMALLOC_REGION_ID) || (region_id == IO_REGION_ID)) mm = &init_mm; + else + mm = current->mm; if (dsisr & DSISR_NOHPTE) flags |= HPTE_NOHPTE_UPDATE; @@ -1535,13 +1559,31 @@ int __hash_page(unsigned long trap, unsigned long ea, unsigned long dsisr, * 2) user space access kernel space. */ access |= _PAGE_PRIVILEGED; - if ((msr & MSR_PR) || (region_id == USER_REGION_ID)) + if (user_mode(regs) || (region_id == USER_REGION_ID)) access &= ~_PAGE_PRIVILEGED; - if (trap == 0x400) + if (regs->trap == 0x400) access |= _PAGE_EXEC; - return hash_page_mm(mm, ea, access, trap, flags); + err = hash_page_mm(mm, ea, access, regs->trap, flags); + if (unlikely(err < 0)) { + // failed to instert a hash PTE due to an hypervisor error + if (user_mode(regs)) { + if (IS_ENABLED(CONFIG_PPC_SUBPAGE_PROT) && err == -2) + _exception(SIGSEGV, regs, SEGV_ACCERR, ea); + else + _exception(SIGBUS, regs, BUS_ADRERR, ea); + } else { + bad_page_fault(regs, ea, SIGBUS); + } + err = 0; + + } else if (err) { +_do_page_fault: + err = hash__do_page_fault(regs, ea, dsisr); + } + + return err; } #ifdef CONFIG_PPC_MM_SLICES @@ -1841,27 +1883,6 @@ void flush_hash_range(unsigned long number, int local) } } -/* - * low_hash_fault is called when we the low level hash code failed - * to instert a PTE due to an hypervisor error - */ -void low_hash_fault(struct pt_regs *regs, unsigned long address, int rc) -{ - enum ctx_state prev_state = exception_enter(); - - if (user_mode(regs)) { -#ifdef CONFIG_PPC_SUBPAGE_PROT - if (rc == -2) - _exception(SIGSEGV, regs, SEGV_ACCERR, address); - else -#endif - _exception(SIGBUS, regs, BUS_ADRERR, address); - } else - bad_page_fault(regs, address, SIGBUS); - - exception_exit(prev_state); -} - long hpte_insert_repeating(unsigned long hash, unsigned long vpn, unsigned long pa, unsigned long rflags, unsigned long vflags, int psize, int ssize) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 0add963a849b..e65a49f246ef 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -370,7 +370,9 @@ static void sanity_check_fault(bool is_write, bool is_user, #define page_fault_is_write(__err) ((__err) & DSISR_ISSTORE) #if defined(CONFIG_PPC_8xx) #define page_fault_is_bad(__err) ((__err) & DSISR_NOEXEC_OR_G) -#elif defined(CONFIG_PPC64) +#elif defined(CONFIG_PPC_BOOK3S_64) +#define page_fault_is_bad(__err) ((__err) & (DSISR_BAD_FAULT_64S | DSISR_DABRMATCH)) +#elif defined(CONFIG_PPC_BOOK3E_64) #define page_fault_is_bad(__err) ((__err) & DSISR_BAD_FAULT_64S) #else #define page_fault_is_bad(__err) ((__err) & DSISR_BAD_FAULT_32S) @@ -406,6 +408,9 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, return 0; if (unlikely(page_fault_is_bad(error_code))) { + if (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && (error_code & DSISR_DABRMATCH)) + return -1; + if (is_user) { _exception(SIGBUS, regs, BUS_OBJERR, address); return 0; @@ -548,12 +553,58 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, unsigned long error_code) { enum ctx_state prev_state = exception_enter(); - int rc = __do_page_fault(regs, address, error_code); + int err; + + err = __do_page_fault(regs, address, error_code); + +#ifdef CONFIG_PPC_BOOK3S_64 + /* 32 and 64e handle errors in their asm code */ + if (unlikely(err)) { + if (err > 0) { + bad_page_fault(regs, address, err); + err = 0; + } else { + /* + * do_break() may change NV GPRS while handling the + * breakpoint. Return -ve to caller to do that. + */ + do_break(regs, address, error_code); + } + } +#endif + exception_exit(prev_state); - return rc; + + return err; } NOKPROBE_SYMBOL(do_page_fault); +#ifdef CONFIG_PPC_BOOK3S_64 +/* Same as do_page_fault but interrupt entry has already run in do_hash_fault */ +int hash__do_page_fault(struct pt_regs *regs, unsigned long address, + unsigned long error_code) +{ + int err; + + err = __do_page_fault(regs, address, error_code); + if (unlikely(err)) { + if (err > 0) { + bad_page_fault(regs, address, err); + err = 0; + } else { + /* + * do_break() may change NV GPRS while handling the + * breakpoint. Return -ve to caller to do that. + */ + do_break(regs, address, error_code); + } + } + + return err; +} +NOKPROBE_SYMBOL(hash__do_page_fault); +#endif + /* * bad_page_fault is called when we have a bad access from the kernel. * It is called from the DSI and ISI handlers in head.S and from some From patchwork Sat Nov 28 14:40:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407528 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvR96pjYz9s1l for ; Sun, 29 Nov 2020 01:44:53 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Lk91B/hX; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvR96JtvzDrBk for ; Sun, 29 Nov 2020 01:44:53 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52e; helo=mail-pg1-x52e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=Lk91B/hX; dkim-atps=neutral Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMF1PC9zDsGv for ; Sun, 29 Nov 2020 01:41:28 +1100 (AEDT) Received: by mail-pg1-x52e.google.com with SMTP id s63so6606418pgc.8 for ; Sat, 28 Nov 2020 06:41:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BXv9OC+q1DQSg1qFGpa+RnyshL9wJAm6So0bdq0nQLQ=; b=Lk91B/hX4wj325SxwZqkCmvfywuklR/5Rl9i3y9P2vNaHSkJgW3LlKbRlgStpwUDZG dfa8ZWo+S52Exc/1E0hNsT0eqChY8/nmmGF/M2Lmz5Wa8IMt/W0rmOQg5DyHCTRvIhV+ U8TQ3AEZDO2kjE7bQ+EhFpbu/7L0yJXBAvrkCaAj4122uSlSmUEOf1bKOxxfWJQ0yw5Y gTE9ilQ5NMKGYsnhFDvbb2RIadBOrJGcSc7QDcMByOFYJ/W5miITTItVLyP6AbRAIqNK ML9/tZqiglYP3/8sfyH4AkpfWiEbnr38BKYeqNtXohKlV/7HnNCxJTBfeu43wOnRDXO6 JBTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BXv9OC+q1DQSg1qFGpa+RnyshL9wJAm6So0bdq0nQLQ=; b=WxFCzbE2+xcf0PJnS5MUh+kzdGa/TIsyx1KfOMKgT3CGEfCWPFh5t/0HROU+C0kVGJ 2SyUxq6vJk84/Eq8+59+9ooFiXkOdUtmFQvsgoUhcWkz9dJiJxnD/fH86g2GC/56QkBh cOdHRHb/VuqrSlRGBAhRZmABE4d2RePwEU0u8r9Rs2LK9FBB+M/VfdmKWdTrIvD7T1TY O4uY0oBd/FRjtmWSTKBr7H2zoP3YwiMq34oJ4a3zTRw+SA4JNKCg/HQ3H8IYCBdVqZhE SmZSVfonHFGOhOOVP/Djr/2XphP4v5oL9bZCqKaUsiMKa3Z1DERPnB1ljgfIEjQeXM4V 5Yjw== X-Gm-Message-State: AOAM532r5b5HNvOnJ1jQUaSgC0C6c/bogYLVYvGWjUyxX13ys5Tj2Bwf fpGTAOeJ5WQc/aG4SzIzhVqXECiO0jg= X-Google-Smtp-Source: ABdhPJz9h1Ti9vI5OYCK3dUyWA/00QWENAQYhjUCBmoJYRtEIvW5o0x+6b4SfYggZbN5zPbVmxctlA== X-Received: by 2002:a17:90b:4d0e:: with SMTP id mw14mr16240889pjb.92.1606574486086; Sat, 28 Nov 2020 06:41:26 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:25 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 02/19] powerpc: remove arguments from fault handler functions Date: Sun, 29 Nov 2020 00:40:57 +1000 Message-Id: <20201128144114.944000-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Make mm fault handlers all just take the pt_regs * argument and load DAR/DSISR from that. Make those that return a value return long. This is done to make the function signatures match other handlers, which will help with a future patch to add wrappers. Explicit arguments could be added for performance but that would require more wrapper macro variants. Signed-off-by: Nicholas Piggin Reviewed-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/asm-prototypes.h | 4 ++-- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 2 +- arch/powerpc/include/asm/bug.h | 4 ++-- arch/powerpc/kernel/exceptions-64e.S | 2 -- arch/powerpc/kernel/exceptions-64s.S | 14 ++------------ arch/powerpc/kernel/head_40x.S | 10 +++++----- arch/powerpc/kernel/head_8xx.S | 6 +++--- arch/powerpc/kernel/head_book3s_32.S | 6 ++---- arch/powerpc/kernel/head_booke.h | 4 +--- arch/powerpc/mm/book3s64/hash_utils.c | 8 +++++--- arch/powerpc/mm/book3s64/slb.c | 11 +++++++---- arch/powerpc/mm/fault.c | 16 +++++++++------- 12 files changed, 39 insertions(+), 48 deletions(-) diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h index d0b832cbbec8..22c9d08fa3a4 100644 --- a/arch/powerpc/include/asm/asm-prototypes.h +++ b/arch/powerpc/include/asm/asm-prototypes.h @@ -82,8 +82,8 @@ void kernel_bad_stack(struct pt_regs *regs); void system_reset_exception(struct pt_regs *regs); void machine_check_exception(struct pt_regs *regs); void emulation_assist_interrupt(struct pt_regs *regs); -long do_slb_fault(struct pt_regs *regs, unsigned long ea); -void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err); +long do_slb_fault(struct pt_regs *regs); +void do_bad_slb_fault(struct pt_regs *regs); /* signals, syscalls and interrupts */ long sys_swapcontext(struct ucontext __user *old_ctx, diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index bc8c91f2d26f..e843d0b193d3 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -453,7 +453,7 @@ static inline unsigned long hpt_hash(unsigned long vpn, #define HPTE_LOCAL_UPDATE 0x1 #define HPTE_NOHPTE_UPDATE 0x2 -int do_hash_fault(struct pt_regs *regs, unsigned long ea, unsigned long dsisr); +long do_hash_fault(struct pt_regs *regs); extern int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, pte_t *ptep, unsigned long trap, unsigned long flags, int ssize, int subpage_prot); diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h index c0e9b7a967a8..897bad6b6bbb 100644 --- a/arch/powerpc/include/asm/bug.h +++ b/arch/powerpc/include/asm/bug.h @@ -111,8 +111,8 @@ #ifndef __ASSEMBLY__ struct pt_regs; -extern int do_page_fault(struct pt_regs *, unsigned long, unsigned long); -int hash__do_page_fault(struct pt_regs *, unsigned long, unsigned long); +long do_page_fault(struct pt_regs *); +long hash__do_page_fault(struct pt_regs *); extern void bad_page_fault(struct pt_regs *, unsigned long, int); extern void _exception(int, struct pt_regs *, int, unsigned long); extern void _exception_pkey(struct pt_regs *, unsigned long, int); diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S index f579ce46eef2..25fa7d5a643c 100644 --- a/arch/powerpc/kernel/exceptions-64e.S +++ b/arch/powerpc/kernel/exceptions-64e.S @@ -1011,8 +1011,6 @@ storage_fault_common: std r14,_DAR(r1) std r15,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD - mr r4,r14 - mr r5,r15 ld r14,PACA_EXGEN+EX_R14(r13) ld r15,PACA_EXGEN+EX_R15(r13) bl do_page_fault diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 336fa1fa39d1..690058043b17 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -1438,8 +1438,6 @@ EXC_VIRT_BEGIN(data_access, 0x4300, 0x80) EXC_VIRT_END(data_access, 0x4300, 0x80) EXC_COMMON_BEGIN(data_access_common) GEN_COMMON data_access - ld r4,_DAR(r1) - ld r5,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION bl do_hash_fault @@ -1492,10 +1490,9 @@ EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80) EXC_VIRT_END(data_access_slb, 0x4380, 0x80) EXC_COMMON_BEGIN(data_access_slb_common) GEN_COMMON data_access_slb - ld r4,_DAR(r1) - addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION /* HPT case, do SLB fault */ + addi r3,r1,STACK_FRAME_OVERHEAD bl do_slb_fault cmpdi r3,0 bne- 1f @@ -1507,8 +1504,6 @@ MMU_FTR_SECTION_ELSE ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) std r3,RESULT(r1) RECONCILE_IRQ_STATE(r10, r11) - ld r4,_DAR(r1) - ld r5,RESULT(r1) addi r3,r1,STACK_FRAME_OVERHEAD bl do_bad_slb_fault b interrupt_return @@ -1543,8 +1538,6 @@ EXC_VIRT_BEGIN(instruction_access, 0x4400, 0x80) EXC_VIRT_END(instruction_access, 0x4400, 0x80) EXC_COMMON_BEGIN(instruction_access_common) GEN_COMMON instruction_access - ld r4,_DAR(r1) - ld r5,_DSISR(r1) addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION bl do_hash_fault @@ -1588,10 +1581,9 @@ EXC_VIRT_BEGIN(instruction_access_slb, 0x4480, 0x80) EXC_VIRT_END(instruction_access_slb, 0x4480, 0x80) EXC_COMMON_BEGIN(instruction_access_slb_common) GEN_COMMON instruction_access_slb - ld r4,_DAR(r1) - addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION /* HPT case, do SLB fault */ + addi r3,r1,STACK_FRAME_OVERHEAD bl do_slb_fault cmpdi r3,0 bne- 1f @@ -1603,8 +1595,6 @@ MMU_FTR_SECTION_ELSE ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) std r3,RESULT(r1) RECONCILE_IRQ_STATE(r10, r11) - ld r4,_DAR(r1) - ld r5,RESULT(r1) addi r3,r1,STACK_FRAME_OVERHEAD bl do_bad_slb_fault b interrupt_return diff --git a/arch/powerpc/kernel/head_40x.S b/arch/powerpc/kernel/head_40x.S index a1ae00689e0f..3c5577ac4dc8 100644 --- a/arch/powerpc/kernel/head_40x.S +++ b/arch/powerpc/kernel/head_40x.S @@ -179,9 +179,9 @@ _ENTRY(saved_ksp_limit) */ START_EXCEPTION(0x0300, DataStorage) EXCEPTION_PROLOG - mfspr r5, SPRN_ESR /* Grab the ESR, save it, pass arg3 */ + mfspr r5, SPRN_ESR /* Grab the ESR, save it */ stw r5, _ESR(r11) - mfspr r4, SPRN_DEAR /* Grab the DEAR, save it, pass arg2 */ + mfspr r4, SPRN_DEAR /* Grab the DEAR, save it */ stw r4, _DEAR(r11) EXC_XFER_LITE(0x300, handle_page_fault) @@ -191,9 +191,9 @@ _ENTRY(saved_ksp_limit) */ START_EXCEPTION(0x0400, InstructionAccess) EXCEPTION_PROLOG - mr r4,r12 /* Pass SRR0 as arg2 */ - stw r4, _DEAR(r11) - li r5,0 /* Pass zero as arg3 */ + li r5,0 + stw r5, _ESR(r11) /* Zero ESR */ + stw r12, _DEAR(r11) /* SRR0 as DEAR */ EXC_XFER_LITE(0x400, handle_page_fault) /* 0x0500 - External Interrupt Exception */ diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index ee0bfebc375f..8acd365a2be6 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -324,14 +324,14 @@ DataStoreTLBMiss: . = 0x1300 InstructionTLBError: EXCEPTION_PROLOG - mr r4,r12 andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */ andis. r10,r9,SRR1_ISI_NOPT@h beq+ .Litlbie - tlbie r4 + tlbie r12 /* 0x400 is InstructionAccess exception, needed by bad_page_fault() */ .Litlbie: - stw r4, _DAR(r11) + stw r12, _DAR(r11) + stw r5, _DSISR(r11) EXC_XFER_LITE(0x400, handle_page_fault) /* This is the data TLB error on the MPC8xx. This could be due to diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S index a0dda2a1f2df..7addf67832f9 100644 --- a/arch/powerpc/kernel/head_book3s_32.S +++ b/arch/powerpc/kernel/head_book3s_32.S @@ -370,9 +370,9 @@ BEGIN_MMU_FTR_SECTION bl hash_page END_MMU_FTR_SECTION_IFSET(MMU_FTR_HPTE_TABLE) #endif /* CONFIG_VMAP_STACK */ -1: mr r4,r12 andis. r5,r9,DSISR_SRR1_MATCH_32S@h /* Filter relevant SRR1 bits */ - stw r4, _DAR(r11) + stw r5, _DSISR(r11) + stw r12, _DAR(r11) EXC_XFER_LITE(0x400, handle_page_fault) /* External interrupt */ @@ -687,8 +687,6 @@ handle_page_fault_tramp_1: #ifdef CONFIG_VMAP_STACK EXCEPTION_PROLOG_2 handle_dar_dsisr=1 #endif - lwz r4, _DAR(r11) - lwz r5, _DSISR(r11) /* fall through */ handle_page_fault_tramp_2: EXC_XFER_LITE(0x300, handle_page_fault) diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h index 71c359d438b5..1da0c1d1b0a1 100644 --- a/arch/powerpc/kernel/head_booke.h +++ b/arch/powerpc/kernel/head_booke.h @@ -477,9 +477,7 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV) NORMAL_EXCEPTION_PROLOG(INST_STORAGE); \ mfspr r5,SPRN_ESR; /* Grab the ESR and save it */ \ stw r5,_ESR(r11); \ - mr r4,r12; /* Pass SRR0 as arg2 */ \ - stw r4, _DEAR(r11); \ - li r5,0; /* Pass zero as arg3 */ \ + stw r12, _DEAR(r11); /* Pass SRR0 as arg2 */ \ EXC_XFER_LITE(0x0400, handle_page_fault) #define ALIGNMENT_EXCEPTION \ diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index bfa1b1966218..0f0bd4af4b2d 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1510,13 +1510,15 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap, } EXPORT_SYMBOL_GPL(hash_page); -int do_hash_fault(struct pt_regs *regs, unsigned long ea, unsigned long dsisr) +long do_hash_fault(struct pt_regs *regs) { + unsigned long ea = regs->dar; + unsigned long dsisr = regs->dsisr; unsigned long access = _PAGE_PRESENT | _PAGE_READ; unsigned long flags = 0; struct mm_struct *mm; unsigned int region_id; - int err; + long err; if (unlikely(dsisr & (DSISR_BAD_FAULT_64S | DSISR_DABRMATCH | DSISR_KEYFAULT))) goto _do_page_fault; @@ -1580,7 +1582,7 @@ int do_hash_fault(struct pt_regs *regs, unsigned long ea, unsigned long dsisr) } else if (err) { _do_page_fault: - err = hash__do_page_fault(regs, ea, dsisr); + err = hash__do_page_fault(regs); } return err; diff --git a/arch/powerpc/mm/book3s64/slb.c b/arch/powerpc/mm/book3s64/slb.c index c30fcbfa0e32..cc34d50874c1 100644 --- a/arch/powerpc/mm/book3s64/slb.c +++ b/arch/powerpc/mm/book3s64/slb.c @@ -837,8 +837,9 @@ static long slb_allocate_user(struct mm_struct *mm, unsigned long ea) return slb_insert_entry(ea, context, flags, ssize, false); } -long do_slb_fault(struct pt_regs *regs, unsigned long ea) +long do_slb_fault(struct pt_regs *regs) { + unsigned long ea = regs->dar; unsigned long id = get_region_id(ea); /* IRQs are not reconciled here, so can't check irqs_disabled */ @@ -889,13 +890,15 @@ long do_slb_fault(struct pt_regs *regs, unsigned long ea) } } -void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err) +void do_bad_slb_fault(struct pt_regs *regs) { + int err = regs->result; + if (err == -EFAULT) { if (user_mode(regs)) - _exception(SIGSEGV, regs, SEGV_BNDERR, ea); + _exception(SIGSEGV, regs, SEGV_BNDERR, regs->dar); else - bad_page_fault(regs, ea, SIGSEGV); + bad_page_fault(regs, regs->dar, SIGSEGV); } else if (err == -EINVAL) { unrecoverable_exception(regs); } else { diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index e65a49f246ef..390a296b16a3 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -549,11 +549,12 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, } NOKPROBE_SYMBOL(__do_page_fault); -int do_page_fault(struct pt_regs *regs, unsigned long address, - unsigned long error_code) +long do_page_fault(struct pt_regs *regs) { enum ctx_state prev_state = exception_enter(); - int err; + unsigned long address = regs->dar; + unsigned long error_code = regs->dsisr; + long err; err = __do_page_fault(regs, address, error_code); @@ -580,11 +581,12 @@ int do_page_fault(struct pt_regs *regs, unsigned long address, NOKPROBE_SYMBOL(do_page_fault); #ifdef CONFIG_PPC_BOOK3S_64 -/* Same as do_page_fault but interrupt entry has already run in do_hash_fault */ -int hash__do_page_fault(struct pt_regs *regs, unsigned long address, - unsigned long error_code) +/* Same as do_page_fault but no interrupt entry */ +long hash__do_page_fault(struct pt_regs *regs) { - int err; + unsigned long address = regs->dar; + unsigned long error_code = regs->dsisr; + long err; err = __do_page_fault(regs, address, error_code); if (unlikely(err)) { From patchwork Sat Nov 28 14:40:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407530 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvW20K15z9s1l for ; Sun, 29 Nov 2020 01:48:14 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=se+sZFpz; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvW16vbmzF0fh for ; Sun, 29 Nov 2020 01:48:13 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=se+sZFpz; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMH2NNRzDsGv for ; Sun, 29 Nov 2020 01:41:31 +1100 (AEDT) Received: by mail-pf1-x442.google.com with SMTP id w187so6983417pfd.5 for ; Sat, 28 Nov 2020 06:41:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rpMO52XtWIrfmYYj7DbKRpCl2ev4UQjiOUNRNC3Wqk0=; b=se+sZFpzaNieXexwJvspBvA0uGoWR2kGDwr1gb7P/oiOjFTzAyKZ6/+5qcCAG04xhz G0m3SqmcMhdrtnFMQF2iDeQLMH66+CmzFBjw1LcFaDE/httJj4BUnVxvEbPQjp3Ek8Lq PKBJL5QdaYJ5bm5/8kqWX1recDdiq0nr2zByrrTOuAXCcEWN3jXeEcrgQ1nauFQ+2CsR paPDiZvitw+OfviyIXBuhpnEOb3Q65pM/Fp2+n2txxvb6y4YwBfpEL2I8q8yczm5mHPK xoXEJV8FZhwhvtav/37lhgpEI26PTt4FaY2A+CkOvQWibPnZsBddA4/86tal5DHYi7H9 JqXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rpMO52XtWIrfmYYj7DbKRpCl2ev4UQjiOUNRNC3Wqk0=; b=WN18lFkXNXbVUHCjNj+ssEGGNH/oYdv1y75lsJc7+c4t6IPDnfqvaW8d2sqF7u9NxS 9KH+arFrPuYlEo5lpQxeBF90sIaFW538tr/6vQwGRvQUqpFSZZU9z4H/1G5Xe5ousc2s 8S5iIBV9mAw1wzb3WAAUcKNX1fvslv9NPAN0Lv/VbXIoJvsQL7MG0o1LuMD1P3iE2UY+ DF0XnDYWb9L/0rrwjAt8/U+08fp/v/h9sKbYUMaPmmnNaMQ2ycvT2iJikabN6XpPr0WL dG7Usuaye7q8TLlKfw/b9ywyhEnfGlauengMj3q0/lSA1dkbW9mT7waYNhqyZNs/7jM7 kNHQ== X-Gm-Message-State: AOAM532DvubNunydyXImYB9J4f63lD/th4jbdkpZDpC1uyZHTR9eUUzk /qV5JqmfM4y5n3FP9lVmtQ9lF1vl1aE= X-Google-Smtp-Source: ABdhPJyyehhKpSV0/ZaFk4dAXoRA2GTAZbPoxR+md/d3SffPWQpsoji7CBODe7ORTZPhPKxjndpeEA== X-Received: by 2002:aa7:9f9a:0:b029:18b:a203:3146 with SMTP id z26-20020aa79f9a0000b029018ba2033146mr11388518pfr.36.1606574488352; Sat, 28 Nov 2020 06:41:28 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:28 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 03/19] powerpc: bad_page_fault, do_break get registers from regs Date: Sun, 29 Nov 2020 00:40:58 +1000 Message-Id: <20201128144114.944000-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Similar to the previous patch this makes interrupt handler function types more regular so they can be wrapped with the next patch. bad_page_fault and do_break are not performance critical. [32s DABR code from Christophe Leroy ] Signed-off-by: Nicholas Piggin Reviewed-by: Aneesh Kumar K.V --- arch/powerpc/include/asm/bug.h | 2 +- arch/powerpc/include/asm/debug.h | 3 +-- arch/powerpc/kernel/entry_32.S | 18 +----------------- arch/powerpc/kernel/exceptions-64e.S | 3 +-- arch/powerpc/kernel/exceptions-64s.S | 3 +-- arch/powerpc/kernel/head_8xx.S | 5 ++--- arch/powerpc/kernel/head_book3s_32.S | 3 +++ arch/powerpc/kernel/process.c | 7 +++---- arch/powerpc/kernel/traps.c | 2 +- arch/powerpc/mm/book3s64/hash_utils.c | 4 ++-- arch/powerpc/mm/book3s64/slb.c | 2 +- arch/powerpc/mm/fault.c | 10 +++++----- arch/powerpc/platforms/8xx/machine_check.c | 2 +- 13 files changed, 23 insertions(+), 41 deletions(-) diff --git a/arch/powerpc/include/asm/bug.h b/arch/powerpc/include/asm/bug.h index 897bad6b6bbb..49162faba33f 100644 --- a/arch/powerpc/include/asm/bug.h +++ b/arch/powerpc/include/asm/bug.h @@ -113,7 +113,7 @@ struct pt_regs; long do_page_fault(struct pt_regs *); long hash__do_page_fault(struct pt_regs *); -extern void bad_page_fault(struct pt_regs *, unsigned long, int); +void bad_page_fault(struct pt_regs *, int); extern void _exception(int, struct pt_regs *, int, unsigned long); extern void _exception_pkey(struct pt_regs *, unsigned long, int); extern void die(const char *, struct pt_regs *, long); diff --git a/arch/powerpc/include/asm/debug.h b/arch/powerpc/include/asm/debug.h index ec57daf87f40..0550eceab3ca 100644 --- a/arch/powerpc/include/asm/debug.h +++ b/arch/powerpc/include/asm/debug.h @@ -52,8 +52,7 @@ extern void do_send_trap(struct pt_regs *regs, unsigned long address, unsigned long error_code, int brkpt); #else -extern void do_break(struct pt_regs *regs, unsigned long address, - unsigned long error_code); +void do_break(struct pt_regs *regs); #endif #endif /* _ASM_POWERPC_DEBUG_H */ diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S index 8cdc8bcde703..57b8e95ea2a0 100644 --- a/arch/powerpc/kernel/entry_32.S +++ b/arch/powerpc/kernel/entry_32.S @@ -657,10 +657,6 @@ ppc_swapcontext: .globl handle_page_fault handle_page_fault: addi r3,r1,STACK_FRAME_OVERHEAD -#ifdef CONFIG_PPC_BOOK3S_32 - andis. r0,r5,DSISR_DABRMATCH@h - bne- handle_dabr_fault -#endif bl do_page_fault cmpwi r3,0 beq+ ret_from_except @@ -668,23 +664,11 @@ handle_page_fault: lwz r0,_TRAP(r1) clrrwi r0,r0,1 stw r0,_TRAP(r1) - mr r5,r3 + mr r4,r3 /* err arg for bad_page_fault */ addi r3,r1,STACK_FRAME_OVERHEAD - lwz r4,_DAR(r1) bl bad_page_fault b ret_from_except_full -#ifdef CONFIG_PPC_BOOK3S_32 - /* We have a data breakpoint exception - handle it */ -handle_dabr_fault: - SAVE_NVGPRS(r1) - lwz r0,_TRAP(r1) - clrrwi r0,r0,1 - stw r0,_TRAP(r1) - bl do_break - b ret_from_except_full -#endif - /* * This routine switches between two different tasks. The process * state of one is saved on its kernel stack. Then the state diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S index 25fa7d5a643c..dc728bb1c89a 100644 --- a/arch/powerpc/kernel/exceptions-64e.S +++ b/arch/powerpc/kernel/exceptions-64e.S @@ -1018,9 +1018,8 @@ storage_fault_common: bne- 1f b ret_from_except_lite 1: bl save_nvgprs - mr r5,r3 + mr r4,r3 addi r3,r1,STACK_FRAME_OVERHEAD - ld r4,_DAR(r1) bl bad_page_fault b ret_from_except diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 690058043b17..77b730f515c4 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -2136,8 +2136,7 @@ EXC_COMMON_BEGIN(h_data_storage_common) GEN_COMMON h_data_storage addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION - ld r4,_DAR(r1) - li r5,SIGSEGV + li r4,SIGSEGV bl bad_page_fault MMU_FTR_SECTION_ELSE bl unknown_exception diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index 8acd365a2be6..71ad7ce28469 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -376,10 +376,9 @@ do_databreakpoint: addi r3,r1,STACK_FRAME_OVERHEAD mfspr r4,SPRN_BAR stw r4,_DAR(r11) -#ifdef CONFIG_VMAP_STACK - lwz r5,_DSISR(r11) -#else +#ifndef CONFIG_VMAP_STACK mfspr r5,SPRN_DSISR + stw r5,_DSISR(r11) #endif EXC_XFER_STD(0x1c00, do_break) diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S index 7addf67832f9..5875f8795d5b 100644 --- a/arch/powerpc/kernel/head_book3s_32.S +++ b/arch/powerpc/kernel/head_book3s_32.S @@ -689,7 +689,10 @@ handle_page_fault_tramp_1: #endif /* fall through */ handle_page_fault_tramp_2: + andis. r0, r5, DSISR_DABRMATCH@h + bne- 1f EXC_XFER_LITE(0x300, handle_page_fault) +1: EXC_XFER_STD(0x300, do_break) #ifdef CONFIG_VMAP_STACK .macro save_regs_thread thread diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index d421a2c7f822..0bdd3ed653df 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -660,11 +660,10 @@ static void do_break_handler(struct pt_regs *regs) } } -void do_break (struct pt_regs *regs, unsigned long address, - unsigned long error_code) +void do_break(struct pt_regs *regs) { current->thread.trap_nr = TRAP_HWBKPT; - if (notify_die(DIE_DABR_MATCH, "dabr_match", regs, error_code, + if (notify_die(DIE_DABR_MATCH, "dabr_match", regs, regs->dsisr, 11, SIGSEGV) == NOTIFY_STOP) return; @@ -682,7 +681,7 @@ void do_break (struct pt_regs *regs, unsigned long address, do_break_handler(regs); /* Deliver the signal to userspace */ - force_sig_fault(SIGTRAP, TRAP_HWBKPT, (void __user *)address); + force_sig_fault(SIGTRAP, TRAP_HWBKPT, (void __user *)regs->dar); } #endif /* CONFIG_PPC_ADV_DEBUG_REGS */ diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 5006dcbe1d9f..902fcbd1a778 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -1641,7 +1641,7 @@ void alignment_exception(struct pt_regs *regs) if (user_mode(regs)) _exception(sig, regs, code, regs->dar); else - bad_page_fault(regs, regs->dar, sig); + bad_page_fault(regs, sig); bail: exception_exit(prev_state); diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 0f0bd4af4b2d..731518e7d56f 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1537,7 +1537,7 @@ long do_hash_fault(struct pt_regs *regs) * the access, or panic if there isn't a handler. */ if (unlikely(in_nmi())) { - bad_page_fault(regs, ea, SIGSEGV); + bad_page_fault(regs, SIGSEGV); return 0; } @@ -1576,7 +1576,7 @@ long do_hash_fault(struct pt_regs *regs) else _exception(SIGBUS, regs, BUS_ADRERR, ea); } else { - bad_page_fault(regs, ea, SIGBUS); + bad_page_fault(regs, SIGBUS); } err = 0; diff --git a/arch/powerpc/mm/book3s64/slb.c b/arch/powerpc/mm/book3s64/slb.c index cc34d50874c1..ae89ad516247 100644 --- a/arch/powerpc/mm/book3s64/slb.c +++ b/arch/powerpc/mm/book3s64/slb.c @@ -898,7 +898,7 @@ void do_bad_slb_fault(struct pt_regs *regs) if (user_mode(regs)) _exception(SIGSEGV, regs, SEGV_BNDERR, regs->dar); else - bad_page_fault(regs, regs->dar, SIGSEGV); + bad_page_fault(regs, SIGSEGV); } else if (err == -EINVAL) { unrecoverable_exception(regs); } else { diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 390a296b16a3..e11989be8f1c 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -562,14 +562,14 @@ long do_page_fault(struct pt_regs *regs) /* 32 and 64e handle errors in their asm code */ if (unlikely(err)) { if (err > 0) { - bad_page_fault(regs, address, err); + bad_page_fault(regs, err); err = 0; } else { /* * do_break() may change NV GPRS while handling the * breakpoint. Return -ve to caller to do that. */ - do_break(regs, address, error_code); + do_break(regs); } } #endif @@ -591,14 +591,14 @@ long hash__do_page_fault(struct pt_regs *regs) err = __do_page_fault(regs, address, error_code); if (unlikely(err)) { if (err > 0) { - bad_page_fault(regs, address, err); + bad_page_fault(regs, err); err = 0; } else { /* * do_break() may change NV GPRS while handling the * breakpoint. Return -ve to caller to do that. */ - do_break(regs, address, error_code); + do_break(regs); } } @@ -612,7 +612,7 @@ NOKPROBE_SYMBOL(hash__do_page_fault); * It is called from the DSI and ISI handlers in head.S and from some * of the procedures in traps.c. */ -void bad_page_fault(struct pt_regs *regs, unsigned long address, int sig) +void bad_page_fault(struct pt_regs *regs, int sig) { const struct exception_table_entry *entry; int is_write = page_fault_is_write(regs->dsisr); diff --git a/arch/powerpc/platforms/8xx/machine_check.c b/arch/powerpc/platforms/8xx/machine_check.c index 88dedf38eccd..656365975895 100644 --- a/arch/powerpc/platforms/8xx/machine_check.c +++ b/arch/powerpc/platforms/8xx/machine_check.c @@ -26,7 +26,7 @@ int machine_check_8xx(struct pt_regs *regs) * to deal with that than having a wart in the mcheck handler. * -- BenH */ - bad_page_fault(regs, regs->dar, SIGBUS); + bad_page_fault(regs, SIGBUS); return 1; #else return 0; From patchwork Sat Nov 28 14:40:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407531 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvXl46tTz9s1l for ; Sun, 29 Nov 2020 01:49:43 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=JoEEFRom; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvXl39bhzF0x5 for ; Sun, 29 Nov 2020 01:49:43 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=JoEEFRom; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMJ3hhPzDsGv for ; Sun, 29 Nov 2020 01:41:32 +1100 (AEDT) Received: by mail-pf1-x442.google.com with SMTP id 131so6960956pfb.9 for ; Sat, 28 Nov 2020 06:41:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Epu9ANqD8QC3efzS4M5Om3MIjaPYiGpwNIUscp4pCWU=; b=JoEEFRomrrQvkFL36iqjgl54QtTrcCfpUI5Vm2UluesXpf2RVuooUotJ5GEZDM6RSw MggItqjjgFnyHkZ5OE8BB9YUVWbQoXgd+jEtJlotOY6siKiR8kd8D2IShRQZoOvKxesP oNSQOF7y2Rk6FJhoCpzOlaYL/kOMwFjwa2YgSHFbx8/LhzcDzgB1ItagG35bGvQFAs15 AGgYacgdMBak0eGW9t876FV3/8flLIMCyNC74h7/clBrAcDbD3IUf+xKeTAdT9O4CQU9 xhe42mQQhAIQ6pvVE9hS5Aa5KkGUP3bqcfvce+6l9ru6I8dGF3gzRTaoIi4aclrFctFc 2FDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Epu9ANqD8QC3efzS4M5Om3MIjaPYiGpwNIUscp4pCWU=; b=JLpFVP2Ax5DKMVxG4Nl9j9E7C0p3Zm4WPz3CBceuYOpxxgEmieq8HeMLlNtLXAWYeh GuhikhsPcKQJVmZoPvuJILB2a0tB9+64oQr5IBkfW+MREzjAI1AhTZHFpJYxrIK81LTQ 0XuFJl0IV/oKZrDqbHEizoJCoJraceUzbHXelWk71YZ26YlGHRVoejravDQPMOlT1EWJ K82rmx2Sv6t5DNJteMwfUH41ea2HCbFYQohbzApKN6h8W52a3LVSBDnT4aqqeuuwnYmO cUPRhquRepHxvJOUn0qSw8s6Y/mN7KhxXciDTSL9s6xuVjlAw+wF1ZaaVfwxlB5NUbau e4+w== X-Gm-Message-State: AOAM530SOreUkd46ThJrR+4IHeDjxEO+ntp9rKRCTk2S4dcMS9WMzZRF M+HesgWGCVkAT5ieANd1Pd24tOiB+3M= X-Google-Smtp-Source: ABdhPJwNf04ojqZNYwtOVdBUkn7+xksd31RAMj9Oo7dkbZ5oA7M85F1z6GzUTbleqWfKPVkrYRGXPw== X-Received: by 2002:a17:90b:249:: with SMTP id fz9mr15922988pjb.233.1606574490524; Sat, 28 Nov 2020 06:41:30 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:30 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 04/19] powerpc/perf: move perf irq/nmi handling details into traps.c Date: Sun, 29 Nov 2020 00:40:59 +1000 Message-Id: <20201128144114.944000-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This is required in order to allow more significant differences between NMI type interrupt handlers and regular asynchronous handlers. Signed-off-by: Nicholas Piggin --- arch/powerpc/kernel/traps.c | 31 +++++++++++++++++++++++++++- arch/powerpc/perf/core-book3s.c | 35 ++------------------------------ arch/powerpc/perf/core-fsl-emb.c | 25 ----------------------- 3 files changed, 32 insertions(+), 59 deletions(-) diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 902fcbd1a778..7dda72eb97cc 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -1919,11 +1919,40 @@ void vsx_unavailable_tm(struct pt_regs *regs) } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ -void performance_monitor_exception(struct pt_regs *regs) +static void performance_monitor_exception_nmi(struct pt_regs *regs) +{ + nmi_enter(); + + __this_cpu_inc(irq_stat.pmu_irqs); + + perf_irq(regs); + + nmi_exit(); +} + +static void performance_monitor_exception_async(struct pt_regs *regs) { + irq_enter(); + __this_cpu_inc(irq_stat.pmu_irqs); perf_irq(regs); + + irq_exit(); +} + +void performance_monitor_exception(struct pt_regs *regs) +{ + /* + * On 64-bit, if perf interrupts hit in a local_irq_disable + * (soft-masked) region, we consider them as NMIs. This is required to + * prevent hash faults on user addresses when reading callchains (and + * looks better from an irq tracing perspective). + */ + if (IS_ENABLED(CONFIG_PPC64) && unlikely(arch_irq_disabled_regs(regs))) + performance_monitor_exception_nmi(regs); + else + performance_monitor_exception_async(regs); } #ifdef CONFIG_PPC_ADV_DEBUG_REGS diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c index 08643cba1494..9fd8cae09218 100644 --- a/arch/powerpc/perf/core-book3s.c +++ b/arch/powerpc/perf/core-book3s.c @@ -109,10 +109,6 @@ static inline void perf_read_regs(struct pt_regs *regs) { regs->result = 0; } -static inline int perf_intr_is_nmi(struct pt_regs *regs) -{ - return 0; -} static inline int siar_valid(struct pt_regs *regs) { @@ -328,15 +324,6 @@ static inline void perf_read_regs(struct pt_regs *regs) regs->result = use_siar; } -/* - * If interrupts were soft-disabled when a PMU interrupt occurs, treat - * it as an NMI. - */ -static inline int perf_intr_is_nmi(struct pt_regs *regs) -{ - return (regs->softe & IRQS_DISABLED); -} - /* * On processors like P7+ that have the SIAR-Valid bit, marked instructions * must be sampled only if the SIAR-valid bit is set. @@ -2224,7 +2211,6 @@ static void __perf_event_interrupt(struct pt_regs *regs) struct perf_event *event; unsigned long val[8]; int found, active; - int nmi; if (cpuhw->n_limited) freeze_limited_counters(cpuhw, mfspr(SPRN_PMC5), @@ -2232,18 +2218,6 @@ static void __perf_event_interrupt(struct pt_regs *regs) perf_read_regs(regs); - /* - * If perf interrupts hit in a local_irq_disable (soft-masked) region, - * we consider them as NMIs. This is required to prevent hash faults on - * user addresses when reading callchains. See the NMI test in - * do_hash_page. - */ - nmi = perf_intr_is_nmi(regs); - if (nmi) - nmi_enter(); - else - irq_enter(); - /* Read all the PMCs since we'll need them a bunch of times */ for (i = 0; i < ppmu->n_counter; ++i) val[i] = read_pmc(i + 1); @@ -2289,8 +2263,8 @@ static void __perf_event_interrupt(struct pt_regs *regs) } } } - if (!found && !nmi && printk_ratelimit()) - printk(KERN_WARNING "Can't find PMC that caused IRQ\n"); + if (unlikely(!found) && !arch_irq_disabled_regs(regs)) + printk_ratelimited(KERN_WARNING "Can't find PMC that caused IRQ\n"); /* * Reset MMCR0 to its normal value. This will set PMXE and @@ -2300,11 +2274,6 @@ static void __perf_event_interrupt(struct pt_regs *regs) * we get back out of this interrupt. */ write_mmcr0(cpuhw, cpuhw->mmcr.mmcr0); - - if (nmi) - nmi_exit(); - else - irq_exit(); } static void perf_event_interrupt(struct pt_regs *regs) diff --git a/arch/powerpc/perf/core-fsl-emb.c b/arch/powerpc/perf/core-fsl-emb.c index e0e7e276bfd2..ee721f420a7b 100644 --- a/arch/powerpc/perf/core-fsl-emb.c +++ b/arch/powerpc/perf/core-fsl-emb.c @@ -31,19 +31,6 @@ static atomic_t num_events; /* Used to avoid races in calling reserve/release_pmc_hardware */ static DEFINE_MUTEX(pmc_reserve_mutex); -/* - * If interrupts were soft-disabled when a PMU interrupt occurs, treat - * it as an NMI. - */ -static inline int perf_intr_is_nmi(struct pt_regs *regs) -{ -#ifdef __powerpc64__ - return (regs->softe & IRQS_DISABLED); -#else - return 0; -#endif -} - static void perf_event_interrupt(struct pt_regs *regs); /* @@ -659,13 +646,6 @@ static void perf_event_interrupt(struct pt_regs *regs) struct perf_event *event; unsigned long val; int found = 0; - int nmi; - - nmi = perf_intr_is_nmi(regs); - if (nmi) - nmi_enter(); - else - irq_enter(); for (i = 0; i < ppmu->n_counter; ++i) { event = cpuhw->event[i]; @@ -690,11 +670,6 @@ static void perf_event_interrupt(struct pt_regs *regs) mtmsr(mfmsr() | MSR_PMM); mtpmr(PMRN_PMGC0, PMGC0_PMIE | PMGC0_FCECE); isync(); - - if (nmi) - nmi_exit(); - else - irq_exit(); } void hw_perf_event_setup(int cpu) From patchwork Sat Nov 28 14:41:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407532 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvb72plhz9s1l for ; Sun, 29 Nov 2020 01:51:47 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=pQf9tC5T; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvb72MwmzDrKj for ; Sun, 29 Nov 2020 01:51:47 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=pQf9tC5T; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMM5K0dzDsGv for ; Sun, 29 Nov 2020 01:41:35 +1100 (AEDT) Received: by mail-pf1-x442.google.com with SMTP id e8so6986696pfh.2 for ; Sat, 28 Nov 2020 06:41:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZPNWJmp8epCqiF+CzeQJvG10RU1oLxzzMCNLrD7SUxc=; b=pQf9tC5TptVRiB0c+SOv4NYuE+fxIBTuD1hi//L5Qr6gs33teQLajR/nZJXWoKPUpt OOgjRPTVBJ1eG37NxMa5/7V6rqgTE+F/Md5gwHkpMJew+PhC+lcanqkJ+iwygaBjnkOI 3QU+nsj2kMQAp07xyc4XVlw67gotnbNEbIKo4QQZD8JdQPJpKpAW/9nTVnVvFH8t8xaB xhLoktJwaikQglGcA9xrPVZzpO1BWeQpe9pcqmHH5EnYfaywDAgoGxfDBC4j42hj+kO3 H4h2sh5CcaBIYf/xvD3Hr7HjoxDXOY68wRPbkrXayXTmu7PaZaMrBWOVDdtcoBm9iQse fRqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZPNWJmp8epCqiF+CzeQJvG10RU1oLxzzMCNLrD7SUxc=; b=pDRBhYz7b0ZoO/b+654NE6hUHUHmJBNJ2ma5KXj3LCsuhRUB/4q5Nd7DiPUQxGUJqh cwJOL8DYBCgghCEa9p5K59Y2YJsyp33JmDo1D1CeERPPQnfyIwNsYe6oEetUyI6fyv/R au5oCcFf/Rh6trhTVkXyPaTn2EnwJHkUlsz1pxatwnwsI8ZIMFzTrYDuw2z0umNHJOak eb5iydqppeKFqaZgi3FCWwvopFm3CfE4UZi0u91p32y5p78fagFtKjSrkkbU2A4pEtFb uoG4ZIT88+k7Hdpdinu+KqFMmcVlZTG8fl3OysbVBw8fdWYLkOIfB/IAUMkX2smBQ1pB 1vrg== X-Gm-Message-State: AOAM5302YZBU3eHDVdcEjaUX59TwPEXJWMnjT9VwaRASF48FutWPETNd KOGIgWy9ipCEsynjJUVezasjtNiy6YE= X-Google-Smtp-Source: ABdhPJxk9/ScrPfbWbTOb2hn+63y6FmsWboDOfA94QSn7XyqsB1Vdf9iQhuCA3kurVaQumrtDo3L9g== X-Received: by 2002:a17:90b:4393:: with SMTP id in19mr16619476pjb.4.1606574493331; Sat, 28 Nov 2020 06:41:33 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:32 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 05/19] powerpc: interrupt handler wrapper functions Date: Sun, 29 Nov 2020 00:41:00 +1000 Message-Id: <20201128144114.944000-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add wrapper functions (derived from x86 macros) for interrupt handler functions. This allows interrupt entry code to be written in C. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/asm-prototypes.h | 29 --- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 1 - arch/powerpc/include/asm/hw_irq.h | 9 - arch/powerpc/include/asm/interrupt.h | 205 ++++++++++++++++++ arch/powerpc/include/asm/time.h | 2 + arch/powerpc/kernel/dbell.c | 12 +- arch/powerpc/kernel/exceptions-64s.S | 7 +- arch/powerpc/kernel/head_book3s_32.S | 6 +- arch/powerpc/kernel/irq.c | 3 +- arch/powerpc/kernel/mce.c | 5 +- arch/powerpc/kernel/syscall_64.c | 1 + arch/powerpc/kernel/tau_6xx.c | 2 +- arch/powerpc/kernel/time.c | 3 +- arch/powerpc/kernel/traps.c | 90 +++++--- arch/powerpc/kernel/watchdog.c | 7 +- arch/powerpc/kvm/book3s_hv.c | 1 + arch/powerpc/kvm/book3s_hv_builtin.c | 1 + arch/powerpc/kvm/booke.c | 1 + arch/powerpc/mm/book3s64/hash_utils.c | 3 +- arch/powerpc/mm/book3s64/slb.c | 5 +- arch/powerpc/mm/fault.c | 10 +- arch/powerpc/platforms/powernv/idle.c | 1 + 22 files changed, 311 insertions(+), 93 deletions(-) create mode 100644 arch/powerpc/include/asm/interrupt.h diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h index 22c9d08fa3a4..939f3c94c8f3 100644 --- a/arch/powerpc/include/asm/asm-prototypes.h +++ b/arch/powerpc/include/asm/asm-prototypes.h @@ -56,35 +56,6 @@ int exit_vmx_usercopy(void); int enter_vmx_ops(void); void *exit_vmx_ops(void *dest); -/* Traps */ -long machine_check_early(struct pt_regs *regs); -long hmi_exception_realmode(struct pt_regs *regs); -void SMIException(struct pt_regs *regs); -void handle_hmi_exception(struct pt_regs *regs); -void instruction_breakpoint_exception(struct pt_regs *regs); -void RunModeException(struct pt_regs *regs); -void single_step_exception(struct pt_regs *regs); -void program_check_exception(struct pt_regs *regs); -void alignment_exception(struct pt_regs *regs); -void StackOverflow(struct pt_regs *regs); -void stack_overflow_exception(struct pt_regs *regs); -void kernel_fp_unavailable_exception(struct pt_regs *regs); -void altivec_unavailable_exception(struct pt_regs *regs); -void vsx_unavailable_exception(struct pt_regs *regs); -void fp_unavailable_tm(struct pt_regs *regs); -void altivec_unavailable_tm(struct pt_regs *regs); -void vsx_unavailable_tm(struct pt_regs *regs); -void facility_unavailable_exception(struct pt_regs *regs); -void TAUException(struct pt_regs *regs); -void altivec_assist_exception(struct pt_regs *regs); -void unrecoverable_exception(struct pt_regs *regs); -void kernel_bad_stack(struct pt_regs *regs); -void system_reset_exception(struct pt_regs *regs); -void machine_check_exception(struct pt_regs *regs); -void emulation_assist_interrupt(struct pt_regs *regs); -long do_slb_fault(struct pt_regs *regs); -void do_bad_slb_fault(struct pt_regs *regs); - /* signals, syscalls and interrupts */ long sys_swapcontext(struct ucontext __user *old_ctx, struct ucontext __user *new_ctx, diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index e843d0b193d3..683a9c7d1b03 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -453,7 +453,6 @@ static inline unsigned long hpt_hash(unsigned long vpn, #define HPTE_LOCAL_UPDATE 0x1 #define HPTE_NOHPTE_UPDATE 0x2 -long do_hash_fault(struct pt_regs *regs); extern int __hash_page_4K(unsigned long ea, unsigned long access, unsigned long vsid, pte_t *ptep, unsigned long trap, unsigned long flags, int ssize, int subpage_prot); diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h index 0363734ff56e..614957f74cee 100644 --- a/arch/powerpc/include/asm/hw_irq.h +++ b/arch/powerpc/include/asm/hw_irq.h @@ -50,15 +50,6 @@ #ifndef __ASSEMBLY__ -extern void replay_system_reset(void); -extern void replay_soft_interrupts(void); - -extern void timer_interrupt(struct pt_regs *); -extern void timer_broadcast_interrupt(void); -extern void performance_monitor_exception(struct pt_regs *regs); -extern void WatchdogException(struct pt_regs *regs); -extern void unknown_exception(struct pt_regs *regs); - #ifdef CONFIG_PPC64 #include diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h new file mode 100644 index 000000000000..5f6eeb6d43f2 --- /dev/null +++ b/arch/powerpc/include/asm/interrupt.h @@ -0,0 +1,205 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _ASM_POWERPC_INTERRUPT_H +#define _ASM_POWERPC_INTERRUPT_H + +#include +#include + +/** + * DECLARE_INTERRUPT_HANDLER_RAW - Declare raw interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + */ +#define DECLARE_INTERRUPT_HANDLER_RAW(func) \ + __visible long func(struct pt_regs *regs) + +/** + * DEFINE_INTERRUPT_HANDLER_RAW - Define raw interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + * + * @func is called from ASM entry code. + * + * This is a plain function which does no tracing, reconciling, etc. + * The macro is written so it acts as function definition. Append the + * body with a pair of curly brackets. + */ +#define DEFINE_INTERRUPT_HANDLER_RAW(func) \ +static __always_inline long ___##func(struct pt_regs *regs); \ + \ +__visible noinstr long func(struct pt_regs *regs) \ +{ \ + long ret; \ + \ + ret = ___##func (regs); \ + \ + return ret; \ +} \ + \ +static __always_inline long ___##func(struct pt_regs *regs) + +/** + * DECLARE_INTERRUPT_HANDLER - Declare synchronous interrupt handler function + * @func: Function name of the entry point + */ +#define DECLARE_INTERRUPT_HANDLER(func) \ + __visible void func(struct pt_regs *regs) + +/** + * DEFINE_INTERRUPT_HANDLER - Define synchronous interrupt handler function + * @func: Function name of the entry point + * + * @func is called from ASM entry code. + * + * The macro is written so it acts as function definition. Append the + * body with a pair of curly brackets. + */ +#define DEFINE_INTERRUPT_HANDLER(func) \ +static __always_inline void ___##func(struct pt_regs *regs); \ + \ +__visible noinstr void func(struct pt_regs *regs) \ +{ \ + ___##func (regs); \ +} \ + \ +static __always_inline void ___##func(struct pt_regs *regs) + +/** + * DECLARE_INTERRUPT_HANDLER_RET - Declare synchronous interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + */ +#define DECLARE_INTERRUPT_HANDLER_RET(func) \ + __visible long func(struct pt_regs *regs) + +/** + * DEFINE_INTERRUPT_HANDLER_RET - Define synchronous interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + * + * @func is called from ASM entry code. + * + * The macro is written so it acts as function definition. Append the + * body with a pair of curly brackets. + */ +#define DEFINE_INTERRUPT_HANDLER_RET(func) \ +static __always_inline long ___##func(struct pt_regs *regs); \ + \ +__visible noinstr long func(struct pt_regs *regs) \ +{ \ + long ret; \ + \ + ret = ___##func (regs); \ + \ + return ret; \ +} \ + \ +static __always_inline long ___##func(struct pt_regs *regs) + +/** + * DECLARE_INTERRUPT_HANDLER_ASYNC - Declare asynchronous interrupt handler function + * @func: Function name of the entry point + */ +#define DECLARE_INTERRUPT_HANDLER_ASYNC(func) \ + __visible void func(struct pt_regs *regs) + +/** + * DEFINE_INTERRUPT_HANDLER_ASYNC - Define asynchronous interrupt handler function + * @func: Function name of the entry point + * + * @func is called from ASM entry code. + * + * The macro is written so it acts as function definition. Append the + * body with a pair of curly brackets. + */ +#define DEFINE_INTERRUPT_HANDLER_ASYNC(func) \ +static __always_inline void ___##func(struct pt_regs *regs); \ + \ +__visible noinstr void func(struct pt_regs *regs) \ +{ \ + ___##func (regs); \ +} \ + \ +static __always_inline void ___##func(struct pt_regs *regs) + +/** + * DECLARE_INTERRUPT_HANDLER_NMI - Declare NMI interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + */ +#define DECLARE_INTERRUPT_HANDLER_NMI(func) \ + __visible long func(struct pt_regs *regs) + +/** + * DEFINE_INTERRUPT_HANDLER_NMI - Define NMI interrupt handler function + * @func: Function name of the entry point + * @returns: Returns a value back to asm caller + * + * @func is called from ASM entry code. + * + * The macro is written so it acts as function definition. Append the + * body with a pair of curly brackets. + */ +#define DEFINE_INTERRUPT_HANDLER_NMI(func) \ +static __always_inline long ___##func(struct pt_regs *regs); \ + \ +__visible noinstr long func(struct pt_regs *regs) \ +{ \ + long ret; \ + \ + ret = ___##func (regs); \ + \ + return ret; \ +} \ + \ +static __always_inline long ___##func(struct pt_regs *regs) + + +/* Interrupt handlers */ +DECLARE_INTERRUPT_HANDLER_NMI(machine_check_early); +DECLARE_INTERRUPT_HANDLER_NMI(hmi_exception_realmode); +DECLARE_INTERRUPT_HANDLER(SMIException); +DECLARE_INTERRUPT_HANDLER(handle_hmi_exception); +DECLARE_INTERRUPT_HANDLER(instruction_breakpoint_exception); +DECLARE_INTERRUPT_HANDLER(RunModeException); +DECLARE_INTERRUPT_HANDLER(single_step_exception); +DECLARE_INTERRUPT_HANDLER(program_check_exception); +DECLARE_INTERRUPT_HANDLER(alignment_exception); +DECLARE_INTERRUPT_HANDLER(StackOverflow); +DECLARE_INTERRUPT_HANDLER(stack_overflow_exception); +DECLARE_INTERRUPT_HANDLER(kernel_fp_unavailable_exception); +DECLARE_INTERRUPT_HANDLER(altivec_unavailable_exception); +DECLARE_INTERRUPT_HANDLER(vsx_unavailable_exception); +DECLARE_INTERRUPT_HANDLER(fp_unavailable_tm); +DECLARE_INTERRUPT_HANDLER(altivec_unavailable_tm); +DECLARE_INTERRUPT_HANDLER(vsx_unavailable_tm); +DECLARE_INTERRUPT_HANDLER(facility_unavailable_exception); +DECLARE_INTERRUPT_HANDLER_ASYNC(TAUException); +DECLARE_INTERRUPT_HANDLER(altivec_assist_exception); +DECLARE_INTERRUPT_HANDLER(unrecoverable_exception); +DECLARE_INTERRUPT_HANDLER(kernel_bad_stack); +DECLARE_INTERRUPT_HANDLER_NMI(system_reset_exception); +#ifdef CONFIG_PPC_BOOK3S_64 +DECLARE_INTERRUPT_HANDLER_ASYNC(machine_check_exception); +#else +DECLARE_INTERRUPT_HANDLER_NMI(machine_check_exception); +#endif +DECLARE_INTERRUPT_HANDLER(emulation_assist_interrupt); +DECLARE_INTERRUPT_HANDLER_RAW(do_slb_fault); +DECLARE_INTERRUPT_HANDLER(do_bad_slb_fault); +DECLARE_INTERRUPT_HANDLER_RET(do_hash_fault); +DECLARE_INTERRUPT_HANDLER_RET(do_page_fault); +DECLARE_INTERRUPT_HANDLER(do_bad_page_fault); + +DECLARE_INTERRUPT_HANDLER_ASYNC(timer_interrupt); +DECLARE_INTERRUPT_HANDLER_NMI(performance_monitor_exception_nmi); +DECLARE_INTERRUPT_HANDLER_ASYNC(performance_monitor_exception_async); +DECLARE_INTERRUPT_HANDLER_RAW(performance_monitor_exception); +DECLARE_INTERRUPT_HANDLER(WatchdogException); +DECLARE_INTERRUPT_HANDLER(unknown_exception); +DECLARE_INTERRUPT_HANDLER_ASYNC(unknown_async_exception); + +void replay_system_reset(void); +void replay_soft_interrupts(void); + +#endif /* _ASM_POWERPC_INTERRUPT_H */ diff --git a/arch/powerpc/include/asm/time.h b/arch/powerpc/include/asm/time.h index 2f566c1a754c..335d6fd589a7 100644 --- a/arch/powerpc/include/asm/time.h +++ b/arch/powerpc/include/asm/time.h @@ -131,6 +131,8 @@ DECLARE_PER_CPU(u64, decrementers_next_tb); /* Convert timebase ticks to nanoseconds */ unsigned long long tb_to_ns(unsigned long long tb_ticks); +void timer_broadcast_interrupt(void); + /* SPLPAR */ void accumulate_stolen_time(void); diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c index 52680cf07c9d..c0f99f8ffa7d 100644 --- a/arch/powerpc/kernel/dbell.c +++ b/arch/powerpc/kernel/dbell.c @@ -12,14 +12,14 @@ #include #include +#include #include #include #include -#ifdef CONFIG_SMP - -void doorbell_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(doorbell_exception) { +#ifdef CONFIG_SMP struct pt_regs *old_regs = set_irq_regs(regs); irq_enter(); @@ -37,11 +37,7 @@ void doorbell_exception(struct pt_regs *regs) trace_doorbell_exit(regs); irq_exit(); set_irq_regs(old_regs); -} #else /* CONFIG_SMP */ -void doorbell_exception(struct pt_regs *regs) -{ printk(KERN_WARNING "Received doorbell on non-smp system\n"); -} #endif /* CONFIG_SMP */ - +} diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 77b730f515c4..f6ec455db13c 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -1923,7 +1923,7 @@ EXC_COMMON_BEGIN(doorbell_super_common) #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception #else - bl unknown_exception + bl unknown_async_exception #endif b interrupt_return @@ -2136,8 +2136,7 @@ EXC_COMMON_BEGIN(h_data_storage_common) GEN_COMMON h_data_storage addi r3,r1,STACK_FRAME_OVERHEAD BEGIN_MMU_FTR_SECTION - li r4,SIGSEGV - bl bad_page_fault + bl do_bad_page_fault MMU_FTR_SECTION_ELSE bl unknown_exception ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_TYPE_RADIX) @@ -2310,7 +2309,7 @@ EXC_COMMON_BEGIN(h_doorbell_common) #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception #else - bl unknown_exception + bl unknown_async_exception #endif b interrupt_return diff --git a/arch/powerpc/kernel/head_book3s_32.S b/arch/powerpc/kernel/head_book3s_32.S index 5875f8795d5b..2185c6c89f33 100644 --- a/arch/powerpc/kernel/head_book3s_32.S +++ b/arch/powerpc/kernel/head_book3s_32.S @@ -240,8 +240,8 @@ __secondary_hold_acknowledge: /* System reset */ /* core99 pmac starts the seconary here by changing the vector, and - putting it back to what it was (unknown_exception) when done. */ - EXCEPTION(0x100, Reset, unknown_exception, EXC_XFER_STD) + putting it back to what it was (unknown_async_exception) when done. */ + EXCEPTION(0x100, Reset, unknown_async_exception, EXC_XFER_STD) /* Machine check */ /* @@ -642,7 +642,7 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_NEED_DTLB_SW_LRU) #endif #ifndef CONFIG_TAU_INT -#define TAUException unknown_exception +#define TAUException unknown_async_exception #endif EXCEPTION(0x1300, Trap_13, instruction_breakpoint_exception, EXC_XFER_STD) diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c index 6b1eca53e36c..2055d204d08e 100644 --- a/arch/powerpc/kernel/irq.c +++ b/arch/powerpc/kernel/irq.c @@ -54,6 +54,7 @@ #include #include +#include #include #include #include @@ -665,7 +666,7 @@ void __do_irq(struct pt_regs *regs) irq_exit(); } -void do_IRQ(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(do_IRQ) { struct pt_regs *old_regs = set_irq_regs(regs); void *cursp, *irqsp, *sirqsp; diff --git a/arch/powerpc/kernel/mce.c b/arch/powerpc/kernel/mce.c index 63702c0badb9..b84459f45b1a 100644 --- a/arch/powerpc/kernel/mce.c +++ b/arch/powerpc/kernel/mce.c @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -588,7 +589,7 @@ EXPORT_SYMBOL_GPL(machine_check_print_event_info); * * regs->nip and regs->msr contains srr0 and ssr1. */ -long notrace machine_check_early(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_NMI(machine_check_early) { long handled = 0; u8 ftrace_enabled = this_cpu_get_ftrace_enabled(); @@ -722,7 +723,7 @@ long hmi_handle_debugtrig(struct pt_regs *regs) /* * Return values: */ -long hmi_exception_realmode(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_NMI(hmi_exception_realmode) { int ret; diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c index 310bcd768cd5..16eb2b5dafde 100644 --- a/arch/powerpc/kernel/syscall_64.c +++ b/arch/powerpc/kernel/syscall_64.c @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/powerpc/kernel/tau_6xx.c b/arch/powerpc/kernel/tau_6xx.c index 0b4694b8d248..46b2e5de4ef5 100644 --- a/arch/powerpc/kernel/tau_6xx.c +++ b/arch/powerpc/kernel/tau_6xx.c @@ -100,7 +100,7 @@ static void TAUupdate(int cpu) * with interrupts disabled */ -void TAUException(struct pt_regs * regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(TAUException) { int cpu = smp_processor_id(); diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index 7d372ff3504b..0089df57bb49 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -56,6 +56,7 @@ #include #include +#include #include #include #include @@ -545,7 +546,7 @@ void arch_irq_work_raise(void) * timer_interrupt - gets called when the decrementer overflows, * with interrupts disabled. */ -void timer_interrupt(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(timer_interrupt) { struct clock_event_device *evt = this_cpu_ptr(&decrementers); u64 *next_tb = this_cpu_ptr(&decrementers_next_tb); diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 7dda72eb97cc..e726fe5c384d 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include #include @@ -436,8 +437,7 @@ void hv_nmi_check_nonrecoverable(struct pt_regs *regs) regs->msr &= ~MSR_RI; #endif } - -void system_reset_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_NMI(system_reset_exception) { unsigned long hsrr0, hsrr1; bool saved_hsrrs = false; @@ -522,7 +522,10 @@ void system_reset_exception(struct pt_regs *regs) this_cpu_set_ftrace_enabled(ftrace_enabled); /* What should we do here? We could issue a shutdown or hard reset. */ + + return 0; } +NOKPROBE_SYMBOL(system_reset_exception); /* * I/O accesses can cause machine checks on powermacs. @@ -819,7 +822,12 @@ int machine_check_generic(struct pt_regs *regs) } #endif /* everything else */ -void machine_check_exception(struct pt_regs *regs) + +#ifdef CONFIG_PPC_BOOK3S_64 +DEFINE_INTERRUPT_HANDLER_ASYNC(machine_check_exception) +#else +DEFINE_INTERRUPT_HANDLER_NMI(machine_check_exception) +#endif { int recover = 0; @@ -869,13 +877,21 @@ void machine_check_exception(struct pt_regs *regs) if (!(regs->msr & MSR_RI)) die("Unrecoverable Machine check", regs, SIGBUS); +#ifdef CONFIG_PPC_BOOK3S_64 +bail: return; +#else + return 0; bail: if (nmi) nmi_exit(); + + return 0; +#endif } +NOKPROBE_SYMBOL(machine_check_exception); -void SMIException(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(SMIException) /* async? */ { die("System Management Interrupt", regs, SIGABRT); } @@ -1061,7 +1077,7 @@ static void p9_hmi_special_emu(struct pt_regs *regs) } #endif /* CONFIG_VSX */ -void handle_hmi_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(handle_hmi_exception) { struct pt_regs *old_regs; @@ -1090,7 +1106,7 @@ void handle_hmi_exception(struct pt_regs *regs) set_irq_regs(old_regs); } -void unknown_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(unknown_exception) { enum ctx_state prev_state = exception_enter(); @@ -1102,7 +1118,19 @@ void unknown_exception(struct pt_regs *regs) exception_exit(prev_state); } -void instruction_breakpoint_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(unknown_async_exception) +{ + enum ctx_state prev_state = exception_enter(); + + printk("Bad trap at PC: %lx, SR: %lx, vector=%lx\n", + regs->nip, regs->msr, regs->trap); + + _exception(SIGTRAP, regs, TRAP_UNK, 0); + + exception_exit(prev_state); +} + +DEFINE_INTERRUPT_HANDLER(instruction_breakpoint_exception) { enum ctx_state prev_state = exception_enter(); @@ -1117,12 +1145,12 @@ void instruction_breakpoint_exception(struct pt_regs *regs) exception_exit(prev_state); } -void RunModeException(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(RunModeException) { _exception(SIGTRAP, regs, TRAP_UNK, 0); } -void single_step_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(single_step_exception) { enum ctx_state prev_state = exception_enter(); @@ -1465,7 +1493,7 @@ static int emulate_math(struct pt_regs *regs) static inline int emulate_math(struct pt_regs *regs) { return -1; } #endif -void program_check_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(program_check_exception) { enum ctx_state prev_state = exception_enter(); unsigned int reason = get_reason(regs); @@ -1590,14 +1618,14 @@ NOKPROBE_SYMBOL(program_check_exception); * This occurs when running in hypervisor mode on POWER6 or later * and an illegal instruction is encountered. */ -void emulation_assist_interrupt(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(emulation_assist_interrupt) { regs->msr |= REASON_ILLEGAL; program_check_exception(regs); } NOKPROBE_SYMBOL(emulation_assist_interrupt); -void alignment_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(alignment_exception) { enum ctx_state prev_state = exception_enter(); int sig, code, fixed = 0; @@ -1647,7 +1675,7 @@ void alignment_exception(struct pt_regs *regs) exception_exit(prev_state); } -void StackOverflow(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(StackOverflow) { pr_crit("Kernel stack overflow in process %s[%d], r1=%lx\n", current->comm, task_pid_nr(current), regs->gpr[1]); @@ -1656,7 +1684,7 @@ void StackOverflow(struct pt_regs *regs) panic("kernel stack overflow"); } -void stack_overflow_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(stack_overflow_exception) { enum ctx_state prev_state = exception_enter(); @@ -1665,7 +1693,7 @@ void stack_overflow_exception(struct pt_regs *regs) exception_exit(prev_state); } -void kernel_fp_unavailable_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(kernel_fp_unavailable_exception) { enum ctx_state prev_state = exception_enter(); @@ -1676,7 +1704,7 @@ void kernel_fp_unavailable_exception(struct pt_regs *regs) exception_exit(prev_state); } -void altivec_unavailable_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(altivec_unavailable_exception) { enum ctx_state prev_state = exception_enter(); @@ -1695,7 +1723,7 @@ void altivec_unavailable_exception(struct pt_regs *regs) exception_exit(prev_state); } -void vsx_unavailable_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(vsx_unavailable_exception) { if (user_mode(regs)) { /* A user program has executed an vsx instruction, @@ -1726,7 +1754,7 @@ static void tm_unavailable(struct pt_regs *regs) die("Unrecoverable TM Unavailable Exception", regs, SIGABRT); } -void facility_unavailable_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(facility_unavailable_exception) { static char *facility_strings[] = { [FSCR_FP_LG] = "FPU", @@ -1846,7 +1874,7 @@ void facility_unavailable_exception(struct pt_regs *regs) #ifdef CONFIG_PPC_TRANSACTIONAL_MEM -void fp_unavailable_tm(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(fp_unavailable_tm) { /* Note: This does not handle any kind of FP laziness. */ @@ -1879,7 +1907,7 @@ void fp_unavailable_tm(struct pt_regs *regs) tm_recheckpoint(¤t->thread); } -void altivec_unavailable_tm(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(altivec_unavailable_tm) { /* See the comments in fp_unavailable_tm(). This function operates * the same way. @@ -1894,7 +1922,7 @@ void altivec_unavailable_tm(struct pt_regs *regs) current->thread.used_vr = 1; } -void vsx_unavailable_tm(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(vsx_unavailable_tm) { /* See the comments in fp_unavailable_tm(). This works similarly, * though we're loading both FP and VEC registers in here. @@ -1919,7 +1947,8 @@ void vsx_unavailable_tm(struct pt_regs *regs) } #endif /* CONFIG_PPC_TRANSACTIONAL_MEM */ -static void performance_monitor_exception_nmi(struct pt_regs *regs) +#ifdef CONFIG_PPC64 +DEFINE_INTERRUPT_HANDLER_NMI(performance_monitor_exception_nmi) { nmi_enter(); @@ -1928,9 +1957,12 @@ static void performance_monitor_exception_nmi(struct pt_regs *regs) perf_irq(regs); nmi_exit(); + + return 0; } +#endif -static void performance_monitor_exception_async(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_ASYNC(performance_monitor_exception_async) { irq_enter(); @@ -1941,7 +1973,7 @@ static void performance_monitor_exception_async(struct pt_regs *regs) irq_exit(); } -void performance_monitor_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_RAW(performance_monitor_exception) { /* * On 64-bit, if perf interrupts hit in a local_irq_disable @@ -1953,6 +1985,8 @@ void performance_monitor_exception(struct pt_regs *regs) performance_monitor_exception_nmi(regs); else performance_monitor_exception_async(regs); + + return 0; } #ifdef CONFIG_PPC_ADV_DEBUG_REGS @@ -2086,7 +2120,7 @@ NOKPROBE_SYMBOL(DebugException); #endif /* CONFIG_PPC_ADV_DEBUG_REGS */ #ifdef CONFIG_ALTIVEC -void altivec_assist_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(altivec_assist_exception) { int err; @@ -2228,7 +2262,7 @@ void SPEFloatingPointRoundException(struct pt_regs *regs) * in the MSR is 0. This indicates that SRR0/1 are live, and that * we therefore lost state by taking this exception. */ -void unrecoverable_exception(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(unrecoverable_exception) { pr_emerg("Unrecoverable exception %lx at %lx (msr=%lx)\n", regs->trap, regs->nip, regs->msr); @@ -2248,7 +2282,7 @@ void __attribute__ ((weak)) WatchdogHandler(struct pt_regs *regs) return; } -void WatchdogException(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(WatchdogException) /* XXX NMI? async? */ { printk (KERN_EMERG "PowerPC Book-E Watchdog Exception\n"); WatchdogHandler(regs); @@ -2259,7 +2293,7 @@ void WatchdogException(struct pt_regs *regs) * We enter here if we discover during exception entry that we are * running in supervisor mode with a userspace value in the stack pointer. */ -void kernel_bad_stack(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(kernel_bad_stack) { printk(KERN_EMERG "Bad kernel stack pointer %lx at %lx\n", regs->gpr[1], regs->nip); diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c index af3c15a1d41e..824b9376ac35 100644 --- a/arch/powerpc/kernel/watchdog.c +++ b/arch/powerpc/kernel/watchdog.c @@ -26,6 +26,7 @@ #include #include +#include #include /* @@ -247,14 +248,14 @@ static void watchdog_timer_interrupt(int cpu) watchdog_smp_panic(cpu, tb); } -void soft_nmi_interrupt(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) { unsigned long flags; int cpu = raw_smp_processor_id(); u64 tb; if (!cpumask_test_cpu(cpu, &wd_cpus_enabled)) - return; + return 0; nmi_enter(); @@ -291,6 +292,8 @@ void soft_nmi_interrupt(struct pt_regs *regs) out: nmi_exit(); + + return 0; } static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index e3b1839fc251..01816de0e0ec 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include #include diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c index 8f58dd20b362..a2f3e6e70361 100644 --- a/arch/powerpc/kvm/book3s_hv_builtin.c +++ b/arch/powerpc/kvm/book3s_hv_builtin.c @@ -17,6 +17,7 @@ #include #include +#include #include #include #include diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c index b1abcb816439..2a9817311ac1 100644 --- a/arch/powerpc/kvm/booke.c +++ b/arch/powerpc/kvm/booke.c @@ -20,6 +20,7 @@ #include #include +#include #include #include #include diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index 731518e7d56f..a48c484b9e9b 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -38,6 +38,7 @@ #include #include +#include #include #include #include @@ -1510,7 +1511,7 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap, } EXPORT_SYMBOL_GPL(hash_page); -long do_hash_fault(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_RET(do_hash_fault) { unsigned long ea = regs->dar; unsigned long dsisr = regs->dsisr; diff --git a/arch/powerpc/mm/book3s64/slb.c b/arch/powerpc/mm/book3s64/slb.c index ae89ad516247..efac69e73ca8 100644 --- a/arch/powerpc/mm/book3s64/slb.c +++ b/arch/powerpc/mm/book3s64/slb.c @@ -10,6 +10,7 @@ */ #include +#include #include #include #include @@ -837,7 +838,7 @@ static long slb_allocate_user(struct mm_struct *mm, unsigned long ea) return slb_insert_entry(ea, context, flags, ssize, false); } -long do_slb_fault(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_RAW(do_slb_fault) { unsigned long ea = regs->dar; unsigned long id = get_region_id(ea); @@ -890,7 +891,7 @@ long do_slb_fault(struct pt_regs *regs) } } -void do_bad_slb_fault(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER(do_bad_slb_fault) { int err = regs->result; diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index e11989be8f1c..93663f7fbe9e 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -34,6 +34,7 @@ #include #include +#include #include #include #include @@ -549,7 +550,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, } NOKPROBE_SYMBOL(__do_page_fault); -long do_page_fault(struct pt_regs *regs) +DEFINE_INTERRUPT_HANDLER_RET(do_page_fault) { enum ctx_state prev_state = exception_enter(); unsigned long address = regs->dar; @@ -656,3 +657,10 @@ void bad_page_fault(struct pt_regs *regs, int sig) die("Kernel access of bad area", regs, sig); } + +#ifdef CONFIG_PPC_BOOK3S_64 +DEFINE_INTERRUPT_HANDLER(do_bad_page_fault) +{ + bad_page_fault(regs, SIGSEGV); +} +#endif diff --git a/arch/powerpc/platforms/powernv/idle.c b/arch/powerpc/platforms/powernv/idle.c index 1ed7c5286487..2fc7049fef53 100644 --- a/arch/powerpc/platforms/powernv/idle.c +++ b/arch/powerpc/platforms/powernv/idle.c @@ -14,6 +14,7 @@ #include #include +#include #include #include #include From patchwork Sat Nov 28 14:41:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407533 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvdD42c9z9s1l for ; Sun, 29 Nov 2020 01:53:36 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=M9XcYXM6; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvdD3K11zF0tD for ; Sun, 29 Nov 2020 01:53:36 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::644; helo=mail-pl1-x644.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=M9XcYXM6; dkim-atps=neutral Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMQ3s57zDsGv for ; Sun, 29 Nov 2020 01:41:38 +1100 (AEDT) Received: by mail-pl1-x644.google.com with SMTP id x4so2319668pln.8 for ; Sat, 28 Nov 2020 06:41:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EEx+v78bxX7ip5MB5LrvbtkDrNTgl+f+WgHGvGOpR74=; b=M9XcYXM6EO/T8EwgIstlxU/1qztA+Z3Ssmt8X8QubngOgDpm+MFSiHc1b08Zo8gEDx CxdHRq5bAIm7kfXX29AnfHezFdBlguvixM1y5C8+MUu3fdtwfzBesZ3428PTs1Tu6yuZ PtODcNEi9X50EapkcCp8kOkwhhMIW5rRV9B9ru4sVojqLj0gRkjbNsHxXExrWvGBzk/l 1nCC6ukWpWdSBnEozvPYvT1F87XZherfyTTu8toXHfxd1edyP1nv8Fq+AVnw9miMiOEP w71Be1S/rl0iFA5BaYpVBQKdoEvwhw9kvBvxC0h5XUgNOudUGv331VV+xNm8ZebYCGWc T03Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EEx+v78bxX7ip5MB5LrvbtkDrNTgl+f+WgHGvGOpR74=; b=hPVWSZfJrwpKRbltmFJ4F9yYPgDNj3dQNVf+SgHZhwqaHz/mftRHsm68g0D9DAW4D/ LEUnyLKb51+O5mIJHQ7aaiMr6cx33vqdJGaxkkG9RnqfwSaaX+QyHu03CKRq8jS9A+4Z pQbrq+nsc6BMduktWCCWpMaX2GjvKBbdvwD26AkUGMSxc0vQEA6KKBB4tYESSuCQrClG J6QUILtDyDfUEyQkihEmylKXsDVkHJyC8hlqn1vvBWeowxJf+gfaB3kicNV5xaPyNvH/ 9g4IKvT9Hpb6WPIak7VN+aML4o0d2wd1JKvJFoRnB9xI+mbmNQJr5GMAcb6vPudNApqB 92yg== X-Gm-Message-State: AOAM530yTLd+1PfhBTI0lP/WkqSE7NobSwkBsqFcTFaHckYOTHMSUgGM 1uk1N326gLi+2s+UzhTo1uHVI0+VH/Q= X-Google-Smtp-Source: ABdhPJyhA6MF90WvFaFjAZ0mF6+E1JP81JWAk0divIYkioK1SXMeIRXiD0+Hu/zzgLu6XF4ZLkRrCQ== X-Received: by 2002:a17:902:854a:b029:da:113b:2d3f with SMTP id d10-20020a170902854ab02900da113b2d3fmr11622502plo.49.1606574495502; Sat, 28 Nov 2020 06:41:35 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:35 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 06/19] powerpc: add interrupt wrapper entry / exit stub functions Date: Sun, 29 Nov 2020 00:41:01 +1000 Message-Id: <20201128144114.944000-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These will be used by subsequent patches. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 66 ++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 5f6eeb6d43f2..516b32f5b3cc 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -5,6 +5,50 @@ #include #include +struct interrupt_state { +}; + +static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) +{ +} + +/* + * Care should be taken to note that interrupt_exit_prepare and + * interrupt_async_exit_prepare do not necessarily return immediately to + * regs context (e.g., if regs is usermode, we don't necessarily return to + * user mode). Other interrupts might be taken between here and return, + * context switch / preemption may occur in the exit path after this, or a + * signal may be delivered, etc. + * + * The real interrupt exit code is platform specific, e.g., + * interrupt_exit_user_prepare / interrupt_exit_kernel_prepare for 64s. + * + * However interrupt_nmi_exit_prepare does return directly to regs, because + * NMIs do not do "exit work" or replay soft-masked interrupts. + */ +static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) +{ +} + +static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) +{ +} + +static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) +{ +} + +struct interrupt_nmi_state { +}; + +static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct interrupt_nmi_state *state) +{ +} + +static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct interrupt_nmi_state *state) +{ +} + /** * DECLARE_INTERRUPT_HANDLER_RAW - Declare raw interrupt handler function * @func: Function name of the entry point @@ -59,7 +103,13 @@ static __always_inline void ___##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ + struct interrupt_state state; \ + \ + interrupt_enter_prepare(regs, &state); \ + \ ___##func (regs); \ + \ + interrupt_exit_prepare(regs, &state); \ } \ \ static __always_inline void ___##func(struct pt_regs *regs) @@ -87,10 +137,15 @@ static __always_inline long ___##func(struct pt_regs *regs); \ \ __visible noinstr long func(struct pt_regs *regs) \ { \ + struct interrupt_state state; \ long ret; \ \ + interrupt_enter_prepare(regs, &state); \ + \ ret = ___##func (regs); \ \ + interrupt_exit_prepare(regs, &state); \ + \ return ret; \ } \ \ @@ -117,7 +172,13 @@ static __always_inline void ___##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ + struct interrupt_state state; \ + \ + interrupt_async_enter_prepare(regs, &state); \ + \ ___##func (regs); \ + \ + interrupt_async_exit_prepare(regs, &state); \ } \ \ static __always_inline void ___##func(struct pt_regs *regs) @@ -145,10 +206,15 @@ static __always_inline long ___##func(struct pt_regs *regs); \ \ __visible noinstr long func(struct pt_regs *regs) \ { \ + struct interrupt_nmi_state state; \ long ret; \ \ + interrupt_nmi_enter_prepare(regs, &state); \ + \ ret = ___##func (regs); \ \ + interrupt_nmi_exit_prepare(regs, &state); \ + \ return ret; \ } \ \ From patchwork Sat Nov 28 14:41:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407534 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvfy4nS1z9s1l for ; Sun, 29 Nov 2020 01:55:06 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=a6BVtBxJ; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvfy3yC2zF16M for ; Sun, 29 Nov 2020 01:55:06 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=a6BVtBxJ; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMS3swVzDsNc for ; Sun, 29 Nov 2020 01:41:40 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id w202so6962374pff.10 for ; Sat, 28 Nov 2020 06:41:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AOuaFMNUV8Ecub6Ukg61d9g61CMrwF1GuOID9Wy/Qb0=; b=a6BVtBxJ8AvO+0+UgD8LTEJMPPD+fgwsC0Q/vX4JxVvnqqvF5TYhxTfJrtUb7y6EUR gIVTKZRImsnpVXaz0Sx56/LKyAfxoWfCjuh4DuSME4odDo6Rxo1sEIEqyZcsUtJKBlty 3+XjFm0UVioIhjcGyfTFB04FLMz/ndoM29h6IjcXps9flmo+1fioChReAJCl63VcdByw NqNHJHKn5mAd/jg+LYObTON5R4fVg6ljHAtH5DbyG4SQbyZ/Cg5/6DVCifwBqGVro+cF JjzPbpHGotJbk0+HJ1tsLqK70sYkiNN5Yw0wkJ7scELNhZYgUSMa/SKUYrqf1wozYJLk dcqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AOuaFMNUV8Ecub6Ukg61d9g61CMrwF1GuOID9Wy/Qb0=; b=dR/A+VUr1gwp51QckO6EjIR9B4FYEwcKRuosOyI/nYeg2G1bA213gcklVLwN8zh6Un P6ItDK2U0LQLKiOwM5Dl9aE+4fiy6o2y9bGC/Uco9bXJxIaEBtUU58yPHWufl7UKLX0F MUsMn5MWYJX9IC79MKR4da/XScAzWrI3ZoGTBZJPFA6m0ygYyKS6MqUEZmxtce/ebvVQ eRuczyvM7qHVAWx65jXlHUrAVULRUGkJsSoPPd0024omC2FUn2YdjV8k7usssbxv6eNj syQ7Rr3ijQsWfyQ1HTJxyyqp8Ai+ebVpMxvRngMhzBLzGT+4cdFLCouhiCg6PLoUa9B3 VI9Q== X-Gm-Message-State: AOAM532+Jgj0gjLE0E2DwoKT7/++XeC10aaklWzCJml4PbQkLEJp5zFB qtkya9SoXvfhaKWnNW9/QkbQSA7ej6c= X-Google-Smtp-Source: ABdhPJwJsm1yxqYEjk7g4IzU+7LaaibdNO86xRxgC1JZjYWfYIoQ38bgxmZBS9ejkDeHhXjZOShywA== X-Received: by 2002:a17:90a:df0a:: with SMTP id gp10mr3401474pjb.216.1606574497628; Sat, 28 Nov 2020 06:41:37 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:37 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 07/19] powerpc: add interrupt_cond_local_irq_enable helper Date: Sun, 29 Nov 2020 00:41:02 +1000 Message-Id: <20201128144114.944000-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Simple helper for synchronous interrupt handlers to use to enable interrupts if they were taken in interrupt-enabled context. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 7 +++++++ arch/powerpc/kernel/traps.c | 24 +++++++----------------- arch/powerpc/mm/fault.c | 4 +--- 3 files changed, 15 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 516b32f5b3cc..2ab0675c27aa 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -3,6 +3,7 @@ #define _ASM_POWERPC_INTERRUPT_H #include +#include #include struct interrupt_state { @@ -268,4 +269,10 @@ DECLARE_INTERRUPT_HANDLER_ASYNC(unknown_async_exception); void replay_system_reset(void); void replay_soft_interrupts(void); +static inline void interrupt_cond_local_irq_enable(struct pt_regs *regs) +{ + if (!arch_irq_disabled_regs(regs)) + local_irq_enable(); +} + #endif /* _ASM_POWERPC_INTERRUPT_H */ diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index e726fe5c384d..8a6cb2838ce6 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -343,8 +343,8 @@ static bool exception_common(int signr, struct pt_regs *regs, int code, show_signal_msg(signr, regs, code, addr); - if (arch_irqs_disabled() && !arch_irq_disabled_regs(regs)) - local_irq_enable(); + if (arch_irqs_disabled()) + interrupt_cond_local_irq_enable(regs); current->thread.trap_nr = code; @@ -1575,9 +1575,7 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) if (!user_mode(regs)) goto sigill; - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); /* (reason & REASON_ILLEGAL) would be the obvious thing here, * but there seems to be a hardware bug on the 405GP (RevD) @@ -1631,9 +1629,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception) int sig, code, fixed = 0; unsigned long reason; - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); reason = get_reason(regs); @@ -1794,9 +1790,7 @@ DEFINE_INTERRUPT_HANDLER(facility_unavailable_exception) die("Unexpected facility unavailable exception", regs, SIGABRT); } - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); if (status == FSCR_DSCR_LG) { /* @@ -2176,9 +2170,7 @@ void SPEFloatingPointException(struct pt_regs *regs) int code = FPE_FLTUNK; int err; - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); flush_spe_to_thread(current); @@ -2225,9 +2217,7 @@ void SPEFloatingPointRoundException(struct pt_regs *regs) extern int speround_handler(struct pt_regs *regs); int err; - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); preempt_disable(); if (regs->msr & MSR_SPE) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 93663f7fbe9e..4896f44c8374 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -443,9 +443,7 @@ static int __do_page_fault(struct pt_regs *regs, unsigned long address, return bad_area_nosemaphore(regs, address); } - /* We restore the interrupt state now */ - if (!arch_irq_disabled_regs(regs)) - local_irq_enable(); + interrupt_cond_local_irq_enable(regs); perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, address); From patchwork Sat Nov 28 14:41:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407535 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvjG2rq4z9s1l for ; Sun, 29 Nov 2020 01:57:06 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ZORf4CNn; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvjG2b2vzF156 for ; Sun, 29 Nov 2020 01:57:06 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::541; helo=mail-pg1-x541.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ZORf4CNn; dkim-atps=neutral Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMV4jnKzDsQ7 for ; Sun, 29 Nov 2020 01:41:42 +1100 (AEDT) Received: by mail-pg1-x541.google.com with SMTP id l4so544534pgu.5 for ; Sat, 28 Nov 2020 06:41:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OfHhuTfV+3ngyPqmG9KkeN9zzfuiK23prlzAo3VpEnc=; b=ZORf4CNnmITHRrVZy3vkHo30v2XhLhc9qVYa3UdywahR90JKIpFtofLVHDDTQXnlLC HuhCinXkmIj2Bl96Y1wHsIh09nZkUKk8CPsYKK3lgiE9v1rI6XZTGnygv7xNw3ILXelx 8Ob+4X3qWYYeigSqMxcALPwquIuDluYhF9o7NrfEAAAFQp2ohIxLN/N9RTUCWYaBzaqr X+nP8cMLjQ+VgWzcfBzeEVZcHTJbOS9/xGKJ4yCnPTlTGk9AqIm8ABaZM14Ezjf7JLB3 RAt5xado/Xg9/xAFqcXH8PGlsUsdInmeGOcebxwcSM02oxqgF1rU6llsB1Tl74A403jY 1Hpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OfHhuTfV+3ngyPqmG9KkeN9zzfuiK23prlzAo3VpEnc=; b=QC/6hCEYCEgq/vregnyM04UqEl2x15qUU0pYOLpajUUFCHnhOsu5TJhZJezfZmPWHw TODMXXAT0I8d4MphNASxUYmjpNdn9MxVZx2K96ddRR9NeRkHslO13DSqV+Eg0FDztPdF BpdSRwVzwk6LlDEz6z6uWGbcLU5wZ9feI2eXY1vV3UZRvQYBWYnTHnZGzCzRlRiaqc/e Sd43AmX05NJCy+Zz59wqy+qI7nY5ci1z+pA9kEbhavdO1uSJ8py4/fKiB7vCcyGHDhQ8 S6xbtKaNoOWr2XFS5EDnoVM93wlhdr/LUY0oLY7fTMVJBJj/gNlj1cSKY59Hler2ggkg Qvnw== X-Gm-Message-State: AOAM530eWHzJfbk8DzWiNqZ+cE2xmxEG87PQrT/EitwLhraBgYoOsV96 NQB10WmZB7jC8MoCH4bMtHOWRkNJLjA= X-Google-Smtp-Source: ABdhPJwn98SN6waHCi1eE6a5Yzqs9SkBiRw2iS7qHT1KhLYKYDTOZvva9FzMkjVhlMLEly2DitLhwQ== X-Received: by 2002:a17:90a:c70f:: with SMTP id o15mr3713889pjt.40.1606574499877; Sat, 28 Nov 2020 06:41:39 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:39 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 08/19] powerpc/64: context tracking remove _TIF_NOHZ Date: Sun, 29 Nov 2020 00:41:03 +1000 Message-Id: <20201128144114.944000-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add context tracking to the system call handler explicitly, and remove _TIF_NOHZ. This saves 35 cycles on gettid system call cost on POWER9 with a CONFIG_NOHZ_FULL kernel. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 6 ------ arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/thread_info.h | 4 +--- arch/powerpc/kernel/ptrace/ptrace.c | 4 ---- arch/powerpc/kernel/signal.c | 4 ---- arch/powerpc/kernel/syscall_64.c | 10 ++++++++++ 6 files changed, 11 insertions(+), 18 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 56b6ccc0e32d..a0b6213f7820 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -618,12 +618,6 @@ config HAVE_CONTEXT_TRACKING protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal handling on irq exit still need to be protected. -config HAVE_TIF_NOHZ - bool - help - Arch relies on TIF_NOHZ and syscall slow path to implement context - tracking calls to user_enter()/user_exit(). - config HAVE_VIRT_CPU_ACCOUNTING bool diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e9f13fe08492..6eaf12a504f8 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -193,7 +193,6 @@ config PPC select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13) select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2) select HAVE_CONTEXT_TRACKING if PPC64 - select HAVE_TIF_NOHZ if PPC64 select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW select HAVE_DYNAMIC_FTRACE diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h index 46a210b03d2b..c9443c16e5fb 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -95,7 +95,6 @@ void arch_setup_new_exec(void); #define TIF_PATCH_PENDING 6 /* pending live patching update */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ #define TIF_SINGLESTEP 8 /* singlestepping active */ -#define TIF_NOHZ 9 /* in adaptive nohz mode */ #define TIF_SECCOMP 10 /* secure computing */ #define TIF_RESTOREALL 11 /* Restore all regs (implies NOERROR) */ #define TIF_NOERROR 12 /* Force successful syscall return */ @@ -128,11 +127,10 @@ void arch_setup_new_exec(void); #define _TIF_UPROBE (1<flags) & (_TIF_SYSCALL_EMU | _TIF_SYSCALL_TRACE); @@ -368,8 +366,6 @@ void do_syscall_trace_leave(struct pt_regs *regs) step = test_thread_flag(TIF_SINGLESTEP); if (step || test_thread_flag(TIF_SYSCALL_TRACE)) tracehook_report_syscall_exit(regs, step); - - user_enter(); } void __init pt_regs_check(void); diff --git a/arch/powerpc/kernel/signal.c b/arch/powerpc/kernel/signal.c index d2c356f37077..44ec7b34b27e 100644 --- a/arch/powerpc/kernel/signal.c +++ b/arch/powerpc/kernel/signal.c @@ -310,8 +310,6 @@ static void do_signal(struct task_struct *tsk) void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) { - user_exit(); - if (thread_info_flags & _TIF_UPROBE) uprobe_notify_resume(regs); @@ -327,8 +325,6 @@ void do_notify_resume(struct pt_regs *regs, unsigned long thread_info_flags) tracehook_notify_resume(regs); rseq_handle_notify_resume(NULL, regs); } - - user_enter(); } unsigned long get_tm_stackpointer(struct task_struct *tsk) diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c index 16eb2b5dafde..9e855b25bd15 100644 --- a/arch/powerpc/kernel/syscall_64.c +++ b/arch/powerpc/kernel/syscall_64.c @@ -1,9 +1,11 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#include #include #include #include #include +#include #include #include #include @@ -28,6 +30,9 @@ notrace long system_call_exception(long r3, long r4, long r5, if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG)) BUG_ON(irq_soft_mask_return() != IRQS_ALL_DISABLED); + CT_WARN_ON(ct_state() == CONTEXT_KERNEL); + user_exit_irqoff(); + trace_hardirqs_off(); /* finish reconciling */ if (IS_ENABLED(CONFIG_PPC_BOOK3S)) @@ -158,6 +163,8 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, unsigned long ti_flags; unsigned long ret = 0; + CT_WARN_ON(ct_state() == CONTEXT_USER); + kuap_check_amr(); regs->result = r3; @@ -234,8 +241,11 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3, } } + user_enter_irqoff(); + /* scv need not set RI=0 because SRRs are not used */ if (unlikely(!prep_irq_for_enabled_exit(!scv))) { + user_exit_irqoff(); local_irq_enable(); goto again; } From patchwork Sat Nov 28 14:41:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407537 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvl46sgBz9s1l for ; Sun, 29 Nov 2020 01:58:40 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=HKRMjBr6; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvl46779zDrQS for ; Sun, 29 Nov 2020 01:58:40 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::441; helo=mail-pf1-x441.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=HKRMjBr6; dkim-atps=neutral Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMY4QdBzDsP0 for ; Sun, 29 Nov 2020 01:41:45 +1100 (AEDT) Received: by mail-pf1-x441.google.com with SMTP id s21so6950259pfu.13 for ; Sat, 28 Nov 2020 06:41:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oBNoe6wJJfg8iXXkzlcN36tMDXFywk3NQBWUtEDQoHY=; b=HKRMjBr6CfeAwyigD2EKf5gIxCeg7qLqYwoOIN8xUKyaxnvsI5VFLBLXT+OD8kwgic elicVuhEi8pjJLDv1y55b3OVliWX55XeznKj6WIjjA+8LJNv4ves4Kpv1aXIQDhonsP3 OKVE49vbt1KdABGQ4LWFYAECrzL+IuIf3Juu9LtlDN/N3gHnsNZBcrZGnm619hQPTr0U hko1+xqAs+Uvf1F8w5PPl3qqx43LHx62dCLwtUA2GdQEidZzOkTqYmiZEEp56ygGHkID RLBUj5nUpw+D3VH9IYLXfh18HreogGi8VmuOzvqV2OY1AgP/lRS5GqP4fbRgtdtzseao cGFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oBNoe6wJJfg8iXXkzlcN36tMDXFywk3NQBWUtEDQoHY=; b=WP5Iq8a2Wfy6xiewagHcxXj+isKFBuPOZj9v1vrmzOhiscgT9pXHY25agg4vWr/d3M cdJUatJDj2qOzzUH8FPNVVNpHK5t0RET46Wn0VSlBT00OVsf58LzaLrp/USQQZhZneG9 sSXTb73id4y2m2TVgWRI4PeRbnPfCXOMERV7/DFANAygAum9I0kDQmiGk6S/NmoHX5WU P8OEDRRJ/O+7AdiudG1PHxAk40QiQdWHVy7VOZO0ng5+qsWhDFnAq3/YdHE1L2b9DLDe 6cuCD4H6pSZpp8XQEJ5Ygp7Yg2mQ80Ce9l6iaNn0sfgb2gKF9Pqf4dufo0OTebfbYL2m 1eug== X-Gm-Message-State: AOAM531rVPcSdNJcXBp6OvvMiCgBkTw1stpqnfZLQ4s+c5jwPG4yljhP J9iFkQuT40+Q/KDzOL17bJf7/tlw2Yk= X-Google-Smtp-Source: ABdhPJwv3sQxxSANMG5IKeHhlp0a8TlHgMkcdUdy+ODo7/DLchUUD0l4lfRg1W9NZeaOZ3dIHbe7Og== X-Received: by 2002:a17:90a:bd0c:: with SMTP id y12mr11775248pjr.154.1606574502269; Sat, 28 Nov 2020 06:41:42 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:41 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 09/19] powerpc/64: context tracking move to interrupt wrappers Date: Sun, 29 Nov 2020 00:41:04 +1000 Message-Id: <20201128144114.944000-10-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This moves exception_enter/exit calls to wrapper functions for synchronous interrupts. More interrupt handlers are covered by this than previously. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 9 ++++ arch/powerpc/kernel/traps.c | 74 ++++++--------------------- arch/powerpc/mm/book3s64/hash_utils.c | 2 - arch/powerpc/mm/fault.c | 3 -- 4 files changed, 26 insertions(+), 62 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 2ab0675c27aa..e2410f54c6e9 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -7,10 +7,16 @@ #include struct interrupt_state { +#ifdef CONFIG_PPC64 + enum ctx_state ctx_state; +#endif }; static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { +#ifdef CONFIG_PPC64 + state->ctx_state = exception_enter(); +#endif } /* @@ -29,6 +35,9 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup */ static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) { +#ifdef CONFIG_PPC64 + exception_exit(state->ctx_state); +#endif } static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 8a6cb2838ce6..f40b45086b93 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -1108,41 +1108,28 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(handle_hmi_exception) DEFINE_INTERRUPT_HANDLER(unknown_exception) { - enum ctx_state prev_state = exception_enter(); - printk("Bad trap at PC: %lx, SR: %lx, vector=%lx\n", regs->nip, regs->msr, regs->trap); _exception(SIGTRAP, regs, TRAP_UNK, 0); - - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER_ASYNC(unknown_async_exception) { - enum ctx_state prev_state = exception_enter(); - printk("Bad trap at PC: %lx, SR: %lx, vector=%lx\n", regs->nip, regs->msr, regs->trap); _exception(SIGTRAP, regs, TRAP_UNK, 0); - - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(instruction_breakpoint_exception) { - enum ctx_state prev_state = exception_enter(); - if (notify_die(DIE_IABR_MATCH, "iabr_match", regs, 5, 5, SIGTRAP) == NOTIFY_STOP) - goto bail; + return; if (debugger_iabr_match(regs)) - goto bail; + return; _exception(SIGTRAP, regs, TRAP_BRKPT, regs->nip); - -bail: - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(RunModeException) @@ -1152,8 +1139,6 @@ DEFINE_INTERRUPT_HANDLER(RunModeException) DEFINE_INTERRUPT_HANDLER(single_step_exception) { - enum ctx_state prev_state = exception_enter(); - clear_single_step(regs); clear_br_trace(regs); @@ -1162,14 +1147,11 @@ DEFINE_INTERRUPT_HANDLER(single_step_exception) if (notify_die(DIE_SSTEP, "single_step", regs, 5, 5, SIGTRAP) == NOTIFY_STOP) - goto bail; + return; if (debugger_sstep(regs)) - goto bail; + return; _exception(SIGTRAP, regs, TRAP_TRACE, regs->nip); - -bail: - exception_exit(prev_state); } NOKPROBE_SYMBOL(single_step_exception); @@ -1495,7 +1477,6 @@ static inline int emulate_math(struct pt_regs *regs) { return -1; } DEFINE_INTERRUPT_HANDLER(program_check_exception) { - enum ctx_state prev_state = exception_enter(); unsigned int reason = get_reason(regs); /* We can now get here via a FP Unavailable exception if the core @@ -1504,22 +1485,22 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) if (reason & REASON_FP) { /* IEEE FP exception */ parse_fpe(regs); - goto bail; + return; } if (reason & REASON_TRAP) { unsigned long bugaddr; /* Debugger is first in line to stop recursive faults in * rcu_lock, notify_die, or atomic_notifier_call_chain */ if (debugger_bpt(regs)) - goto bail; + return; if (kprobe_handler(regs)) - goto bail; + return; /* trap exception */ if (notify_die(DIE_BPT, "breakpoint", regs, 5, 5, SIGTRAP) == NOTIFY_STOP) - goto bail; + return; bugaddr = regs->nip; /* @@ -1531,10 +1512,10 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) if (!(regs->msr & MSR_PR) && /* not user-mode */ report_bug(bugaddr, regs) == BUG_TRAP_TYPE_WARN) { regs->nip += 4; - goto bail; + return; } _exception(SIGTRAP, regs, TRAP_BRKPT, regs->nip); - goto bail; + return; } #ifdef CONFIG_PPC_TRANSACTIONAL_MEM if (reason & REASON_TM) { @@ -1555,7 +1536,7 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) */ if (user_mode(regs)) { _exception(SIGILL, regs, ILL_ILLOPN, regs->nip); - goto bail; + return; } else { printk(KERN_EMERG "Unexpected TM Bad Thing exception " "at %lx (msr 0x%lx) tm_scratch=%llx\n", @@ -1586,7 +1567,7 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) * pattern to occurrences etc. -dgibson 31/Mar/2003 */ if (!emulate_math(regs)) - goto bail; + return; /* Try to emulate it if we should. */ if (reason & (REASON_ILLEGAL | REASON_PRIVILEGED)) { @@ -1594,10 +1575,10 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) case 0: regs->nip += 4; emulate_single_step(regs); - goto bail; + return; case -EFAULT: _exception(SIGSEGV, regs, SEGV_MAPERR, regs->nip); - goto bail; + return; } } @@ -1606,9 +1587,6 @@ DEFINE_INTERRUPT_HANDLER(program_check_exception) _exception(SIGILL, regs, ILL_PRVOPC, regs->nip); else _exception(SIGILL, regs, ILL_ILLOPC, regs->nip); - -bail: - exception_exit(prev_state); } NOKPROBE_SYMBOL(program_check_exception); @@ -1625,14 +1603,12 @@ NOKPROBE_SYMBOL(emulation_assist_interrupt); DEFINE_INTERRUPT_HANDLER(alignment_exception) { - enum ctx_state prev_state = exception_enter(); int sig, code, fixed = 0; unsigned long reason; interrupt_cond_local_irq_enable(regs); reason = get_reason(regs); - if (reason & REASON_BOUNDARY) { sig = SIGBUS; code = BUS_ADRALN; @@ -1640,7 +1616,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception) } if (tm_abort_check(regs, TM_CAUSE_ALIGNMENT | TM_CAUSE_PERSISTENT)) - goto bail; + return; /* we don't implement logging of alignment exceptions */ if (!(current->thread.align_ctl & PR_UNALIGN_SIGBUS)) @@ -1650,7 +1626,7 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception) /* skip over emulated instruction */ regs->nip += inst_length(reason); emulate_single_step(regs); - goto bail; + return; } /* Operand address was bad */ @@ -1666,9 +1642,6 @@ DEFINE_INTERRUPT_HANDLER(alignment_exception) _exception(sig, regs, code, regs->dar); else bad_page_fault(regs, sig); - -bail: - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(StackOverflow) @@ -1682,41 +1655,28 @@ DEFINE_INTERRUPT_HANDLER(StackOverflow) DEFINE_INTERRUPT_HANDLER(stack_overflow_exception) { - enum ctx_state prev_state = exception_enter(); - die("Kernel stack overflow", regs, SIGSEGV); - - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(kernel_fp_unavailable_exception) { - enum ctx_state prev_state = exception_enter(); - printk(KERN_EMERG "Unrecoverable FP Unavailable Exception " "%lx at %lx\n", regs->trap, regs->nip); die("Unrecoverable FP Unavailable Exception", regs, SIGABRT); - - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(altivec_unavailable_exception) { - enum ctx_state prev_state = exception_enter(); - if (user_mode(regs)) { /* A user program has executed an altivec instruction, but this kernel doesn't support altivec. */ _exception(SIGILL, regs, ILL_ILLOPC, regs->nip); - goto bail; + return; } printk(KERN_EMERG "Unrecoverable VMX/Altivec Unavailable Exception " "%lx at %lx\n", regs->trap, regs->nip); die("Unrecoverable VMX/Altivec Unavailable Exception", regs, SIGABRT); - -bail: - exception_exit(prev_state); } DEFINE_INTERRUPT_HANDLER(vsx_unavailable_exception) diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c index a48c484b9e9b..39dcfaf7ba36 100644 --- a/arch/powerpc/mm/book3s64/hash_utils.c +++ b/arch/powerpc/mm/book3s64/hash_utils.c @@ -1289,7 +1289,6 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, unsigned long flags) { bool is_thp; - enum ctx_state prev_state = exception_enter(); pgd_t *pgdir; unsigned long vsid; pte_t *ptep; @@ -1489,7 +1488,6 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, DBG_LOW(" -> rc=%d\n", rc); bail: - exception_exit(prev_state); return rc; } EXPORT_SYMBOL_GPL(hash_page_mm); diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 4896f44c8374..2778fe0a41d5 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -550,7 +550,6 @@ NOKPROBE_SYMBOL(__do_page_fault); DEFINE_INTERRUPT_HANDLER_RET(do_page_fault) { - enum ctx_state prev_state = exception_enter(); unsigned long address = regs->dar; unsigned long error_code = regs->dsisr; long err; @@ -573,8 +572,6 @@ DEFINE_INTERRUPT_HANDLER_RET(do_page_fault) } #endif - exception_exit(prev_state); - return err; } NOKPROBE_SYMBOL(do_page_fault); From patchwork Sat Nov 28 14:41:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407538 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvmy3Kq6z9s1l for ; Sun, 29 Nov 2020 02:00:18 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=rB4hmXRJ; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvmx6qJmzDsS0 for ; Sun, 29 Nov 2020 02:00:17 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=rB4hmXRJ; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMZ29RwzDsRw for ; Sun, 29 Nov 2020 01:41:46 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id q10so7002569pfn.0 for ; Sat, 28 Nov 2020 06:41:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U3KknA0alFa0IGxG++tGNKRflaQXroTilKw3EbRtEOs=; b=rB4hmXRJBlp+cXRzy+y6WUVOjPK0eqMGilp8/6sqpyGa1kUQF/B3bqpcZd/t//GYdJ 8+SyHt7CZfXarnoOhb+gZbTWSGSxhZNN2WS2A4pyn1D5WSXx39JIHPQC1ZrkUOwLGFQ/ G312A8w5FOxAHdnMTOtTZPw9ooMcFNR423rIU7J1A4FyBMeripp93/GqKYrGs9m085E2 9II+fHf4EKJCH7d7iF5I0ylkF4/Duj2TI0UEssf0QqpgYiUCJLpPvqkfXT2z6BX6sdbh oCVB34HzRBw6au0iEybhqGn+AJeDKZqgFEZ8uhc0RrXGS3ksXDZxL0ZJ8JOS1wZkV2tu 1qfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U3KknA0alFa0IGxG++tGNKRflaQXroTilKw3EbRtEOs=; b=KAZxKBlWrOMqG8tpnLi9jk3ad+sMtKYG8e4JelrNIw8kWu6V4VnTp5FI1ECicRiu8F tdLkAjxs+5fkbTmhCNmXrDPvyrdKHvTskN9R9an7Pa1KIeXljCbIwBH90/h1wbyJSi8e Ej9AT69HwZul8IiG2tvwD179SeCuDe3t8AX8kssqdqeLT/xGRed3I1Zp5ubHLMvnDwKF 2Awr1Dq6wWHTnph3m8pTejjX+PX4W5fkz+6gO2ccOw+6o4y8T+hMCu0C0Gd3tZD3gssL A5qMqh+jqSygqT9ApQKLjpnnfYuOdDqz8Z5VfAnjk8z5tAzwQeN+E6YMpPpAL7lQxWkz iZhA== X-Gm-Message-State: AOAM531kvqxrmEJE4N06vS8JZe+poDAJ7y7MwzDnlZUF5olkqu0CoQvG jGUqp3VoGJsUaFLFHWktxcbP8J/eb5I= X-Google-Smtp-Source: ABdhPJw3ywDG/HAgl27vOpBDXMnYf+eKgEbBzoOwEy+n/M4ncekKKoYY4nAPqw6zKqZ08uakvzYdYg== X-Received: by 2002:a63:797:: with SMTP id 145mr10953398pgh.264.1606574504443; Sat, 28 Nov 2020 06:41:44 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:44 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 10/19] powerpc/64: add context tracking to asynchronous interrupts Date: Sun, 29 Nov 2020 00:41:05 +1000 Message-Id: <20201128144114.944000-11-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Previously context tracking was not done for asynchronous interrupts, (those that run in interrupt context), and if those would cause a reschedule when they exit, then scheduling functions (schedule_user, preempt_schedule_irq) call exception_enter/exit to fix this up and exit user context. This is a hack we would like to get away from, so do context tracking for asynchronous interrupts too. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index e2410f54c6e9..8f7db1c8cbe0 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -42,10 +42,12 @@ static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { + interrupt_enter_prepare(regs, state); } static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) { + interrupt_exit_prepare(regs, state); } struct interrupt_nmi_state { From patchwork Sat Nov 28 14:41:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407542 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvsJ42Dyz9s1l for ; Sun, 29 Nov 2020 02:04:04 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=CoZw0xUB; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvsJ3JQgzF1Pm for ; Sun, 29 Nov 2020 02:04:04 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=CoZw0xUB; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMc5GjyzDsGv for ; Sun, 29 Nov 2020 01:41:48 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id f17so6616108pge.6 for ; Sat, 28 Nov 2020 06:41:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KsqLyQTjSG+ewC2ayEMdc2tcUwZWrRPv7xdooRoyucU=; b=CoZw0xUB7zHdOvgjhgv7RBBLhM3m+AX0kNDaYXpCv8HCPCV5r+uxgYIn1hdnaXqs3E SG60qyY0Fk+6J/0bCjVxxxVMYfqwGZ/yubEjjO3z1Y6cIc5HKKOv2JpFSMhncGkRq3qJ SJziWEXAb4Bv6yTH4i5/08q8zM3w+EbFdWxET4d8ljMxDhc6C9RQU4GF5Mmm7Ks1vDXd GOdA0XVyVKGK66iqIB6w9+vqJZ3Ly4kh3f/IHJKsrfGx8aGMHCNyPc40gIUl4xOmobVS WQzhvDN3vw/MkFVb+st3jvMoaNNDIzK0eBxmso/bVAmKWSZzPErjXI3toK4haZ8/TC1z BgTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KsqLyQTjSG+ewC2ayEMdc2tcUwZWrRPv7xdooRoyucU=; b=jXAr+EGa3LyU9lBKgQ2X/EQChSZDUJ7T21TyD1YqyMhfsbdNMWSYItCCWOJ462dJTj DfI52G9EHa2fVKRovj4DgeAWZOzU+qDMirol7SOnOOGsIbFXMwlU18vPi/TE9URhB4iq 3zUrnEpcUTlNIY282PhBB5EquB+a4gqL2/g72UebuJgZFEyCzWMX8WnpAShdnKsC0FEN eaT1IBjb4QMtfKsK8N/WZd09kjMfsbA7BZaT4TbTpacjsIK7DjF9DNMy3blyQQsPdZYN 093GUBR9mBhxjRmsmwcsofVp3rgpYS3saSX1KsXhRTEGzwJfVr3As2E1ATkFt+8jj+Pl EHcQ== X-Gm-Message-State: AOAM533M1AByHNCyiVtG/dtJUoSYrdvjzammc688oxf0fjO4gCdi1kj5 k6CMM4TkTXkzVcbxyZR8V3CFkJDulQk= X-Google-Smtp-Source: ABdhPJwOP0Gfd//8+lV423K3s5zbGKUo4dmScLxqyh8doIVFSIlMQsNFElKJGq2IGZHk3fklLhAb6g== X-Received: by 2002:a17:90b:fd1:: with SMTP id gd17mr16317835pjb.148.1606574506636; Sat, 28 Nov 2020 06:41:46 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:46 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 11/19] powerpc: handle irq_enter/irq_exit in interrupt handler wrappers Date: Sun, 29 Nov 2020 00:41:06 +1000 Message-Id: <20201128144114.944000-12-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Move irq_enter/irq_exit into asynchronous interrupt handler wrappers. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 2 ++ arch/powerpc/kernel/dbell.c | 3 +-- arch/powerpc/kernel/irq.c | 4 ---- arch/powerpc/kernel/tau_6xx.c | 3 --- arch/powerpc/kernel/time.c | 4 ++-- arch/powerpc/kernel/traps.c | 6 ------ 6 files changed, 5 insertions(+), 17 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 8f7db1c8cbe0..21acf0d0154f 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -43,10 +43,12 @@ static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { interrupt_enter_prepare(regs, state); + irq_enter(); } static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) { + irq_exit(); interrupt_exit_prepare(regs, state); } diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c index c0f99f8ffa7d..84ee9c511459 100644 --- a/arch/powerpc/kernel/dbell.c +++ b/arch/powerpc/kernel/dbell.c @@ -22,7 +22,6 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(doorbell_exception) #ifdef CONFIG_SMP struct pt_regs *old_regs = set_irq_regs(regs); - irq_enter(); trace_doorbell_entry(regs); ppc_msgsync(); @@ -35,7 +34,7 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(doorbell_exception) smp_ipi_demux_relaxed(); /* already performed the barrier */ trace_doorbell_exit(regs); - irq_exit(); + set_irq_regs(old_regs); #else /* CONFIG_SMP */ printk(KERN_WARNING "Received doorbell on non-smp system\n"); diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c index 2055d204d08e..681abb7c0507 100644 --- a/arch/powerpc/kernel/irq.c +++ b/arch/powerpc/kernel/irq.c @@ -641,8 +641,6 @@ void __do_irq(struct pt_regs *regs) { unsigned int irq; - irq_enter(); - trace_irq_entry(regs); /* @@ -662,8 +660,6 @@ void __do_irq(struct pt_regs *regs) generic_handle_irq(irq); trace_irq_exit(regs); - - irq_exit(); } DEFINE_INTERRUPT_HANDLER_ASYNC(do_IRQ) diff --git a/arch/powerpc/kernel/tau_6xx.c b/arch/powerpc/kernel/tau_6xx.c index 46b2e5de4ef5..d864f07bab74 100644 --- a/arch/powerpc/kernel/tau_6xx.c +++ b/arch/powerpc/kernel/tau_6xx.c @@ -104,12 +104,9 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(TAUException) { int cpu = smp_processor_id(); - irq_enter(); tau[cpu].interrupts++; TAUupdate(cpu); - - irq_exit(); } #endif /* CONFIG_TAU_INT */ diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index 0089df57bb49..7dd0ab55e700 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -585,7 +585,7 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(timer_interrupt) #endif old_regs = set_irq_regs(regs); - irq_enter(); + trace_timer_interrupt_entry(regs); if (test_irq_work_pending()) { @@ -610,7 +610,7 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(timer_interrupt) } trace_timer_interrupt_exit(regs); - irq_exit(); + set_irq_regs(old_regs); } EXPORT_SYMBOL(timer_interrupt); diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index f40b45086b93..06b638a7f0b0 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -1082,7 +1082,6 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(handle_hmi_exception) struct pt_regs *old_regs; old_regs = set_irq_regs(regs); - irq_enter(); #ifdef CONFIG_VSX /* Real mode flagged P9 special emu is needed */ @@ -1102,7 +1101,6 @@ DEFINE_INTERRUPT_HANDLER_ASYNC(handle_hmi_exception) if (ppc_md.handle_hmi_exception) ppc_md.handle_hmi_exception(regs); - irq_exit(); set_irq_regs(old_regs); } @@ -1918,13 +1916,9 @@ DEFINE_INTERRUPT_HANDLER_NMI(performance_monitor_exception_nmi) DEFINE_INTERRUPT_HANDLER_ASYNC(performance_monitor_exception_async) { - irq_enter(); - __this_cpu_inc(irq_stat.pmu_irqs); perf_irq(regs); - - irq_exit(); } DEFINE_INTERRUPT_HANDLER_RAW(performance_monitor_exception) From patchwork Sat Nov 28 14:41:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407543 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvvK4hMwz9s1l for ; Sun, 29 Nov 2020 02:05:49 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=oswvTewl; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvvK4R0NzDrPw for ; Sun, 29 Nov 2020 02:05:49 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::542; helo=mail-pg1-x542.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=oswvTewl; dkim-atps=neutral Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMj5rmkzDsM9 for ; Sun, 29 Nov 2020 01:41:53 +1100 (AEDT) Received: by mail-pg1-x542.google.com with SMTP id 34so6599354pgp.10 for ; Sat, 28 Nov 2020 06:41:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oLnmutyJfFM4IdwVIFttl2OG+zX3XwPjUqRfqEGeS+g=; b=oswvTewlyFFDTmlh8BoKE+zqg+qKNOWLyrvHY3+mv2kALEtnPcLvJvs9Wv3RDrZcJR ph914jefvLP+83XmtcSZ/k7OUAJG1kNp3U7M1aKO3FQDYfu+MFzWj2B8YKJDZhyGjGz0 d82lVDUgb8WVPJN7Cv5HB5RZYccA5LjPDjXFTK+9kjQmqOyOT/QIl0JzSFdyWrmrccdv r3TSqk0pB3mQGWfhA2UsBGR/PMC53IZsVKguUCQUY4WiTw0LjVvS7MMuVhBtp4YIgH9K lh8Hf+oK4Ft1e9rS5TPZ9JydzxSpLGUlo9mirp9Iw12geLtvWE0W75b//mfr3OMrmR+Q Q3UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oLnmutyJfFM4IdwVIFttl2OG+zX3XwPjUqRfqEGeS+g=; b=Uczxuga/q4nfLXnpTn7OJWwVCo6afxRVdtgCYXmq/4Aenz1LsSy73NvPlsLHycRhyG wvzoHbGB54Ss8rRHjMqTuFhfjcOWuYS/aJg+1M7ReJ6YP8sKxr7o39L3rr7isFa/FDEt AeO1BWeOoGuNAVlFCJOOfOurPujmOYgCe5VVCTtiOV65xSF0E4tEZervACwLc1HW+80F OorPv6DV1fm8iAn7bmTbnnwj/zbmcVjDUrpjJx2fqz3WebGaQRKcjnq+u0u7ea9jyMr5 Xw0d27I7zTJvcvdK5uQkZi0/EbOnr7DWK1sMXbNfIOa4Y8iKw2czZVn1yE7JcBKCL2WG JecA== X-Gm-Message-State: AOAM5304NUZ1T5KjR1nLte0LBFKgrgQSRpl99FY6Y8g06Tf0yXi2+a8g fsHEtzfpSA1zZbPedhfOgRcRfj9FQ6g= X-Google-Smtp-Source: ABdhPJykMNDmj3bXm0MZATWWwDCNjUDOfk4KhvyUwnW++52byuKtjOi9a9qJvQlqnxIxR7Qj4VpvEQ== X-Received: by 2002:a63:d1b:: with SMTP id c27mr10711420pgl.25.1606574508823; Sat, 28 Nov 2020 06:41:48 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:48 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 12/19] powerpc/64s: move context tracking exit to interrupt exit path Date: Sun, 29 Nov 2020 00:41:07 +1000 Message-Id: <20201128144114.944000-13-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The interrupt handler wrapper functions are not the ideal place to maintain context tracking because after they return, the low level exit code must then determine if there are interrupts to replay, or if the task should be preempted, etc. Those paths (e.g., schedule_user) include their own exception_enter/exit pairs to fix this up but it's a bit hacky (see schedule_user() comments). Ideally context tracking will go to user mode only when there are no more interrupts or context switches or other exit processing work to handle. 64e can not do this because it does not use the C interrupt exit code. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 34 +++++++++++++++++++++++++--- arch/powerpc/kernel/syscall_64.c | 9 ++++++++ 2 files changed, 40 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 21acf0d0154f..542ad8e9c313 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -7,16 +7,30 @@ #include struct interrupt_state { -#ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3E_64 enum ctx_state ctx_state; #endif }; static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { -#ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3E_64 state->ctx_state = exception_enter(); #endif + +#ifdef CONFIG_PPC_BOOK3S_64 + if (user_mode(regs)) { + CT_WARN_ON(ct_state() != CONTEXT_USER); + user_exit_irqoff(); + } else { + /* + * CT_WARN_ON comes here via program_check_exception, + * so avoid recursion. + */ + if (TRAP(regs) != 0x700) + CT_WARN_ON(ct_state() != CONTEXT_KERNEL); + } +#endif } /* @@ -35,9 +49,23 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup */ static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt_state *state) { -#ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3E_64 exception_exit(state->ctx_state); #endif + + /* + * Book3S exits to user via interrupt_exit_user_prepare(), which does + * context tracking, which is a cleaner way to handle PREEMPT=y + * and avoid context entry/exit in e.g., preempt_schedule_irq()), + * which is likely to be where the core code wants to end up. + * + * The above comment explains why we can't do the + * + * if (user_mode(regs)) + * user_exit_irqoff(); + * + * sequence here. + */ } static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c index 9e855b25bd15..668a4557fc3f 100644 --- a/arch/powerpc/kernel/syscall_64.c +++ b/arch/powerpc/kernel/syscall_64.c @@ -275,6 +275,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned BUG_ON(!(regs->msr & MSR_PR)); BUG_ON(!FULL_REGS(regs)); BUG_ON(regs->softe != IRQS_ENABLED); + CT_WARN_ON(ct_state() == CONTEXT_USER); /* * We don't need to restore AMR on the way back to userspace for KUAP. @@ -317,7 +318,9 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned } } + user_enter_irqoff(); if (unlikely(!prep_irq_for_enabled_exit(true))) { + user_exit_irqoff(); local_irq_enable(); local_irq_disable(); goto again; @@ -358,6 +361,12 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign unrecoverable_exception(regs); BUG_ON(regs->msr & MSR_PR); BUG_ON(!FULL_REGS(regs)); + /* + * CT_WARN_ON comes here via program_check_exception, + * so avoid recursion. + */ + if (TRAP(regs) != 0x700) + CT_WARN_ON(ct_state() == CONTEXT_USER); amr = kuap_get_and_check_amr(); From patchwork Sat Nov 28 14:41:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407539 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjvqH40whz9s1l for ; Sun, 29 Nov 2020 02:02:19 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=N4O7tgQ8; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjvqH3Mj6zDrQt for ; Sun, 29 Nov 2020 02:02:19 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=N4O7tgQ8; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMj4F54zDsLx for ; Sun, 29 Nov 2020 01:41:53 +1100 (AEDT) Received: by mail-pf1-x442.google.com with SMTP id q10so7002721pfn.0 for ; Sat, 28 Nov 2020 06:41:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JJZxCyhduZmE4fi5z/tF73PdntvTsu7Wek2XFB59zGA=; b=N4O7tgQ8FP5nhglvfQiAUgapDFGbMw+51zWz53zMoVxxR1VHmImNhBXxq+aJizYC2o 3TasK1/ayzwr1IMych+5OKRYzH/mwPLIG4v7gphB0glP/bNWiNz4cNu9Ag3jyH171Xi1 pJBYQfG/66z137d3o0X5z/x+Si5N65B0223MCh+l8xwXLlCqoGA+MLENyWcj2yGHmlRX 1apwnkJhAd3CzdBGjBL+9aFovzdu+g402s7roKhl5Fywopc0RELQ13skNDaEbP74xnkr NB/5Wfx4QqZuegJSerzEC/IIhTXtSmtGQWTZcVzcz0B479RdcUJrKayOXws58LjE4WZL dlFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JJZxCyhduZmE4fi5z/tF73PdntvTsu7Wek2XFB59zGA=; b=nq8st4+B/9Jhu6+Hs4ZrR+4yIsvz7XzkXD9GnKSN6LSGbFcbkekSrR3wZO8+deDnMU nF2Jv6LEPJOA4d7VnDYYf9nfvOgJhveOE4nDcg5PrbxXrnueM90e7JFh+YLr1yStkeLi 83ytaWbJDiGyL7ygaHq9huDArfaEf29sxSEqYc4yeGomknbsjK8l0qS+tZnsmiQUGhdw reeCFYbLM9ExgrTjw4t6s7ZKPcl0P9ANITUNUfRSHQHplefjSkKMYpSgvjkFFA7Y0BLZ nO5tjYnFWBNSYoKab6L0qrzDLw75T+9urtgvfeYZAFj8ANd5Kl5yvlW+bNVZHJVmzn8t n9Cw== X-Gm-Message-State: AOAM533ySH6ItAgZ2qwL4ph4Zyaq4VrK9NnDuEnxsrs1BnmDJ2DaOCs+ gCvAt2AFkkBwQUSPEgDkR4S3omZvw6w= X-Google-Smtp-Source: ABdhPJyDTpLRP41a+xJsq7bOnlqjwjK7wqaz0j4PeKD0PeILnf2ECdRna6PqvwDJPb0NyGgwHv7big== X-Received: by 2002:a62:7b47:0:b029:197:cd7f:e044 with SMTP id w68-20020a627b470000b0290197cd7fe044mr11413301pfc.68.1606574511020; Sat, 28 Nov 2020 06:41:51 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:50 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 13/19] powerpc/64s: reconcile interrupts in C Date: Sun, 29 Nov 2020 00:41:08 +1000 Message-Id: <20201128144114.944000-14-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" There is no need for this to be in asm, use the new intrrupt entry wrapper. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 15 +++++++++++---- arch/powerpc/kernel/exceptions-64s.S | 26 -------------------------- 2 files changed, 11 insertions(+), 30 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 542ad8e9c313..c31fa26b3f60 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -14,11 +14,14 @@ struct interrupt_state { static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { -#ifdef CONFIG_PPC_BOOK3E_64 - state->ctx_state = exception_enter(); -#endif - + /* + * Book3E reconciles irq soft mask in asm + */ #ifdef CONFIG_PPC_BOOK3S_64 + if (irq_soft_mask_set_return(IRQS_ALL_DISABLED) == IRQS_ENABLED) + trace_hardirqs_off(); + local_paca->irq_happened |= PACA_IRQ_HARD_DIS; + if (user_mode(regs)) { CT_WARN_ON(ct_state() != CONTEXT_USER); user_exit_irqoff(); @@ -31,6 +34,10 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup CT_WARN_ON(ct_state() != CONTEXT_KERNEL); } #endif + +#ifdef CONFIG_PPC_BOOK3E_64 + state->ctx_state = exception_enter(); +#endif } /* diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index f6ec455db13c..647b7743bc9c 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -139,7 +139,6 @@ name: #define IKVM_VIRT .L_IKVM_VIRT_\name\() /* Virt entry tests KVM */ #define ISTACK .L_ISTACK_\name\() /* Set regular kernel stack */ #define __ISTACK(name) .L_ISTACK_ ## name -#define IRECONCILE .L_IRECONCILE_\name\() /* Do RECONCILE_IRQ_STATE */ #define IKUAP .L_IKUAP_\name\() /* Do KUAP lock */ #define INT_DEFINE_BEGIN(n) \ @@ -203,9 +202,6 @@ do_define_int n .ifndef ISTACK ISTACK=1 .endif - .ifndef IRECONCILE - IRECONCILE=1 - .endif .ifndef IKUAP IKUAP=1 .endif @@ -653,10 +649,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) .if ISTACK ACCOUNT_STOLEN_TIME .endif - - .if IRECONCILE - RECONCILE_IRQ_STATE(r10, r11) - .endif .endm /* @@ -935,7 +927,6 @@ INT_DEFINE_BEGIN(system_reset) */ ISET_RI=0 ISTACK=0 - IRECONCILE=0 IKVM_REAL=1 INT_DEFINE_END(system_reset) @@ -1123,7 +1114,6 @@ INT_DEFINE_BEGIN(machine_check_early) ISTACK=0 IDAR=1 IDSISR=1 - IRECONCILE=0 IKUAP=0 /* We don't touch AMR here, we never go to virtual mode */ INT_DEFINE_END(machine_check_early) @@ -1476,7 +1466,6 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) INT_DEFINE_BEGIN(data_access_slb) IVEC=0x380 IAREA=PACA_EXSLB - IRECONCILE=0 IDAR=1 IKVM_SKIP=1 IKVM_REAL=1 @@ -1503,7 +1492,6 @@ MMU_FTR_SECTION_ELSE li r3,-EFAULT ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) std r3,RESULT(r1) - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl do_bad_slb_fault b interrupt_return @@ -1565,7 +1553,6 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) INT_DEFINE_BEGIN(instruction_access_slb) IVEC=0x480 IAREA=PACA_EXSLB - IRECONCILE=0 IISIDE=1 IDAR=1 #ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE @@ -1594,7 +1581,6 @@ MMU_FTR_SECTION_ELSE li r3,-EFAULT ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) std r3,RESULT(r1) - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl do_bad_slb_fault b interrupt_return @@ -1754,7 +1740,6 @@ EXC_COMMON_BEGIN(program_check_common) */ INT_DEFINE_BEGIN(fp_unavailable) IVEC=0x800 - IRECONCILE=0 #ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE IKVM_REAL=1 #endif @@ -1769,7 +1754,6 @@ EXC_VIRT_END(fp_unavailable, 0x4800, 0x100) EXC_COMMON_BEGIN(fp_unavailable_common) GEN_COMMON fp_unavailable bne 1f /* if from user, just load it up */ - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl kernel_fp_unavailable_exception 0: trap @@ -1788,7 +1772,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_TM) b fast_interrupt_return #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl fp_unavailable_tm b interrupt_return @@ -1853,7 +1836,6 @@ INT_DEFINE_BEGIN(hdecrementer) IVEC=0x980 IHSRR=1 ISTACK=0 - IRECONCILE=0 IKVM_REAL=1 IKVM_VIRT=1 INT_DEFINE_END(hdecrementer) @@ -2227,7 +2209,6 @@ INT_DEFINE_BEGIN(hmi_exception_early) IHSRR=1 IREALMODE_COMMON=1 ISTACK=0 - IRECONCILE=0 IKUAP=0 /* We don't touch AMR here, we never go to virtual mode */ IKVM_REAL=1 INT_DEFINE_END(hmi_exception_early) @@ -2401,7 +2382,6 @@ EXC_COMMON_BEGIN(performance_monitor_common) */ INT_DEFINE_BEGIN(altivec_unavailable) IVEC=0xf20 - IRECONCILE=0 #ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE IKVM_REAL=1 #endif @@ -2431,7 +2411,6 @@ BEGIN_FTR_SECTION b fast_interrupt_return #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl altivec_unavailable_tm b interrupt_return @@ -2439,7 +2418,6 @@ BEGIN_FTR_SECTION 1: END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) #endif - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl altivec_unavailable_exception b interrupt_return @@ -2455,7 +2433,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_ALTIVEC) */ INT_DEFINE_BEGIN(vsx_unavailable) IVEC=0xf40 - IRECONCILE=0 #ifdef CONFIG_KVM_BOOK3S_PR_POSSIBLE IKVM_REAL=1 #endif @@ -2484,7 +2461,6 @@ BEGIN_FTR_SECTION b load_up_vsx #ifdef CONFIG_PPC_TRANSACTIONAL_MEM 2: /* User process was in a transaction */ - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl vsx_unavailable_tm b interrupt_return @@ -2492,7 +2468,6 @@ BEGIN_FTR_SECTION 1: END_FTR_SECTION_IFSET(CPU_FTR_VSX) #endif - RECONCILE_IRQ_STATE(r10, r11) addi r3,r1,STACK_FRAME_OVERHEAD bl vsx_unavailable_exception b interrupt_return @@ -2827,7 +2802,6 @@ EXC_VIRT_NONE(0x5800, 0x100) INT_DEFINE_BEGIN(soft_nmi) IVEC=0x900 ISTACK=0 - IRECONCILE=0 /* Soft-NMI may fire under local_irq_disable */ INT_DEFINE_END(soft_nmi) /* From patchwork Sat Nov 28 14:41:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407544 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvx5261mz9s1l for ; Sun, 29 Nov 2020 02:07:21 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=AvV+rP27; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvx51rDjzF110 for ; Sun, 29 Nov 2020 02:07:21 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=AvV+rP27; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMk6vgxzDsNP for ; Sun, 29 Nov 2020 01:41:54 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id w16so6603136pga.9 for ; Sat, 28 Nov 2020 06:41:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q1oxRS5FyyLJX4t9U5i9EoJMPiKHbvtMI0FvqNKIIcM=; b=AvV+rP27q9xJ2GyuJ8hOuvObTUy8eB7wRkg8FPkGxZEr5ImnnN2pdijfNmeXHkPX9P BH8YxUw4q65C00CHJZ16WsQQkQk98Vjp1ZBiXCgy9d3SCXUNedMjbjVop+BprBnzWsWX 8vO2rEQG26r8WbOgJy+1WsMojWd0VYglqMTntF3jM6acGFhGSVcwaN9Y6YexbHtAVnml 9/MtsDZBdEgDGsUlssNKYsPqT1AG13DVX8l4OfoeZon3iEHVdUXNhGMjuQb6zYO7QfKg ybQERm9T9IfJbqBQW+TH0o9pgrZP2zy9GwEtCH+ucSOIZhOpMw/wwRcphBrr/WCv1k2T OIHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q1oxRS5FyyLJX4t9U5i9EoJMPiKHbvtMI0FvqNKIIcM=; b=fBA6nvhYF3qYxdr2Q61GRVxOFg0Ensozsd1dUVT4OW+gsmfVnUFi6BX5OdH/Zk1n3I C1OD8u+lUmVLNmTNMpRCCIZlhJzIcxNl5TkNZZVsSKsJkYg0C+rprozpC/qjOMKWFDwr lQ59VcrC78NpjM/ecj28uuHogGgRdy0XfRic0Ot0uLMb7si3Ybt8jHn9dIUSHzZElLjn C+1dBSfpb+mTlSx23HtEQ7nrsSy/AKCD8xUbYJlF+XzKfcAN/3X3IzUvWoL9/jFFR4nG KN0HwDVPIWNTYRpwoaA7tCILUK516MSbdY58BOMqDkK39kHtYC4lyQB0oldlp/2FF2IB RWrg== X-Gm-Message-State: AOAM530c5xN2vfYo9isMIWayWpC6hPf/Gv235i6G9cNhgy5oiSZJVEv4 eBhlrId7m15j6vbUgxbZT+qM8A7si+4= X-Google-Smtp-Source: ABdhPJyqEgNf49sqNkfLx54MlJsiy1K7+FLF58N0D43zW5jghHFkgp4OoJhL6wxUkObxlV2kYFIyVg== X-Received: by 2002:a17:90a:5518:: with SMTP id b24mr16951346pji.138.1606574513157; Sat, 28 Nov 2020 06:41:53 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:52 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 14/19] powerpc/64: move account_stolen_time into its own function Date: Sun, 29 Nov 2020 00:41:09 +1000 Message-Id: <20201128144114.944000-15-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This will be used by interrupt entry as well. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/cputime.h | 15 +++++++++++++++ arch/powerpc/kernel/syscall_64.c | 10 +--------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/include/asm/cputime.h b/arch/powerpc/include/asm/cputime.h index ed75d1c318e3..3f61604e1fcf 100644 --- a/arch/powerpc/include/asm/cputime.h +++ b/arch/powerpc/include/asm/cputime.h @@ -87,6 +87,18 @@ static notrace inline void account_cpu_user_exit(void) acct->starttime_user = tb; } +static notrace inline void account_stolen_time(void) +{ +#ifdef CONFIG_PPC_SPLPAR + if (IS_ENABLED(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) && + firmware_has_feature(FW_FEATURE_SPLPAR)) { + struct lppaca *lp = local_paca->lppaca_ptr; + + if (unlikely(local_paca->dtl_ridx != be64_to_cpu(lp->dtl_idx))) + accumulate_stolen_time(); + } +#endif +} #endif /* __KERNEL__ */ #else /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ @@ -96,5 +108,8 @@ static inline void account_cpu_user_entry(void) static inline void account_cpu_user_exit(void) { } +static notrace inline void account_stolen_time(void) +{ +} #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ #endif /* __POWERPC_CPUTIME_H */ diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/syscall_64.c index 668a4557fc3f..8ab5e16d640c 100644 --- a/arch/powerpc/kernel/syscall_64.c +++ b/arch/powerpc/kernel/syscall_64.c @@ -45,15 +45,7 @@ notrace long system_call_exception(long r3, long r4, long r5, account_cpu_user_entry(); -#ifdef CONFIG_PPC_SPLPAR - if (IS_ENABLED(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) && - firmware_has_feature(FW_FEATURE_SPLPAR)) { - struct lppaca *lp = local_paca->lppaca_ptr; - - if (unlikely(local_paca->dtl_ridx != be64_to_cpu(lp->dtl_idx))) - accumulate_stolen_time(); - } -#endif + account_stolen_time(); /* * This is not required for the syscall exit path, but makes the From patchwork Sat Nov 28 14:41:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407545 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjvz92bp3z9s1l for ; Sun, 29 Nov 2020 02:09:09 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=cZZZSa+j; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjvz838hWzF1Rs for ; Sun, 29 Nov 2020 02:09:08 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::543; helo=mail-pg1-x543.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=cZZZSa+j; dkim-atps=neutral Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMp1HhpzDsPb for ; Sun, 29 Nov 2020 01:41:58 +1100 (AEDT) Received: by mail-pg1-x543.google.com with SMTP id w4so6594881pgg.13 for ; Sat, 28 Nov 2020 06:41:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MoHEdWT/5r4IEmXHnEbBhIw2fC3448bwrowA9zN0RfE=; b=cZZZSa+jd2cL2bQqMDb4RDPtkI3hb4KxosWMauxPukYe4yydDrT518JwBIjZOFgWXU DqZ+VMCVnxjC1uoIWv0RRCyv1H1CMwppPaMhgyrI3zBHOruw6JGM77VYVtQvDbvXD8Ps rmyg2E9ivSDUQB5xqu3MbEarX2njE3UUnmj4RK7ont5Ilx02GXMvXLE5d5WinQ5Owi8t E34Yf26n9bDTES7CyHI5D49s+f5IbhXxLByoWM5tr7aVcAfnjdmHz5OlhcsVKF1eqfSX 3MvNLjEa7jYEF+6cXfrCKEagHut62dQvazrcakFkgRotVXsTZBBCbHDAF2hyjk6UMcQu SSbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MoHEdWT/5r4IEmXHnEbBhIw2fC3448bwrowA9zN0RfE=; b=t7bAoJTcIj7xgBWQhUabh7P+/aDfjNuU0ZI7n7JCrTY7ki7haGtsPJR88rzqObBthw 0fkZ06VJxP+wyDScA1XhzryT94Ue9X4gyWFE/dFEqA74sJkMMd87ILgkhFuUc4wsfq3H maY82sEfUBJ0OspHkcMEIaD0sQPIDVLlsRsh8LdgotgO922A5SAiFnRrMjGlCpkEKDHV 34vGO75zw7rekrVs+7Rb95Mz6u34Xkls4AfFILeTUh5UX1adaHmo6KJieg79EOTHz+Pi apkJDmbHruOK1Zr9WhVyZw2uNLwqyfxG9qYYknP3hhqFQ1/xh2cH3AZwyEWUeijLuD4c fXxQ== X-Gm-Message-State: AOAM532jKaotXM79lj44NMKyjpOCADErJvXZf+ayIxKhAcQFzJAyWsO9 L6zXSUSYdRhA5zswslYBwpoSZ7BopOE= X-Google-Smtp-Source: ABdhPJxQnCmANOdtUfY/lheTca37NQbu3BbVOHZ81jvpZ/Z9G0hdC7XPJhXP0YrOzTfopbfcaiCebg== X-Received: by 2002:a62:7596:0:b029:197:de7a:b7a7 with SMTP id q144-20020a6275960000b0290197de7ab7a7mr11752137pfc.74.1606574515302; Sat, 28 Nov 2020 06:41:55 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:54 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 15/19] powerpc/64: entry cpu time accounting in C Date: Sun, 29 Nov 2020 00:41:10 +1000 Message-Id: <20201128144114.944000-16-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" There is no need for this to be in asm, use the new intrrupt entry wrapper. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 7 +++++++ arch/powerpc/include/asm/ppc_asm.h | 24 ------------------------ arch/powerpc/kernel/exceptions-64e.S | 1 - arch/powerpc/kernel/exceptions-64s.S | 5 ----- 4 files changed, 7 insertions(+), 30 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index c31fa26b3f60..26aa897d251b 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -4,6 +4,7 @@ #include #include +#include #include struct interrupt_state { @@ -25,6 +26,9 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup if (user_mode(regs)) { CT_WARN_ON(ct_state() != CONTEXT_USER); user_exit_irqoff(); + + account_cpu_user_entry(); + account_stolen_time(); } else { /* * CT_WARN_ON comes here via program_check_exception, @@ -38,6 +42,9 @@ static inline void interrupt_enter_prepare(struct pt_regs *regs, struct interrup #ifdef CONFIG_PPC_BOOK3E_64 state->ctx_state = exception_enter(); #endif + + if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64) && user_mode(regs)) + account_cpu_user_entry(); } /* diff --git a/arch/powerpc/include/asm/ppc_asm.h b/arch/powerpc/include/asm/ppc_asm.h index 511786f0e40d..c1199f6c75a3 100644 --- a/arch/powerpc/include/asm/ppc_asm.h +++ b/arch/powerpc/include/asm/ppc_asm.h @@ -25,7 +25,6 @@ #ifndef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE #define ACCOUNT_CPU_USER_ENTRY(ptr, ra, rb) #define ACCOUNT_CPU_USER_EXIT(ptr, ra, rb) -#define ACCOUNT_STOLEN_TIME #else #define ACCOUNT_CPU_USER_ENTRY(ptr, ra, rb) \ MFTB(ra); /* get timebase */ \ @@ -44,29 +43,6 @@ PPC_LL ra, ACCOUNT_SYSTEM_TIME(ptr); \ add ra,ra,rb; /* add on to system time */ \ PPC_STL ra, ACCOUNT_SYSTEM_TIME(ptr) - -#ifdef CONFIG_PPC_SPLPAR -#define ACCOUNT_STOLEN_TIME \ -BEGIN_FW_FTR_SECTION; \ - beq 33f; \ - /* from user - see if there are any DTL entries to process */ \ - ld r10,PACALPPACAPTR(r13); /* get ptr to VPA */ \ - ld r11,PACA_DTL_RIDX(r13); /* get log read index */ \ - addi r10,r10,LPPACA_DTLIDX; \ - LDX_BE r10,0,r10; /* get log write index */ \ - cmpd cr1,r11,r10; \ - beq+ cr1,33f; \ - bl accumulate_stolen_time; \ - ld r12,_MSR(r1); \ - andi. r10,r12,MSR_PR; /* Restore cr0 (coming from user) */ \ -33: \ -END_FW_FTR_SECTION_IFSET(FW_FEATURE_SPLPAR) - -#else /* CONFIG_PPC_SPLPAR */ -#define ACCOUNT_STOLEN_TIME - -#endif /* CONFIG_PPC_SPLPAR */ - #endif /* CONFIG_VIRT_CPU_ACCOUNTING_NATIVE */ /* diff --git a/arch/powerpc/kernel/exceptions-64e.S b/arch/powerpc/kernel/exceptions-64e.S index dc728bb1c89a..e6b5a362fc91 100644 --- a/arch/powerpc/kernel/exceptions-64e.S +++ b/arch/powerpc/kernel/exceptions-64e.S @@ -398,7 +398,6 @@ exc_##n##_common: \ std r10,_NIP(r1); /* save SRR0 to stackframe */ \ std r11,_MSR(r1); /* save SRR1 to stackframe */ \ beq 2f; /* if from kernel mode */ \ - ACCOUNT_CPU_USER_ENTRY(r13,r10,r11);/* accounting (uses cr0+eq) */ \ 2: ld r3,excf+EX_R10(r13); /* get back r10 */ \ ld r4,excf+EX_R11(r13); /* get back r11 */ \ mfspr r5,scratch; /* get back r13 */ \ diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 647b7743bc9c..2b447dd15d4c 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -577,7 +577,6 @@ DEFINE_FIXED_SYMBOL(\name\()_common_real) kuap_save_amr_and_lock r9, r10, cr1, cr0 .endif beq 101f /* if from kernel mode */ - ACCOUNT_CPU_USER_ENTRY(r13, r9, r10) BEGIN_FTR_SECTION ld r9,IAREA+EX_PPR(r13) /* Read PPR from paca */ std r9,_PPR(r1) @@ -645,10 +644,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) ld r11,exception_marker@toc(r2) std r10,RESULT(r1) /* clear regs->result */ std r11,STACK_FRAME_OVERHEAD-16(r1) /* mark the frame */ - - .if ISTACK - ACCOUNT_STOLEN_TIME - .endif .endm /* From patchwork Sat Nov 28 14:41:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407546 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjw146STzz9s1l for ; Sun, 29 Nov 2020 02:10:48 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=IsyJcth9; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjw1462BzzF1MY for ; Sun, 29 Nov 2020 02:10:48 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::434; helo=mail-pf1-x434.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=IsyJcth9; dkim-atps=neutral Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMs4Dr0zDsM4 for ; Sun, 29 Nov 2020 01:42:00 +1100 (AEDT) Received: by mail-pf1-x434.google.com with SMTP id s21so6950577pfu.13 for ; Sat, 28 Nov 2020 06:42:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dndwMTXUHkaEtFYV+rQ1khAGitI3MWaUDWkJLeJKS48=; b=IsyJcth9PMoG4rrkN12zXPujNNTe1/4TiyPYJUVAjIYzfKARQoKVn5cC7aC+bHWDCT m4ruRhf8GDAIGDU5SF1BDs0QhNcOxRopjvUhWaU6gXVuE5yi4AeLz5NS9EjVweR+RJwQ KMJ7CM5sRVmwfafJzPPyCHvTWNjU+jODUR3r2WUqWgpGv1L/pDZcr8I6nxLb6+GaDtY6 NH9oU+R2TSOwFdFiaSBTFU63hLAqo/MXOOszZEaJ35gN9hnXSMfdQeUoV6fQDghl1U2Q AK5NbstdDVF+a2CpE0HJZlXE5OlgayeGzPrDNNajcGpp+VfquXRlq532Xw1Ov38/jHfu ITNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dndwMTXUHkaEtFYV+rQ1khAGitI3MWaUDWkJLeJKS48=; b=QQLtbikRZbMpbB7CV3GVhzIyhK/xhYQoaUJhSFDJ+RHvBI7aGv4vqPKH1ddsfowny2 gCLp6RkzFeqQH00ynBjCghSrnA6JG0e0hbHanLQz0gT5Kr6JFoF2/QpFdCbo7pYEPYVg Bar6W6raWifOomHVsqX2A8fRGQXAnwLdeTDvf9j2BdVBsEDT4Sjwz6Ha0sa1aeGEC/oJ rHoWC0e58XEBltUuLEhBkDrxus64MAWDurZ4JNFTPpLUzeF1b+sb5ECiiYn1nGoYAJ+3 lc368Meu/y8+Nka3gpJnKeEyq0Vbrm2Pxc6udTc1L97UuQVzFzUSrevJUMoANl/FhujH JMYQ== X-Gm-Message-State: AOAM530OgSi3FiQMAYka9aelhaMmc0ty1jU7WqsSWGUIkXJFMpbiRt/m 4ZWbwM1ejSpN/vKWqYKHkzSKXGFx9dE= X-Google-Smtp-Source: ABdhPJxpETKTgkOQI6J1cA4WyFzDKk4O4w1bEnw07cPGytx8bcWgM7xWf7tN5Cg7CrRhwodSvky5AA== X-Received: by 2002:a17:90b:4595:: with SMTP id hd21mr16416079pjb.127.1606574517457; Sat, 28 Nov 2020 06:41:57 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:57 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 16/19] powerpc: move NMI entry/exit code into wrapper Date: Sun, 29 Nov 2020 00:41:11 +1000 Message-Id: <20201128144114.944000-17-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This moves the common NMI entry and exit code into the interrupt handler wrappers. This changes the behaviour of soft-NMI (watchdog) and HMI interrupts, and also MCE interrupts on 64e, by adding missing parts of the NMI entry to them. It fixes a bug with sreset on pseries HPT guests which shouldn't call nmi_enter(). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 24 ++++++++++++++++ arch/powerpc/kernel/mce.c | 11 -------- arch/powerpc/kernel/traps.c | 42 +++++----------------------- arch/powerpc/kernel/watchdog.c | 10 +++---- 4 files changed, 35 insertions(+), 52 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 26aa897d251b..8508b99d3455 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -95,14 +95,38 @@ static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct int } struct interrupt_nmi_state { +#ifdef CONFIG_PPC64 + u8 ftrace_enabled; +#endif }; static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct interrupt_nmi_state *state) { +#ifdef CONFIG_PPC64 + state->ftrace_enabled = this_cpu_get_ftrace_enabled(); + this_cpu_set_ftrace_enabled(0); +#endif + + /* + * Do not use nmi_enter() for pseries hash guest taking a real-mode + * NMI because not everything it touches is within the RMA limit. + */ + if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64) || + !firmware_has_feature(FW_FEATURE_LPAR) || + radix_enabled() || (mfmsr() & MSR_DR)) + nmi_enter(); } static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct interrupt_nmi_state *state) { + if (!IS_ENABLED(CONFIG_PPC_BOOK3S_64) || + !firmware_has_feature(FW_FEATURE_LPAR) || + radix_enabled() || (mfmsr() & MSR_DR)) + nmi_exit(); + +#ifdef CONFIG_PPC64 + this_cpu_set_ftrace_enabled(state->ftrace_enabled); +#endif } /** diff --git a/arch/powerpc/kernel/mce.c b/arch/powerpc/kernel/mce.c index b84459f45b1a..9f39deed4fca 100644 --- a/arch/powerpc/kernel/mce.c +++ b/arch/powerpc/kernel/mce.c @@ -592,12 +592,6 @@ EXPORT_SYMBOL_GPL(machine_check_print_event_info); DEFINE_INTERRUPT_HANDLER_NMI(machine_check_early) { long handled = 0; - u8 ftrace_enabled = this_cpu_get_ftrace_enabled(); - - this_cpu_set_ftrace_enabled(0); - /* Do not use nmi_enter/exit for pseries hpte guest */ - if (radix_enabled() || !firmware_has_feature(FW_FEATURE_LPAR)) - nmi_enter(); hv_nmi_check_nonrecoverable(regs); @@ -607,11 +601,6 @@ DEFINE_INTERRUPT_HANDLER_NMI(machine_check_early) if (ppc_md.machine_check_early) handled = ppc_md.machine_check_early(regs); - if (radix_enabled() || !firmware_has_feature(FW_FEATURE_LPAR)) - nmi_exit(); - - this_cpu_set_ftrace_enabled(ftrace_enabled); - return handled; } diff --git a/arch/powerpc/kernel/traps.c b/arch/powerpc/kernel/traps.c index 06b638a7f0b0..1b881ef5f07b 100644 --- a/arch/powerpc/kernel/traps.c +++ b/arch/powerpc/kernel/traps.c @@ -441,11 +441,6 @@ DEFINE_INTERRUPT_HANDLER_NMI(system_reset_exception) { unsigned long hsrr0, hsrr1; bool saved_hsrrs = false; - u8 ftrace_enabled = this_cpu_get_ftrace_enabled(); - - this_cpu_set_ftrace_enabled(0); - - nmi_enter(); /* * System reset can interrupt code where HSRRs are live and MSR[RI]=1. @@ -517,10 +512,6 @@ DEFINE_INTERRUPT_HANDLER_NMI(system_reset_exception) mtspr(SPRN_HSRR1, hsrr1); } - nmi_exit(); - - this_cpu_set_ftrace_enabled(ftrace_enabled); - /* What should we do here? We could issue a shutdown or hard reset. */ return 0; @@ -823,6 +814,12 @@ int machine_check_generic(struct pt_regs *regs) #endif /* everything else */ +/* + * BOOK3S_64 does not call this handler as a non-maskable interrupt + * (it uses its own early real-mode handler to handle the MCE proper + * and then raises irq_work to call this handler when interrupts are + * enabled). + */ #ifdef CONFIG_PPC_BOOK3S_64 DEFINE_INTERRUPT_HANDLER_ASYNC(machine_check_exception) #else @@ -831,20 +828,6 @@ DEFINE_INTERRUPT_HANDLER_NMI(machine_check_exception) { int recover = 0; - /* - * BOOK3S_64 does not call this handler as a non-maskable interrupt - * (it uses its own early real-mode handler to handle the MCE proper - * and then raises irq_work to call this handler when interrupts are - * enabled). - * - * This is silly. The BOOK3S_64 should just call a different function - * rather than expecting semantics to magically change. Something - * like 'non_nmi_machine_check_exception()', perhaps? - */ - const bool nmi = !IS_ENABLED(CONFIG_PPC_BOOK3S_64); - - if (nmi) nmi_enter(); - __this_cpu_inc(irq_stat.mce_exceptions); add_taint(TAINT_MACHINE_CHECK, LOCKDEP_NOW_UNRELIABLE); @@ -869,24 +852,17 @@ DEFINE_INTERRUPT_HANDLER_NMI(machine_check_exception) if (check_io_access(regs)) goto bail; - if (nmi) nmi_exit(); - die("Machine check", regs, SIGBUS); /* Must die if the interrupt is not recoverable */ if (!(regs->msr & MSR_RI)) die("Unrecoverable Machine check", regs, SIGBUS); -#ifdef CONFIG_PPC_BOOK3S_64 bail: +#ifdef CONFIG_PPC_BOOK3S_64 return; #else return 0; - -bail: - if (nmi) nmi_exit(); - - return 0; #endif } NOKPROBE_SYMBOL(machine_check_exception); @@ -1902,14 +1878,10 @@ DEFINE_INTERRUPT_HANDLER(vsx_unavailable_tm) #ifdef CONFIG_PPC64 DEFINE_INTERRUPT_HANDLER_NMI(performance_monitor_exception_nmi) { - nmi_enter(); - __this_cpu_inc(irq_stat.pmu_irqs); perf_irq(regs); - nmi_exit(); - return 0; } #endif diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c index 824b9376ac35..dc39534836a3 100644 --- a/arch/powerpc/kernel/watchdog.c +++ b/arch/powerpc/kernel/watchdog.c @@ -254,11 +254,12 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) int cpu = raw_smp_processor_id(); u64 tb; + /* should only arrive from kernel, with irqs disabled */ + WARN_ON_ONCE(!arch_irq_disabled_regs(regs)); + if (!cpumask_test_cpu(cpu, &wd_cpus_enabled)) return 0; - nmi_enter(); - __this_cpu_inc(irq_stat.soft_nmi_irqs); tb = get_tb(); @@ -266,7 +267,7 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) wd_smp_lock(&flags); if (cpumask_test_cpu(cpu, &wd_smp_cpus_stuck)) { wd_smp_unlock(&flags); - goto out; + return 0; } set_cpu_stuck(cpu, tb); @@ -290,9 +291,6 @@ DEFINE_INTERRUPT_HANDLER_NMI(soft_nmi_interrupt) if (wd_panic_timeout_tb < 0x7fffffff) mtspr(SPRN_DEC, wd_panic_timeout_tb); -out: - nmi_exit(); - return 0; } From patchwork Sat Nov 28 14:41:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407547 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjw2v2TMRz9s1l for ; Sun, 29 Nov 2020 02:12:23 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=i8KJZERf; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjw2v1c1TzF1nL for ; Sun, 29 Nov 2020 02:12:23 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::444; helo=mail-pf1-x444.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=i8KJZERf; dkim-atps=neutral Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMt3QjTzDsRF for ; Sun, 29 Nov 2020 01:42:02 +1100 (AEDT) Received: by mail-pf1-x444.google.com with SMTP id x24so6974691pfn.6 for ; Sat, 28 Nov 2020 06:42:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LsHRTuAm08jD6ZjvVu/iF5Wj1kbeclgOvmSVUI6pCO8=; b=i8KJZERfnmRxokosBSrKFc/yRFdrepzskJqS/tNArVH5VJIfDfLKOXVNfafkIF0EmR hU7NcZ1CKitlMHWfRjItUFG7yFnX/urfIQg85/vhEprvrqWhzs3JZkREI0fJ668pDflB qr/Nn7zvI38LO63jpr904XxQ3EYPMRxEzgzo9UQxtMqGAMSBlGJAJsNONnU7G4U0VJzR usNjzzPcIRf7w5MqAS9v08srZBp5LhlzAo7U4TmPmqpl+v/1gvSgheBKiY33kHQtSeWn 51d2sNV+jakCviTzJcQD+P0xa/YMyOAtTR4V7QWhxwGMGj1eGqvpZk4Pce9ccTX4lWJv 3yeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LsHRTuAm08jD6ZjvVu/iF5Wj1kbeclgOvmSVUI6pCO8=; b=PEkaLqMLeJtuD3hkUWov2a9Eq/hdV5z+kYBkMZJmCdA0xZu1UySTnir0x+YwuT85VU 9+o/Z7HqRTsQ9mql1SbGl9OVAtvgXAlgi9DlhlsIQIYPJO7sU07dxBRIxrBsQL7oUT1j um1ryaONcVGsYuJnb9a2Zzj/peLL7qekd78JzsPkpNZCVM4MYTIH/Sp4WufN2pmdNace +3lmb2q8p+ah0vX8H0tZz3EiXIDIy8608VMpoW4yCbD3UCEk64XCV8NUQM2l3wiuYi99 jgcH+V5s/hqZkDrNxzZtA5sbGGkc+uqqeoQtIur7U3gPEUN8Ho9KJGpRwK+TJBN4XsOI J43w== X-Gm-Message-State: AOAM533/V7+20e+fdrPQDFZAun2Bq6JPz2wBVDJ4LEaNHKYvCiYrxk30 MgrCA+5q4nZlkew0rFD7V5dhHpOMGZs= X-Google-Smtp-Source: ABdhPJxoo4PL8aYEjWnLsIti+9zK3w8WkPS1SKc8EavbBIgC0oR2p2DC7qYZxXJ4t7V8oiSb9R6xiQ== X-Received: by 2002:a63:540d:: with SMTP id i13mr10953533pgb.37.1606574519619; Sat, 28 Nov 2020 06:41:59 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.41.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:41:59 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 17/19] powerpc/64s: move NMI soft-mask handling to C Date: Sun, 29 Nov 2020 00:41:12 +1000 Message-Id: <20201128144114.944000-18-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Saving and restoring soft-mask state can now be done in C using the interrupt handler wrapper functions. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 26 ++++++++++++ arch/powerpc/kernel/exceptions-64s.S | 60 ---------------------------- 2 files changed, 26 insertions(+), 60 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 8508b99d3455..5da7b0de545e 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -96,6 +96,10 @@ static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct int struct interrupt_nmi_state { #ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3S_64 + u8 irq_soft_mask; + u8 irq_happened; +#endif u8 ftrace_enabled; #endif }; @@ -103,6 +107,21 @@ struct interrupt_nmi_state { static inline void interrupt_nmi_enter_prepare(struct pt_regs *regs, struct interrupt_nmi_state *state) { #ifdef CONFIG_PPC64 +#ifdef CONFIG_PPC_BOOK3S_64 + state->irq_soft_mask = local_paca->irq_soft_mask; + state->irq_happened = local_paca->irq_happened; + + /* + * Set IRQS_ALL_DISABLED unconditionally so irqs_disabled() does + * the right thing, and set IRQ_HARD_DIS. We do not want to reconcile + * because that goes through irq tracing which we don't want in NMI. + */ + local_paca->irq_soft_mask = IRQS_ALL_DISABLED; + local_paca->irq_happened |= PACA_IRQ_HARD_DIS; + + /* Don't do any per-CPU operations until interrupt state is fixed */ + state->ftrace_enabled = this_cpu_get_ftrace_enabled(); +#endif state->ftrace_enabled = this_cpu_get_ftrace_enabled(); this_cpu_set_ftrace_enabled(0); #endif @@ -126,6 +145,13 @@ static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct inter #ifdef CONFIG_PPC64 this_cpu_set_ftrace_enabled(state->ftrace_enabled); + +#ifdef CONFIG_PPC_BOOK3S_64 + /* Check we didn't change the pending interrupt mask. */ + WARN_ON_ONCE((state->irq_happened | PACA_IRQ_HARD_DIS) != local_paca->irq_happened); + local_paca->irq_happened = state->irq_happened; + local_paca->irq_soft_mask = state->irq_soft_mask; +#endif #endif } diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 2b447dd15d4c..b6d914db01da 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -1008,20 +1008,6 @@ EXC_COMMON_BEGIN(system_reset_common) ld r1,PACA_NMI_EMERG_SP(r13) subi r1,r1,INT_FRAME_SIZE __GEN_COMMON_BODY system_reset - /* - * Set IRQS_ALL_DISABLED unconditionally so irqs_disabled() does - * the right thing. We do not want to reconcile because that goes - * through irq tracing which we don't want in NMI. - * - * Save PACAIRQHAPPENED to RESULT (otherwise unused), and set HARD_DIS - * as we are running with MSR[EE]=0. - */ - li r10,IRQS_ALL_DISABLED - stb r10,PACAIRQSOFTMASK(r13) - lbz r10,PACAIRQHAPPENED(r13) - std r10,RESULT(r1) - ori r10,r10,PACA_IRQ_HARD_DIS - stb r10,PACAIRQHAPPENED(r13) addi r3,r1,STACK_FRAME_OVERHEAD bl system_reset_exception @@ -1037,14 +1023,6 @@ EXC_COMMON_BEGIN(system_reset_common) subi r10,r10,1 sth r10,PACA_IN_NMI(r13) - /* - * Restore soft mask settings. - */ - ld r10,RESULT(r1) - stb r10,PACAIRQHAPPENED(r13) - ld r10,SOFTE(r1) - stb r10,PACAIRQSOFTMASK(r13) - kuap_restore_amr r9, r10 EXCEPTION_RESTORE_REGS RFI_TO_USER_OR_KERNEL @@ -1190,30 +1168,11 @@ END_FTR_SECTION_IFSET(CPU_FTR_HVMODE) li r10,MSR_RI mtmsrd r10,1 - /* - * Set IRQS_ALL_DISABLED and save PACAIRQHAPPENED (see - * system_reset_common) - */ - li r10,IRQS_ALL_DISABLED - stb r10,PACAIRQSOFTMASK(r13) - lbz r10,PACAIRQHAPPENED(r13) - std r10,RESULT(r1) - ori r10,r10,PACA_IRQ_HARD_DIS - stb r10,PACAIRQHAPPENED(r13) - addi r3,r1,STACK_FRAME_OVERHEAD bl machine_check_early std r3,RESULT(r1) /* Save result */ ld r12,_MSR(r1) - /* - * Restore soft mask settings. - */ - ld r10,RESULT(r1) - stb r10,PACAIRQHAPPENED(r13) - ld r10,SOFTE(r1) - stb r10,PACAIRQSOFTMASK(r13) - #ifdef CONFIG_PPC_P7_NAP /* * Check if thread was in power saving mode. We come here when any @@ -2815,17 +2774,6 @@ EXC_COMMON_BEGIN(soft_nmi_common) subi r1,r1,INT_FRAME_SIZE __GEN_COMMON_BODY soft_nmi - /* - * Set IRQS_ALL_DISABLED and save PACAIRQHAPPENED (see - * system_reset_common) - */ - li r10,IRQS_ALL_DISABLED - stb r10,PACAIRQSOFTMASK(r13) - lbz r10,PACAIRQHAPPENED(r13) - std r10,RESULT(r1) - ori r10,r10,PACA_IRQ_HARD_DIS - stb r10,PACAIRQHAPPENED(r13) - addi r3,r1,STACK_FRAME_OVERHEAD bl soft_nmi_interrupt @@ -2833,14 +2781,6 @@ EXC_COMMON_BEGIN(soft_nmi_common) li r9,0 mtmsrd r9,1 - /* - * Restore soft mask settings. - */ - ld r10,RESULT(r1) - stb r10,PACAIRQHAPPENED(r13) - ld r10,SOFTE(r1) - stb r10,PACAIRQSOFTMASK(r13) - kuap_restore_amr r9, r10 EXCEPTION_RESTORE_REGS hsrr=0 RFI_TO_KERNEL From patchwork Sat Nov 28 14:41:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407548 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjw5C2YWGz9s1l for ; Sun, 29 Nov 2020 02:14:23 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XHzB7vKE; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjw5C1FPBzDrFK for ; Sun, 29 Nov 2020 02:14:23 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52c; helo=mail-pg1-x52c.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XHzB7vKE; dkim-atps=neutral Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvMx12nSzDsGv for ; Sun, 29 Nov 2020 01:42:04 +1100 (AEDT) Received: by mail-pg1-x52c.google.com with SMTP id o4so4347975pgj.0 for ; Sat, 28 Nov 2020 06:42:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HcLRlopNEj38dGRzACB8/RTMJQ/fNRqE1495D4e+KdA=; b=XHzB7vKE4AhPylbZdmFKg6o68OUDBJhzAEhoPEj9Wyz7SWj08OssmOTZq9oQZtiwqJ kCVJJcskO7dd8NIwAo6ITzniQDWVPbZLd+StvM9/RIoVA8zfm1r4ruSk5fhKdcbQ4kSw TfNg7llAOu5ieS4UtKYIPLZ0osPrvY3h/TQRRIEj3mm71oODxmAMV65v4kftC3GV1GY3 4SWpA2y0GC2PaPyivleFKbRBqn6w3gt3umOQWUCzovT9t9PXcfo0alMMAPebITeVBeAN U9Or7qyfyh0uE7s7tCbDQT6wxtQIZTsOI6b9dzFQnX3qN8mHbdfB0j5ZxvKGaKtOxt8B l5JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HcLRlopNEj38dGRzACB8/RTMJQ/fNRqE1495D4e+KdA=; b=CaLVhqhJfS7AJTy6s3FcRm1hVxqHq+1WsbIxucbm0dap1MSZYvdSL7PaJvKc7TF/5x rQ98eJC3+9qsQD5/c+sJ+43HjCPejYue1RVjXobwCzzBosujm3faIRO3jkg/DHAphGmc tSjMS8VRJnEp0tw7hTCj7OWYJLf/QX4Ri66Aug/RbYG41BGmVEvQsLqq09GI+VyVG4Y6 nM5PeOtelZI23tPeWlaXLXYPc1SVfJvo0lsMfRlCOe8/d/ELsXRgS8x3sfBQehT0AwcM +wsa0QeD8OTvjUBHqzGvRQ8Ws1B2i90YOH2UF9FgemEbWJMmALYN8OsiQzAwXr1oB4Jd FErA== X-Gm-Message-State: AOAM530FfYWN48LRxmrnzCwqVSrQmMiz9eq4Hk1tM90IzHH/kaFJ3Sni v8Wj4sojC1jNYMiWV1gBMH8LbkIO5SA= X-Google-Smtp-Source: ABdhPJwvi+hUl6DdYXCuMsr4CtZHXLveyRAz0w7wNcR4MEaFx6998FnU4eUEUPT9PAizZvp7Z1cyiw== X-Received: by 2002:a17:90b:2317:: with SMTP id mt23mr16597238pjb.2.1606574521960; Sat, 28 Nov 2020 06:42:01 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.42.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:42:01 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 18/19] powerpc/64s: runlatch interrupt handling in C Date: Sun, 29 Nov 2020 00:41:13 +1000 Message-Id: <20201128144114.944000-19-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" There is no need for this to be in asm, use the new intrrupt entry wrapper. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 7 +++++++ arch/powerpc/kernel/exceptions-64s.S | 18 ------------------ 2 files changed, 7 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 5da7b0de545e..655fb668e2da 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -6,6 +6,7 @@ #include #include #include +#include struct interrupt_state { #ifdef CONFIG_PPC_BOOK3E_64 @@ -84,6 +85,12 @@ static inline void interrupt_exit_prepare(struct pt_regs *regs, struct interrupt static inline void interrupt_async_enter_prepare(struct pt_regs *regs, struct interrupt_state *state) { +#ifdef CONFIG_PPC_BOOK3S_64 + if (cpu_has_feature(CPU_FTR_CTRL) && + !test_thread_local_flags(_TLF_RUNLATCH)) + __ppc64_runlatch_on(); +#endif + interrupt_enter_prepare(regs, state); irq_enter(); } diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index b6d914db01da..e75849a60578 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -692,14 +692,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) ld r1,GPR1(r1) .endm -#define RUNLATCH_ON \ -BEGIN_FTR_SECTION \ - ld r3, PACA_THREAD_INFO(r13); \ - ld r4,TI_LOCAL_FLAGS(r3); \ - andi. r0,r4,_TLF_RUNLATCH; \ - beql ppc64_runlatch_on_trampoline; \ -END_FTR_SECTION_IFSET(CPU_FTR_CTRL) - /* * When the idle code in power4_idle puts the CPU into NAP mode, * it has to do so in a loop, and relies on the external interrupt @@ -1582,7 +1574,6 @@ EXC_VIRT_END(hardware_interrupt, 0x4500, 0x100) EXC_COMMON_BEGIN(hardware_interrupt_common) GEN_COMMON hardware_interrupt FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD bl do_IRQ b interrupt_return @@ -1768,7 +1759,6 @@ EXC_VIRT_END(decrementer, 0x4900, 0x80) EXC_COMMON_BEGIN(decrementer_common) GEN_COMMON decrementer FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD bl timer_interrupt b interrupt_return @@ -1854,7 +1844,6 @@ EXC_VIRT_END(doorbell_super, 0x4a00, 0x100) EXC_COMMON_BEGIN(doorbell_super_common) GEN_COMMON doorbell_super FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception @@ -2209,7 +2198,6 @@ EXC_COMMON_BEGIN(hmi_exception_early_common) EXC_COMMON_BEGIN(hmi_exception_common) GEN_COMMON hmi_exception FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD bl handle_hmi_exception b interrupt_return @@ -2239,7 +2227,6 @@ EXC_VIRT_END(h_doorbell, 0x4e80, 0x20) EXC_COMMON_BEGIN(h_doorbell_common) GEN_COMMON h_doorbell FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception @@ -2273,7 +2260,6 @@ EXC_VIRT_END(h_virt_irq, 0x4ea0, 0x20) EXC_COMMON_BEGIN(h_virt_irq_common) GEN_COMMON h_virt_irq FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD bl do_IRQ b interrupt_return @@ -2320,7 +2306,6 @@ EXC_VIRT_END(performance_monitor, 0x4f00, 0x20) EXC_COMMON_BEGIN(performance_monitor_common) GEN_COMMON performance_monitor FINISH_NAP - RUNLATCH_ON addi r3,r1,STACK_FRAME_OVERHEAD bl performance_monitor_exception b interrupt_return @@ -3035,9 +3020,6 @@ kvmppc_skip_Hinterrupt: * come here. */ -EXC_COMMON_BEGIN(ppc64_runlatch_on_trampoline) - b __ppc64_runlatch_on - USE_FIXED_SECTION(virt_trampolines) /* * All code below __end_interrupts is treated as soft-masked. If From patchwork Sat Nov 28 14:41:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407549 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cjw7H1TMVz9sSf for ; Sun, 29 Nov 2020 02:16:11 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nxOneY8H; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4Cjw7G4WDVzDqDB for ; Sun, 29 Nov 2020 02:16:10 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::436; helo=mail-pf1-x436.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nxOneY8H; dkim-atps=neutral Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4CjvN032QczDsNP for ; Sun, 29 Nov 2020 01:42:08 +1100 (AEDT) Received: by mail-pf1-x436.google.com with SMTP id y7so6952205pfq.11 for ; Sat, 28 Nov 2020 06:42:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rzTIfP54Wrw9lEvJyOk7/S2QStgX/y+VUQ7mJoNy90g=; b=nxOneY8Hd8IHAjvkGFtiCblxOWqh9d3HH1Nn5453JeFrbJ/2/ex/0ACB1g42efQsmf skQbg/3pR+rTLThWjrnctd0fXnTodbwAcDO0nyffH3Jyfa98rlOMhRveCpvMNUzUFEP3 cFu4l2/uZkG17/IP6RNBDW0yaNioztQ3TofHAZttyIGIKx7TWJVkK82CkhBKkF6ccKV+ 86a/8fdzkqEoxYasUGbSeKnSQXjGEHEUhotOmD7DYU2Cfz3QcqHWaO7q9MRKcqZAuevu kXo0/qUjgLEQQfM/Va59isOOzPrXoua3r3rJcHaCNuMELUBvmeWJzZBlVUOreQPptH9v mxgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rzTIfP54Wrw9lEvJyOk7/S2QStgX/y+VUQ7mJoNy90g=; b=HFxo/s7/0+yfFQmbqeifrnBuFAOPCBqFID2tqyklf929EoVkm2BsGnIwTIIxmAPZbt R5CP8z7bx6PhPzPDR0kil+AcWCTcl4c6Gsschg2VJJgSr1t6nBJPK/nFx3yVz4xNjMrD LH7qSvneQZJ13f4lQim6YE6d96j8jcLSIs6OtKLym6Sryzfv6WarhC8/SfJ0dojm2aAB AyBKBsbxoHpiPGtRdqpOkr3TTL+bT5HnZHaIGxJwYy0uahYz7yyAOhOUAgzWFbv1tVYQ zVSn0wc4iYpfH/2keYYRLYGeZ1zS+7HkvbynqrbAbXgW3vZpewZVHzF3Vrj2x1uFQb53 28sg== X-Gm-Message-State: AOAM533bTs/oGBdr6gjGJnVXaHcFfGWNoFyBUnbc7SGePf2R1FTjZzlL Zxnp6k3ZiPFwx21K5JX7TVm1hYF/gMA= X-Google-Smtp-Source: ABdhPJxz+naZpzoeRjY81VzDaP7i3l0+omHVbQr9CNEdsWEm0UIQXg1pDZv/XxOKMZACDUfkTy6V7Q== X-Received: by 2002:a63:3145:: with SMTP id x66mr10550760pgx.179.1606574524173; Sat, 28 Nov 2020 06:42:04 -0800 (PST) Received: from bobo.ozlabs.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id y201sm10750764pfb.2.2020.11.28.06.42.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 06:42:03 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v3 19/19] powerpc/64s: power4 nap fixup in C Date: Sun, 29 Nov 2020 00:41:14 +1000 Message-Id: <20201128144114.944000-20-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128144114.944000-1-npiggin@gmail.com> References: <20201128144114.944000-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" There is no need for this to be in asm, use the new intrrupt entry wrapper. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/interrupt.h | 15 +++++++++ arch/powerpc/include/asm/processor.h | 1 + arch/powerpc/include/asm/thread_info.h | 6 ++++ arch/powerpc/kernel/exceptions-64s.S | 45 -------------------------- arch/powerpc/kernel/idle_book3s.S | 4 +++ 5 files changed, 26 insertions(+), 45 deletions(-) diff --git a/arch/powerpc/include/asm/interrupt.h b/arch/powerpc/include/asm/interrupt.h index 655fb668e2da..29afc7443d0f 100644 --- a/arch/powerpc/include/asm/interrupt.h +++ b/arch/powerpc/include/asm/interrupt.h @@ -8,6 +8,16 @@ #include #include +static inline void nap_adjust_return(struct pt_regs *regs) +{ +#ifdef CONFIG_PPC_970_NAP + if (unlikely(test_thread_local_flags(_TLF_NAPPING))) { + clear_thread_local_flags(_TLF_NAPPING); + regs->nip = (unsigned long)power4_idle_nap_return; + } +#endif +} + struct interrupt_state { #ifdef CONFIG_PPC_BOOK3E_64 enum ctx_state ctx_state; @@ -99,6 +109,9 @@ static inline void interrupt_async_exit_prepare(struct pt_regs *regs, struct int { irq_exit(); interrupt_exit_prepare(regs, state); + + /* Adjust at exit so the main handler sees the true NIA */ + nap_adjust_return(regs); } struct interrupt_nmi_state { @@ -150,6 +163,8 @@ static inline void interrupt_nmi_exit_prepare(struct pt_regs *regs, struct inter radix_enabled() || (mfmsr() & MSR_DR)) nmi_exit(); + nap_adjust_return(regs); + #ifdef CONFIG_PPC64 this_cpu_set_ftrace_enabled(state->ftrace_enabled); diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h index c61c859b51a8..d845850f75e2 100644 --- a/arch/powerpc/include/asm/processor.h +++ b/arch/powerpc/include/asm/processor.h @@ -418,6 +418,7 @@ extern unsigned long isa300_idle_stop_mayloss(unsigned long psscr_val); extern unsigned long isa206_idle_insn_mayloss(unsigned long type); #ifdef CONFIG_PPC_970_NAP extern void power4_idle_nap(void); +void power4_idle_nap_return(void); #endif extern unsigned long cpuidle_disable; diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h index c9443c16e5fb..e7ee220b88ea 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -151,6 +151,12 @@ void arch_setup_new_exec(void); #ifndef __ASSEMBLY__ +static inline void clear_thread_local_flags(unsigned int flags) +{ + struct thread_info *ti = current_thread_info(); + ti->local_flags &= ~flags; +} + static inline bool test_thread_local_flags(unsigned int flags) { struct thread_info *ti = current_thread_info(); diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index e75849a60578..869ad412517b 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -692,25 +692,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_CFAR) ld r1,GPR1(r1) .endm -/* - * When the idle code in power4_idle puts the CPU into NAP mode, - * it has to do so in a loop, and relies on the external interrupt - * and decrementer interrupt entry code to get it out of the loop. - * It sets the _TLF_NAPPING bit in current_thread_info()->local_flags - * to signal that it is in the loop and needs help to get out. - */ -#ifdef CONFIG_PPC_970_NAP -#define FINISH_NAP \ -BEGIN_FTR_SECTION \ - ld r11, PACA_THREAD_INFO(r13); \ - ld r9,TI_LOCAL_FLAGS(r11); \ - andi. r10,r9,_TLF_NAPPING; \ - bnel power4_fixup_nap; \ -END_FTR_SECTION_IFSET(CPU_FTR_CAN_NAP) -#else -#define FINISH_NAP -#endif - /* * There are a few constraints to be concerned with. * - Real mode exceptions code/data must be located at their physical location. @@ -1248,7 +1229,6 @@ EXC_COMMON_BEGIN(machine_check_common) */ GEN_COMMON machine_check - FINISH_NAP /* Enable MSR_RI when finished with PACA_EXMC */ li r10,MSR_RI mtmsrd r10,1 @@ -1573,7 +1553,6 @@ EXC_VIRT_BEGIN(hardware_interrupt, 0x4500, 0x100) EXC_VIRT_END(hardware_interrupt, 0x4500, 0x100) EXC_COMMON_BEGIN(hardware_interrupt_common) GEN_COMMON hardware_interrupt - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD bl do_IRQ b interrupt_return @@ -1758,7 +1737,6 @@ EXC_VIRT_BEGIN(decrementer, 0x4900, 0x80) EXC_VIRT_END(decrementer, 0x4900, 0x80) EXC_COMMON_BEGIN(decrementer_common) GEN_COMMON decrementer - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD bl timer_interrupt b interrupt_return @@ -1843,7 +1821,6 @@ EXC_VIRT_BEGIN(doorbell_super, 0x4a00, 0x100) EXC_VIRT_END(doorbell_super, 0x4a00, 0x100) EXC_COMMON_BEGIN(doorbell_super_common) GEN_COMMON doorbell_super - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception @@ -2197,7 +2174,6 @@ EXC_COMMON_BEGIN(hmi_exception_early_common) EXC_COMMON_BEGIN(hmi_exception_common) GEN_COMMON hmi_exception - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD bl handle_hmi_exception b interrupt_return @@ -2226,7 +2202,6 @@ EXC_VIRT_BEGIN(h_doorbell, 0x4e80, 0x20) EXC_VIRT_END(h_doorbell, 0x4e80, 0x20) EXC_COMMON_BEGIN(h_doorbell_common) GEN_COMMON h_doorbell - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD #ifdef CONFIG_PPC_DOORBELL bl doorbell_exception @@ -2259,7 +2234,6 @@ EXC_VIRT_BEGIN(h_virt_irq, 0x4ea0, 0x20) EXC_VIRT_END(h_virt_irq, 0x4ea0, 0x20) EXC_COMMON_BEGIN(h_virt_irq_common) GEN_COMMON h_virt_irq - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD bl do_IRQ b interrupt_return @@ -2305,7 +2279,6 @@ EXC_VIRT_BEGIN(performance_monitor, 0x4f00, 0x20) EXC_VIRT_END(performance_monitor, 0x4f00, 0x20) EXC_COMMON_BEGIN(performance_monitor_common) GEN_COMMON performance_monitor - FINISH_NAP addi r3,r1,STACK_FRAME_OVERHEAD bl performance_monitor_exception b interrupt_return @@ -3037,24 +3010,6 @@ USE_FIXED_SECTION(virt_trampolines) __end_interrupts: DEFINE_FIXED_SYMBOL(__end_interrupts) -#ifdef CONFIG_PPC_970_NAP - /* - * Called by exception entry code if _TLF_NAPPING was set, this clears - * the NAPPING flag, and redirects the exception exit to - * power4_fixup_nap_return. - */ - .globl power4_fixup_nap -EXC_COMMON_BEGIN(power4_fixup_nap) - andc r9,r9,r10 - std r9,TI_LOCAL_FLAGS(r11) - LOAD_REG_ADDR(r10, power4_idle_nap_return) - std r10,_NIP(r1) - blr - -power4_idle_nap_return: - blr -#endif - CLOSE_FIXED_SECTION(real_vectors); CLOSE_FIXED_SECTION(real_trampolines); CLOSE_FIXED_SECTION(virt_vectors); diff --git a/arch/powerpc/kernel/idle_book3s.S b/arch/powerpc/kernel/idle_book3s.S index 22f249b6f58d..27d2e6a72ec9 100644 --- a/arch/powerpc/kernel/idle_book3s.S +++ b/arch/powerpc/kernel/idle_book3s.S @@ -201,4 +201,8 @@ _GLOBAL(power4_idle_nap) mtmsrd r7 isync b 1b + + .globl power4_idle_nap_return +power4_idle_nap_return: + blr #endif