From patchwork Mon Aug 20 09:41:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 959549 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41v89p0wjRz9s89 for ; Mon, 20 Aug 2018 19:47:42 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="X9njpUeT"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41v89n6T2tzF1bR for ; Mon, 20 Aug 2018 19:47:41 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="X9njpUeT"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="X9njpUeT"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41v83Z3Z9tzF0gv for ; Mon, 20 Aug 2018 19:42:18 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id a4-v6so3341710plm.13 for ; Mon, 20 Aug 2018 02:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hRN76PfKGv5++xAvpLKe4vXO+ZiJWSPZG84qyzQWaPs=; b=X9njpUeTfAb33jCgPRWm9fPKSkaAmq6HIflxKWEqfxRkhz0hoVkQwSirNX75mA4gSs W39jx/gsMGsppqWUY5qZKRbJjx7HVn/s/GPUmQhzByHUOshpBPY6MoE7up5CBQByrZ5G t/F99y/IIZR0K71jYWaf7nIdxJIG8wDtwKBYq5qlFoZx0sEkKW673Csn3jeK4VHjZAnz a/KfVKrMXcCTbnxkO0RU5uVJaitLfrp6QjNPRkVxdmi5KLTEyTwkTjBZycoMO0vYszZc 05Z8kIDZDXNgDtVREsnqof+nAfwti6fKmVS5OKZ5YdqmdUrw+SwGDCRXl2GEBHkCQ6s8 owIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hRN76PfKGv5++xAvpLKe4vXO+ZiJWSPZG84qyzQWaPs=; b=gVVInP/5cjun3Hts4NfXywRT3w6N1B53wPZPzKf9BC+Nq+dKkKeB7pM/SNvSHyX42R B/o3yL3WW+jZcvDtf+csyWbl7BWw4rX39FMkC/ycmw/58P/67iumafiQU53ZTu+g4e/o hMi1v9OuKGU1aznPCuhybtbCntIuedOybWnUnbTbs9y4TZVdzisw8s5Tnrytmd/dMuxN xHQ/QQU6zFCsC+d3b4l6Whvx2v0lIToZQf8mZjuq9Mf6UTnVVDibMf+Iz/ESxBaxi2Jj ppDUumh0DrIlav+Qr21DcXa4P+UmeCFfdP5HEvSRbyS877YEqNqs4bbO2WlwyEiNTu1h Ya3Q== X-Gm-Message-State: AOUpUlGzlAe/JRzc5/t6wskazwtYzV2OBh3Yw0WN9q81NuCR+p43JkVo jBbGCOzf2IhSljiux0DYEHDefmuZ X-Google-Smtp-Source: AA+uWPzbwXq9CItKVme1c7SZn8Dj9Mm4CB6jQf7yvMXGzEmaCJwxoOHUggGSkTxhOeP4DATBoL02/A== X-Received: by 2002:a17:902:9b81:: with SMTP id y1-v6mr44335522plp.239.1534758136078; Mon, 20 Aug 2018 02:42:16 -0700 (PDT) Received: from roar.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id x87-v6sm14793281pfa.143.2018.08.20.02.42.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 02:42:15 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/5] powerpc/64s/hash: convert SLB miss handlers to C Date: Mon, 20 Aug 2018 19:41:56 +1000 Message-Id: <20180820094200.13003-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com> References: <20180820094200.13003-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This patch moves SLB miss handlers completely to C, using the standard exception handler macros to set up the stack and branch to C. This can be done because the segment containing the kernel stack is always bolted, so accessing it with relocation on will not cause an SLB exception. Arbitrary kernel memory may not be accessed when handling kernel space SLB misses, so care should be taken there. However user SLB misses can access any kernel memory, which can be used to move some fields out of the paca (in later patches). User SLB misses could quite easily reconcile IRQs and set up a first class kernel environment and exit via ret_from_except, however that doesn't seem to be necessary at the moment, so we only do that if a bad fault is encountered. [ Credit to Aneesh for bug fixes and improvements to bad address handling ] --- arch/powerpc/include/asm/asm-prototypes.h | 2 + arch/powerpc/kernel/exceptions-64s.S | 200 +++---------- arch/powerpc/mm/Makefile | 2 +- arch/powerpc/mm/slb.c | 237 +++++++-------- arch/powerpc/mm/slb_low.S | 338 ---------------------- 5 files changed, 166 insertions(+), 613 deletions(-) delete mode 100644 arch/powerpc/mm/slb_low.S diff --git a/arch/powerpc/include/asm/asm-prototypes.h b/arch/powerpc/include/asm/asm-prototypes.h index 1f4691ce4126..c330ed10074a 100644 --- a/arch/powerpc/include/asm/asm-prototypes.h +++ b/arch/powerpc/include/asm/asm-prototypes.h @@ -78,6 +78,8 @@ void kernel_bad_stack(struct pt_regs *regs); void system_reset_exception(struct pt_regs *regs); void machine_check_exception(struct pt_regs *regs); void emulation_assist_interrupt(struct pt_regs *regs); +long do_slb_fault(struct pt_regs *regs, unsigned long ea); +void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, unsigned long err); /* signals, syscalls and interrupts */ long sys_swapcontext(struct ucontext __user *old_ctx, diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S index 9dad73722d1a..f22ddb301661 100644 --- a/arch/powerpc/kernel/exceptions-64s.S +++ b/arch/powerpc/kernel/exceptions-64s.S @@ -567,28 +567,35 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) EXC_REAL_BEGIN(data_access_slb, 0x380, 0x80) - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x380) - mr r12,r3 /* save r3 */ - mfspr r3,SPRN_DAR - mfspr r11,SPRN_SRR1 - crset 4*cr6+eq - BRANCH_TO_COMMON(r10, slb_miss_common) +EXCEPTION_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, KVMTEST_PR, 0x380); EXC_REAL_END(data_access_slb, 0x380, 0x80) EXC_VIRT_BEGIN(data_access_slb, 0x4380, 0x80) - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x380) - mr r12,r3 /* save r3 */ - mfspr r3,SPRN_DAR - mfspr r11,SPRN_SRR1 - crset 4*cr6+eq - BRANCH_TO_COMMON(r10, slb_miss_common) +EXCEPTION_RELON_PROLOG(PACA_EXSLB, data_access_slb_common, EXC_STD, NOTEST, 0x380); EXC_VIRT_END(data_access_slb, 0x4380, 0x80) + TRAMP_KVM_SKIP(PACA_EXSLB, 0x380) +EXC_COMMON_BEGIN(data_access_slb_common) + mfspr r10,SPRN_DAR + std r10,PACA_EXSLB+EX_DAR(r13) + EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB) + ld r4,PACA_EXSLB+EX_DAR(r13) + std r4,_DAR(r1) + addi r3,r1,STACK_FRAME_OVERHEAD + bl do_slb_fault + cmpdi r3,0 + bne- 1f + b fast_exception_return +1: /* Error case */ + bl save_nvgprs + RECONCILE_IRQ_STATE(r10, r11) + ld r4,_DAR(r1) + mr r5,r3 + addi r3,r1,STACK_FRAME_OVERHEAD + bl do_bad_slb_fault + b ret_from_except + EXC_REAL(instruction_access, 0x400, 0x80) EXC_VIRT(instruction_access, 0x4400, 0x80, 0x400) @@ -611,160 +618,33 @@ ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_TYPE_RADIX) EXC_REAL_BEGIN(instruction_access_slb, 0x480, 0x80) - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, KVMTEST_PR, 0x480) - mr r12,r3 /* save r3 */ - mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ - mfspr r11,SPRN_SRR1 - crclr 4*cr6+eq - BRANCH_TO_COMMON(r10, slb_miss_common) +EXCEPTION_PROLOG(PACA_EXSLB, instruction_access_slb_common, EXC_STD, KVMTEST_PR, 0x480); EXC_REAL_END(instruction_access_slb, 0x480, 0x80) EXC_VIRT_BEGIN(instruction_access_slb, 0x4480, 0x80) - SET_SCRATCH0(r13) - EXCEPTION_PROLOG_0(PACA_EXSLB) - EXCEPTION_PROLOG_1(PACA_EXSLB, NOTEST, 0x480) - mr r12,r3 /* save r3 */ - mfspr r3,SPRN_SRR0 /* SRR0 is faulting address */ - mfspr r11,SPRN_SRR1 - crclr 4*cr6+eq - BRANCH_TO_COMMON(r10, slb_miss_common) +EXCEPTION_RELON_PROLOG(PACA_EXSLB, instruction_access_slb_common, EXC_STD, NOTEST, 0x480); EXC_VIRT_END(instruction_access_slb, 0x4480, 0x80) -TRAMP_KVM(PACA_EXSLB, 0x480) - - -/* - * This handler is used by the 0x380 and 0x480 SLB miss interrupts, as well as - * the virtual mode 0x4380 and 0x4480 interrupts if AIL is enabled. - */ -EXC_COMMON_BEGIN(slb_miss_common) - /* - * r13 points to the PACA, r9 contains the saved CR, - * r12 contains the saved r3, - * r11 contain the saved SRR1, SRR0 is still ready for return - * r3 has the faulting address - * r9 - r13 are saved in paca->exslb. - * cr6.eq is set for a D-SLB miss, clear for a I-SLB miss - * We assume we aren't going to take any exceptions during this - * procedure. - */ - mflr r10 - stw r9,PACA_EXSLB+EX_CCR(r13) /* save CR in exc. frame */ - std r10,PACA_EXSLB+EX_LR(r13) /* save LR */ - - andi. r9,r11,MSR_PR // Check for exception from userspace - cmpdi cr4,r9,MSR_PR // And save the result in CR4 for later - - /* - * Test MSR_RI before calling slb_allocate_realmode, because the - * MSR in r11 gets clobbered. However we still want to allocate - * SLB in case MSR_RI=0, to minimise the risk of getting stuck in - * recursive SLB faults. So use cr5 for this, which is preserved. - */ - andi. r11,r11,MSR_RI /* check for unrecoverable exception */ - cmpdi cr5,r11,MSR_RI - - crset 4*cr0+eq -#ifdef CONFIG_PPC_BOOK3S_64 -BEGIN_MMU_FTR_SECTION - bl slb_allocate -END_MMU_FTR_SECTION_IFCLR(MMU_FTR_TYPE_RADIX) -#endif - - ld r10,PACA_EXSLB+EX_LR(r13) - lwz r9,PACA_EXSLB+EX_CCR(r13) /* get saved CR */ - mtlr r10 - - /* - * Large address, check whether we have to allocate new contexts. - */ - beq- 8f - - bne- cr5,2f /* if unrecoverable exception, oops */ - - /* All done -- return from exception. */ - - bne cr4,1f /* returning to kernel */ - - mtcrf 0x80,r9 - mtcrf 0x08,r9 /* MSR[PR] indication is in cr4 */ - mtcrf 0x04,r9 /* MSR[RI] indication is in cr5 */ - mtcrf 0x02,r9 /* I/D indication is in cr6 */ - mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */ - - RESTORE_CTR(r9, PACA_EXSLB) - RESTORE_PPR_PACA(PACA_EXSLB, r9) - mr r3,r12 - ld r9,PACA_EXSLB+EX_R9(r13) - ld r10,PACA_EXSLB+EX_R10(r13) - ld r11,PACA_EXSLB+EX_R11(r13) - ld r12,PACA_EXSLB+EX_R12(r13) - ld r13,PACA_EXSLB+EX_R13(r13) - RFI_TO_USER - b . /* prevent speculative execution */ -1: - mtcrf 0x80,r9 - mtcrf 0x08,r9 /* MSR[PR] indication is in cr4 */ - mtcrf 0x04,r9 /* MSR[RI] indication is in cr5 */ - mtcrf 0x02,r9 /* I/D indication is in cr6 */ - mtcrf 0x01,r9 /* slb_allocate uses cr0 and cr7 */ - - RESTORE_CTR(r9, PACA_EXSLB) - RESTORE_PPR_PACA(PACA_EXSLB, r9) - mr r3,r12 - ld r9,PACA_EXSLB+EX_R9(r13) - ld r10,PACA_EXSLB+EX_R10(r13) - ld r11,PACA_EXSLB+EX_R11(r13) - ld r12,PACA_EXSLB+EX_R12(r13) - ld r13,PACA_EXSLB+EX_R13(r13) - RFI_TO_KERNEL - b . /* prevent speculative execution */ - - -2: std r3,PACA_EXSLB+EX_DAR(r13) - mr r3,r12 - mfspr r11,SPRN_SRR0 - mfspr r12,SPRN_SRR1 - LOAD_HANDLER(r10,unrecov_slb) - mtspr SPRN_SRR0,r10 - ld r10,PACAKMSR(r13) - mtspr SPRN_SRR1,r10 - RFI_TO_KERNEL - b . -8: std r3,PACA_EXSLB+EX_DAR(r13) - mr r3,r12 - mfspr r11,SPRN_SRR0 - mfspr r12,SPRN_SRR1 - LOAD_HANDLER(r10, large_addr_slb) - mtspr SPRN_SRR0,r10 - ld r10,PACAKMSR(r13) - mtspr SPRN_SRR1,r10 - RFI_TO_KERNEL - b . +TRAMP_KVM(PACA_EXSLB, 0x480) -EXC_COMMON_BEGIN(unrecov_slb) - EXCEPTION_PROLOG_COMMON(0x4100, PACA_EXSLB) - RECONCILE_IRQ_STATE(r10, r11) +EXC_COMMON_BEGIN(instruction_access_slb_common) + EXCEPTION_PROLOG_COMMON(0x480, PACA_EXSLB) + ld r4,_NIP(r1) + addi r3,r1,STACK_FRAME_OVERHEAD + bl do_slb_fault + cmpdi r3,0 + bne- 1f + b fast_exception_return +1: /* Error case */ bl save_nvgprs -1: addi r3,r1,STACK_FRAME_OVERHEAD - bl unrecoverable_exception - b 1b - -EXC_COMMON_BEGIN(large_addr_slb) - EXCEPTION_PROLOG_COMMON(0x380, PACA_EXSLB) RECONCILE_IRQ_STATE(r10, r11) - ld r3, PACA_EXSLB+EX_DAR(r13) - std r3, _DAR(r1) - beq cr6, 2f - li r10, 0x481 /* fix trap number for I-SLB miss */ - std r10, _TRAP(r1) -2: bl save_nvgprs - addi r3, r1, STACK_FRAME_OVERHEAD - bl slb_miss_large_addr + ld r4,_NIP(r1) + mr r5,r3 + addi r3,r1,STACK_FRAME_OVERHEAD + bl do_bad_slb_fault b ret_from_except + EXC_REAL_BEGIN(hardware_interrupt, 0x500, 0x100) .globl hardware_interrupt_hv; hardware_interrupt_hv: diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile index cdf6a9960046..892d4e061d62 100644 --- a/arch/powerpc/mm/Makefile +++ b/arch/powerpc/mm/Makefile @@ -15,7 +15,7 @@ obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \ obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o hash64-$(CONFIG_PPC_NATIVE) := hash_native_64.o obj-$(CONFIG_PPC_BOOK3E_64) += pgtable-book3e.o -obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb_low.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o +obj-$(CONFIG_PPC_BOOK3S_64) += pgtable-hash64.o hash_utils_64.o slb.o $(hash64-y) mmu_context_book3s64.o pgtable-book3s64.o obj-$(CONFIG_PPC_RADIX_MMU) += pgtable-radix.o tlb-radix.o obj-$(CONFIG_PPC_STD_MMU_32) += ppc_mmu_32.o hash_low_32.o mmu_context_hash32.o obj-$(CONFIG_PPC_STD_MMU) += tlb_hash$(BITS).o diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index a3a6e04296cd..5643879fa2b1 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -34,7 +34,7 @@ enum slb_index { KSTACK_INDEX = 2, /* Kernel stack map */ }; -extern void slb_allocate(unsigned long ea); +static long slb_allocate_user(struct mm_struct *mm, unsigned long ea); #define slb_esid_mask(ssize) \ (((ssize) == MMU_SEGSIZE_256M)? ESID_MASK: ESID_MASK_1T) @@ -287,49 +287,19 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm) is_kernel_addr(exec_base)) return; - slb_allocate(pc); + slb_allocate_user(mm, pc); if (!esids_match(pc, stack)) - slb_allocate(stack); + slb_allocate_user(mm, stack); if (!esids_match(pc, exec_base) && !esids_match(stack, exec_base)) - slb_allocate(exec_base); + slb_allocate_user(mm, exec_base); } -static inline void patch_slb_encoding(unsigned int *insn_addr, - unsigned int immed) -{ - - /* - * This function patches either an li or a cmpldi instruction with - * a new immediate value. This relies on the fact that both li - * (which is actually addi) and cmpldi both take a 16-bit immediate - * value, and it is situated in the same location in the instruction, - * ie. bits 16-31 (Big endian bit order) or the lower 16 bits. - * The signedness of the immediate operand differs between the two - * instructions however this code is only ever patching a small value, - * much less than 1 << 15, so we can get away with it. - * To patch the value we read the existing instruction, clear the - * immediate value, and or in our new value, then write the instruction - * back. - */ - unsigned int insn = (*insn_addr & 0xffff0000) | immed; - patch_instruction(insn_addr, insn); -} - -extern u32 slb_miss_kernel_load_linear[]; -extern u32 slb_miss_kernel_load_io[]; -extern u32 slb_compare_rr_to_size[]; -extern u32 slb_miss_kernel_load_vmemmap[]; - void slb_set_size(u16 size) { - if (mmu_slb_size == size) - return; - mmu_slb_size = size; - patch_slb_encoding(slb_compare_rr_to_size, mmu_slb_size); } void slb_initialize(void) @@ -351,19 +321,9 @@ void slb_initialize(void) #endif if (!slb_encoding_inited) { slb_encoding_inited = 1; - patch_slb_encoding(slb_miss_kernel_load_linear, - SLB_VSID_KERNEL | linear_llp); - patch_slb_encoding(slb_miss_kernel_load_io, - SLB_VSID_KERNEL | io_llp); - patch_slb_encoding(slb_compare_rr_to_size, - mmu_slb_size); - pr_devel("SLB: linear LLP = %04lx\n", linear_llp); pr_devel("SLB: io LLP = %04lx\n", io_llp); - #ifdef CONFIG_SPARSEMEM_VMEMMAP - patch_slb_encoding(slb_miss_kernel_load_vmemmap, - SLB_VSID_KERNEL | vmemmap_llp); pr_devel("SLB: vmemmap LLP = %04lx\n", vmemmap_llp); #endif } @@ -394,49 +354,10 @@ void slb_initialize(void) asm volatile("isync":::"memory"); } -static void insert_slb_entry(unsigned long vsid, unsigned long ea, - int bpsize, int ssize) +static void slb_cache_update(unsigned long esid_data) { - unsigned long flags, vsid_data, esid_data; - enum slb_index index; int slb_cache_index; - /* - * We are irq disabled, hence should be safe to access PACA. - */ - VM_WARN_ON(!irqs_disabled()); - - /* - * We can't take a PMU exception in the following code, so hard - * disable interrupts. - */ - hard_irq_disable(); - - index = get_paca()->stab_rr; - - /* - * simple round-robin replacement of slb starting at SLB_NUM_BOLTED. - */ - if (index < (mmu_slb_size - 1)) - index++; - else - index = SLB_NUM_BOLTED; - - get_paca()->stab_rr = index; - - flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp; - vsid_data = (vsid << slb_vsid_shift(ssize)) | flags | - ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT); - esid_data = mk_esid_data(ea, ssize, index); - - /* - * No need for an isync before or after this slbmte. The exception - * we enter with and the rfid we exit with are context synchronizing. - * Also we only handle user segments here. - */ - asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data) - : "memory"); - if (cpu_has_feature(CPU_FTR_ARCH_300)) return; /* ISAv3.0B and later does not use slb_cache */ @@ -461,58 +382,146 @@ static void insert_slb_entry(unsigned long vsid, unsigned long ea, } } -static void handle_multi_context_slb_miss(int context_id, unsigned long ea) +static enum slb_index alloc_slb_index(void) +{ + enum slb_index index; + + /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */ + index = get_paca()->stab_rr; + if (index < (mmu_slb_size - 1)) + index++; + else + index = SLB_NUM_BOLTED; + get_paca()->stab_rr = index; + + return index; +} + +static void slb_insert_entry(unsigned long ea, unsigned long context, + unsigned long flags, int ssize) { - struct mm_struct *mm = current->mm; unsigned long vsid; - int bpsize; + unsigned long vsid_data, esid_data; + enum slb_index index = alloc_slb_index(); + + vsid = get_vsid(context, ea, ssize); + vsid_data = (vsid << slb_vsid_shift(ssize)) | flags | + ((unsigned long) ssize << SLB_VSID_SSIZE_SHIFT); + esid_data = mk_esid_data(ea, ssize, index); /* - * We are always above 1TB, hence use high user segment size. + * No need for an isync before or after this slbmte. The exception + * we enter with and the rfid we exit with are context synchronizing. + * Also we only handle user segments here. */ - vsid = get_vsid(context_id, ea, mmu_highuser_ssize); - bpsize = get_slice_psize(mm, ea); - insert_slb_entry(vsid, ea, bpsize, mmu_highuser_ssize); + asm volatile("slbmte %0, %1" : : "r" (vsid_data), "r" (esid_data)); + + slb_cache_update(esid_data); } -void slb_miss_large_addr(struct pt_regs *regs) +static long slb_allocate_kernel(unsigned long ea, unsigned long id) { - enum ctx_state prev_state = exception_enter(); - unsigned long ea = regs->dar; - int context; + unsigned long context; + unsigned long flags; + int ssize; - if (REGION_ID(ea) != USER_REGION_ID) - goto slb_bad_addr; + if ((ea & ~REGION_MASK) >= (1ULL << MAX_EA_BITS_PER_CONTEXT)) + return -EFAULT; - /* - * Are we beyound what the page table layout supports ? - */ - if ((ea & ~REGION_MASK) >= H_PGTABLE_RANGE) - goto slb_bad_addr; + if (id == KERNEL_REGION_ID) { + flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_linear_psize].sllp; +#ifdef CONFIG_SPARSEMEM_VMEMMAP + } else if (id == VMEMMAP_REGION_ID) { + flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_vmemmap_psize].sllp; +#endif + } else { + if (ea < H_VMALLOC_END) + flags = get_paca()->vmalloc_sllp; + else + flags = SLB_VSID_KERNEL | mmu_psize_defs[mmu_io_psize].sllp; + } + ssize = MMU_SEGSIZE_1T; + if (!mmu_has_feature(MMU_FTR_1T_SEGMENT)) + ssize = MMU_SEGSIZE_256M; + + context = id - KERNEL_REGION_CONTEXT_OFFSET; - /* Lower address should have been handled by asm code */ - if (ea < (1UL << MAX_EA_BITS_PER_CONTEXT)) - goto slb_bad_addr; + slb_insert_entry(ea, context, flags, ssize); + + return 0; +} + +static long slb_allocate_user(struct mm_struct *mm, unsigned long ea) +{ + unsigned long context; + unsigned long flags; + int bpsize; + int ssize; /* * consider this as bad access if we take a SLB miss * on an address above addr limit. */ - if (ea >= current->mm->context.slb_addr_limit) - goto slb_bad_addr; + if (ea >= mm->context.slb_addr_limit) + return -EFAULT; - context = get_ea_context(¤t->mm->context, ea); + context = get_ea_context(&mm->context, ea); if (!context) - goto slb_bad_addr; + return -EFAULT; + + if (unlikely(ea >= H_PGTABLE_RANGE)) { + WARN_ON(1); + return -EFAULT; + } + + ssize = user_segment_size(ea); + + bpsize = get_slice_psize(mm, ea); + flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp; + + slb_insert_entry(ea, context, flags, ssize); + + return 0; +} + +long do_slb_fault(struct pt_regs *regs, unsigned long ea) +{ + unsigned long id = REGION_ID(ea); + + /* IRQs are not reconciled here, so can't check irqs_disabled */ + VM_WARN_ON(mfmsr() & MSR_EE); + + /* + * SLB kernel faults must be very careful not to touch anything + * that is not bolted. E.g., PACA and global variables are okay, + * mm->context stuff is not. + * + * SLB user faults can access all of kernel memory, but must be + * careful not to touch things like IRQ state because it is not + * "reconciled" here. The difficulty is that we must use + * fast_exception_return to return from kernel SLB faults without + * looking at possible non-bolted memory. We could test user vs + * kernel faults in the interrupt handler asm and do a full fault, + * reconcile, ret_from_except for user faults which would make them + * first class kernel code. But for performance it's probably nicer + * if they go via fast_exception_return too. + */ + if (id >= KERNEL_REGION_ID) { + return slb_allocate_kernel(ea, id); + } else { + struct mm_struct *mm = current->mm; + + if (unlikely(!mm)) + return -EFAULT; - handle_multi_context_slb_miss(context, ea); - exception_exit(prev_state); - return; + return slb_allocate_user(mm, ea); + } +} -slb_bad_addr: +void do_bad_slb_fault(struct pt_regs *regs, unsigned long ea, long err) +{ if (user_mode(regs)) _exception(SIGSEGV, regs, SEGV_BNDERR, ea); else bad_page_fault(regs, ea, SIGSEGV); - exception_exit(prev_state); } diff --git a/arch/powerpc/mm/slb_low.S b/arch/powerpc/mm/slb_low.S deleted file mode 100644 index dbbb7a59eaf8..000000000000 --- a/arch/powerpc/mm/slb_low.S +++ /dev/null @@ -1,338 +0,0 @@ -/* - * Low-level SLB routines - * - * Copyright (C) 2004 David Gibson , IBM - * - * Based on earlier C version: - * Dave Engebretsen and Mike Corrigan {engebret|mikejc}@us.ibm.com - * Copyright (c) 2001 Dave Engebretsen - * Copyright (C) 2002 Anton Blanchard , IBM - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -/* - * This macro generates asm code to compute the VSID scramble - * function. Used in slb_allocate() and do_stab_bolted. The function - * computed is: (protovsid*VSID_MULTIPLIER) % VSID_MODULUS - * - * rt = register containing the proto-VSID and into which the - * VSID will be stored - * rx = scratch register (clobbered) - * rf = flags - * - * - rt and rx must be different registers - * - The answer will end up in the low VSID_BITS bits of rt. The higher - * bits may contain other garbage, so you may need to mask the - * result. - */ -#define ASM_VSID_SCRAMBLE(rt, rx, rf, size) \ - lis rx,VSID_MULTIPLIER_##size@h; \ - ori rx,rx,VSID_MULTIPLIER_##size@l; \ - mulld rt,rt,rx; /* rt = rt * MULTIPLIER */ \ -/* \ - * powermac get slb fault before feature fixup, so make 65 bit part \ - * the default part of feature fixup \ - */ \ -BEGIN_MMU_FTR_SECTION \ - srdi rx,rt,VSID_BITS_65_##size; \ - clrldi rt,rt,(64-VSID_BITS_65_##size); \ - add rt,rt,rx; \ - addi rx,rt,1; \ - srdi rx,rx,VSID_BITS_65_##size; \ - add rt,rt,rx; \ - rldimi rf,rt,SLB_VSID_SHIFT_##size,(64 - (SLB_VSID_SHIFT_##size + VSID_BITS_65_##size)); \ -MMU_FTR_SECTION_ELSE \ - srdi rx,rt,VSID_BITS_##size; \ - clrldi rt,rt,(64-VSID_BITS_##size); \ - add rt,rt,rx; /* add high and low bits */ \ - addi rx,rt,1; \ - srdi rx,rx,VSID_BITS_##size; /* extract 2^VSID_BITS bit */ \ - add rt,rt,rx; \ - rldimi rf,rt,SLB_VSID_SHIFT_##size,(64 - (SLB_VSID_SHIFT_##size + VSID_BITS_##size)); \ -ALT_MMU_FTR_SECTION_END_IFCLR(MMU_FTR_68_BIT_VA) - - -/* void slb_allocate(unsigned long ea); - * - * Create an SLB entry for the given EA (user or kernel). - * r3 = faulting address, r13 = PACA - * r9, r10, r11 are clobbered by this function - * r3 is preserved. - * No other registers are examined or changed. - */ -_GLOBAL(slb_allocate) - /* - * Check if the address falls within the range of the first context, or - * if we may need to handle multi context. For the first context we - * allocate the slb entry via the fast path below. For large address we - * branch out to C-code and see if additional contexts have been - * allocated. - * The test here is: - * (ea & ~REGION_MASK) >= (1ull << MAX_EA_BITS_PER_CONTEXT) - */ - rldicr. r9,r3,4,(63 - MAX_EA_BITS_PER_CONTEXT - 4) - bne- 8f - - srdi r9,r3,60 /* get region */ - srdi r10,r3,SID_SHIFT /* get esid */ - cmpldi cr7,r9,0xc /* cmp PAGE_OFFSET for later use */ - - /* r3 = address, r10 = esid, cr7 = <> PAGE_OFFSET */ - blt cr7,0f /* user or kernel? */ - - /* Check if hitting the linear mapping or some other kernel space - */ - bne cr7,1f - - /* Linear mapping encoding bits, the "li" instruction below will - * be patched by the kernel at boot - */ -.globl slb_miss_kernel_load_linear -slb_miss_kernel_load_linear: - li r11,0 - /* - * context = (ea >> 60) - (0xc - 1) - * r9 = region id. - */ - subi r9,r9,KERNEL_REGION_CONTEXT_OFFSET - -BEGIN_FTR_SECTION - b .Lslb_finish_load -END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT) - b .Lslb_finish_load_1T - -1: -#ifdef CONFIG_SPARSEMEM_VMEMMAP - cmpldi cr0,r9,0xf - bne 1f -/* Check virtual memmap region. To be patched at kernel boot */ -.globl slb_miss_kernel_load_vmemmap -slb_miss_kernel_load_vmemmap: - li r11,0 - b 6f -1: -#endif /* CONFIG_SPARSEMEM_VMEMMAP */ - - /* - * r10 contains the ESID, which is the original faulting EA shifted - * right by 28 bits. We need to compare that with (H_VMALLOC_END >> 28) - * which is 0xd00038000. That can't be used as an immediate, even if we - * ignored the 0xd, so we have to load it into a register, and we only - * have one register free. So we must load all of (H_VMALLOC_END >> 28) - * into a register and compare ESID against that. - */ - lis r11,(H_VMALLOC_END >> 32)@h // r11 = 0xffffffffd0000000 - ori r11,r11,(H_VMALLOC_END >> 32)@l // r11 = 0xffffffffd0003800 - // Rotate left 4, then mask with 0xffffffff0 - rldic r11,r11,4,28 // r11 = 0xd00038000 - cmpld r10,r11 // if r10 >= r11 - bge 5f // goto io_mapping - - /* - * vmalloc mapping gets the encoding from the PACA as the mapping - * can be demoted from 64K -> 4K dynamically on some machines. - */ - lhz r11,PACAVMALLOCSLLP(r13) - b 6f -5: - /* IO mapping */ -.globl slb_miss_kernel_load_io -slb_miss_kernel_load_io: - li r11,0 -6: - /* - * context = (ea >> 60) - (0xc - 1) - * r9 = region id. - */ - subi r9,r9,KERNEL_REGION_CONTEXT_OFFSET - -BEGIN_FTR_SECTION - b .Lslb_finish_load -END_MMU_FTR_SECTION_IFCLR(MMU_FTR_1T_SEGMENT) - b .Lslb_finish_load_1T - -0: /* - * For userspace addresses, make sure this is region 0. - */ - cmpdi r9, 0 - bne- 8f - /* - * user space make sure we are within the allowed limit - */ - ld r11,PACA_SLB_ADDR_LIMIT(r13) - cmpld r3,r11 - bge- 8f - - /* when using slices, we extract the psize off the slice bitmaps - * and then we need to get the sllp encoding off the mmu_psize_defs - * array. - * - * XXX This is a bit inefficient especially for the normal case, - * so we should try to implement a fast path for the standard page - * size using the old sllp value so we avoid the array. We cannot - * really do dynamic patching unfortunately as processes might flip - * between 4k and 64k standard page size - */ -#ifdef CONFIG_PPC_MM_SLICES - /* r10 have esid */ - cmpldi r10,16 - /* below SLICE_LOW_TOP */ - blt 5f - /* - * Handle hpsizes, - * r9 is get_paca()->context.high_slices_psize[index], r11 is mask_index - */ - srdi r11,r10,(SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT + 1) /* index */ - addi r9,r11,PACAHIGHSLICEPSIZE - lbzx r9,r13,r9 /* r9 is hpsizes[r11] */ - /* r11 = (r10 >> (SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT)) & 0x1 */ - rldicl r11,r10,(64 - (SLICE_HIGH_SHIFT - SLICE_LOW_SHIFT)),63 - b 6f - -5: - /* - * Handle lpsizes - * r9 is get_paca()->context.low_slices_psize[index], r11 is mask_index - */ - srdi r11,r10,1 /* index */ - addi r9,r11,PACALOWSLICESPSIZE - lbzx r9,r13,r9 /* r9 is lpsizes[r11] */ - rldicl r11,r10,0,63 /* r11 = r10 & 0x1 */ -6: - sldi r11,r11,2 /* index * 4 */ - /* Extract the psize and multiply to get an array offset */ - srd r9,r9,r11 - andi. r9,r9,0xf - mulli r9,r9,MMUPSIZEDEFSIZE - - /* Now get to the array and obtain the sllp - */ - ld r11,PACATOC(r13) - ld r11,mmu_psize_defs@got(r11) - add r11,r11,r9 - ld r11,MMUPSIZESLLP(r11) - ori r11,r11,SLB_VSID_USER -#else - /* paca context sllp already contains the SLB_VSID_USER bits */ - lhz r11,PACACONTEXTSLLP(r13) -#endif /* CONFIG_PPC_MM_SLICES */ - - ld r9,PACACONTEXTID(r13) -BEGIN_FTR_SECTION - cmpldi r10,0x1000 - bge .Lslb_finish_load_1T -END_MMU_FTR_SECTION_IFSET(MMU_FTR_1T_SEGMENT) - b .Lslb_finish_load - -8: /* invalid EA - return an error indication */ - crset 4*cr0+eq /* indicate failure */ - blr - -/* - * Finish loading of an SLB entry and return - * - * r3 = EA, r9 = context, r10 = ESID, r11 = flags, clobbers r9, cr7 = <> PAGE_OFFSET - */ -.Lslb_finish_load: - rldimi r10,r9,ESID_BITS,0 - ASM_VSID_SCRAMBLE(r10,r9,r11,256M) - /* r3 = EA, r11 = VSID data */ - /* - * Find a slot, round robin. Previously we tried to find a - * free slot first but that took too long. Unfortunately we - * dont have any LRU information to help us choose a slot. - */ - - mr r9,r3 - - /* slb_finish_load_1T continues here. r9=EA with non-ESID bits clear */ -7: ld r10,PACASTABRR(r13) - addi r10,r10,1 - /* This gets soft patched on boot. */ -.globl slb_compare_rr_to_size -slb_compare_rr_to_size: - cmpldi r10,0 - - blt+ 4f - li r10,SLB_NUM_BOLTED - -4: - std r10,PACASTABRR(r13) - -3: - rldimi r9,r10,0,36 /* r9 = EA[0:35] | entry */ - oris r10,r9,SLB_ESID_V@h /* r10 = r9 | SLB_ESID_V */ - - /* r9 = ESID data, r11 = VSID data */ - - /* - * No need for an isync before or after this slbmte. The exception - * we enter with and the rfid we exit with are context synchronizing. - */ - slbmte r11,r10 - - crclr 4*cr0+eq /* set result to "success" */ -BEGIN_FTR_SECTION - blr /* No SLB cache, see switch_slb */ -FTR_SECTION_ELSE - bgelr cr7 /* we're done for kernel addresses */ -ALT_FTR_SECTION_END_IFSET(CPU_FTR_ARCH_300) - - /* Update the slb cache for user addresses */ - lhz r9,PACASLBCACHEPTR(r13) /* offset = paca->slb_cache_ptr */ - cmpldi r9,SLB_CACHE_ENTRIES - bge 1f - - /* still room in the slb cache */ - sldi r11,r9,2 /* r11 = offset * sizeof(u32) */ - srdi r10,r10,28 /* get the 36 bits of the ESID */ - add r11,r11,r13 /* r11 = (u32 *)paca + offset */ - stw r10,PACASLBCACHE(r11) /* paca->slb_cache[offset] = esid */ - addi r9,r9,1 /* offset++ */ - b 2f -1: /* offset >= SLB_CACHE_ENTRIES */ - li r9,SLB_CACHE_ENTRIES+1 -2: - sth r9,PACASLBCACHEPTR(r13) /* paca->slb_cache_ptr = offset */ - crclr 4*cr0+eq /* set result to "success" */ - blr - -/* - * Finish loading of a 1T SLB entry (for the kernel linear mapping) and return. - * - * r3 = EA, r9 = context, r10 = ESID(256MB), r11 = flags, clobbers r9 - */ -.Lslb_finish_load_1T: - srdi r10,r10,(SID_SHIFT_1T - SID_SHIFT) /* get 1T ESID */ - rldimi r10,r9,ESID_BITS_1T,0 - ASM_VSID_SCRAMBLE(r10,r9,r11,1T) - - li r10,MMU_SEGSIZE_1T - rldimi r11,r10,SLB_VSID_SSIZE_SHIFT,0 /* insert segment size */ - - /* r3 = EA, r11 = VSID data */ - clrrdi r9,r3,SID_SHIFT_1T /* clear out non-ESID bits */ - b 7b - - -_ASM_NOKPROBE_SYMBOL(slb_allocate) -_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_linear) -_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_io) -_ASM_NOKPROBE_SYMBOL(slb_compare_rr_to_size) -#ifdef CONFIG_SPARSEMEM_VMEMMAP -_ASM_NOKPROBE_SYMBOL(slb_miss_kernel_load_vmemmap) -#endif From patchwork Mon Aug 20 09:41:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 959550 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41v8Dc4HxRz9s3Z for ; Mon, 20 Aug 2018 19:50:08 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="REpGKzFI"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41v8Db52mXzF1Wr for ; Mon, 20 Aug 2018 19:50:07 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="REpGKzFI"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::242; helo=mail-pl0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="REpGKzFI"; dkim-atps=neutral Received: from mail-pl0-x242.google.com (mail-pl0-x242.google.com [IPv6:2607:f8b0:400e:c01::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41v83c4THWzF0h9 for ; Mon, 20 Aug 2018 19:42:20 +1000 (AEST) Received: by mail-pl0-x242.google.com with SMTP id j8-v6so6822774pll.12 for ; Mon, 20 Aug 2018 02:42:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SyzIx4LsaimaSoZCAGOuyZ6a7ZRJKbV6BIuAYPDgEUE=; b=REpGKzFILMxcMME++ioI3HA8+ctQUpBtakq+0/Y3H/+TlVf+pn0Fx1pWYawy/PhhBz xJp5eU+rCZ62aWcMRn3nmLOFrtjJWBVKh8mCsw7W8rq28Nr1gCGZlOEUnfshYj9gIF0G Tplv5jRTYQwE/3+sTYGg7ZI5gJx5yM/+1ORUXD1xBHcvNN68Lvr2pl4auhbI5JVIIWWf H0I91N1j9K34gvwJHPbgN3oa6KlVN1Hf1ZzDnQoxesyaS+5HTiDa5Jsk5ozp2H0yIhUU lkz2HEy4v+9ph3HzMp9eTTvDGha0Fm5YD8lHvr7YE3QDHgpacgFdAAx4wbL5zq8wiAu7 uxYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SyzIx4LsaimaSoZCAGOuyZ6a7ZRJKbV6BIuAYPDgEUE=; b=P62sjTeeS2buv8OCd58Mi5N7EKcQt27NwDZQ48yUWJBI7Aw0z1Qu6pg7l1wFYPnlMx 1IzKIdswVYcdTQa1lQ+ley3A/cBvxZxje1QuSOcv/wLY1lB1cDD6QmigWFNOajVzXUMo QvT3ReuJ7+y1y3XfpjZW2NyfOMpepB+X8naKWA0QYqWn54m2CXILPUAz02N5uy9KlUw3 r4ZFkkEtT4mSGYwn5cgXldVJAitb7RoABY0kGn9+Vxifhtc5pYDM38lPJHk/RC8sn+3j nFhe4duVmIPi7vyoH3TRrACEnDSQvkKvmqzUmbjd+UquXjx9RTh6qWxDRulWiOHnvLn3 SUnQ== X-Gm-Message-State: AOUpUlEZidSNbj48hQzdKAXlseUnv3RWoNH5EEDWluYu2Qp1LlqG8f+k 4eX7ZTJa9Yii/a4sYwlOZWt2b/QY X-Google-Smtp-Source: AA+uWPwuzZ8BP3k9LYT+u5ka4BBYYXdLgGJKs5Ldh7Nz+eAw2Jv/BGS0E3l0oBPzmnkXaiLm4okUFg== X-Received: by 2002:a17:902:3041:: with SMTP id u59-v6mr43785439plb.99.1534758138373; Mon, 20 Aug 2018 02:42:18 -0700 (PDT) Received: from roar.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id x87-v6sm14793281pfa.143.2018.08.20.02.42.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 02:42:17 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/5] powerpc/64s/hash: remove user SLB data from the paca Date: Mon, 20 Aug 2018 19:41:57 +1000 Message-Id: <20180820094200.13003-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com> References: <20180820094200.13003-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" User SLB mappig data is copied into the PACA from the mm->context so it can be accessed by the SLB miss handlers. After the previous patch, SLB miss handlers now run with relocation on, and user SLB misses are able to take recursive kernel SLB misses, so the user SLB mapping data can be removed from the paca and accessed directly. --- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 1 + arch/powerpc/include/asm/paca.h | 13 ------ arch/powerpc/kernel/asm-offsets.c | 9 ---- arch/powerpc/kernel/paca.c | 21 --------- arch/powerpc/mm/hash_utils_64.c | 46 +++++-------------- arch/powerpc/mm/mmu_context.c | 3 +- arch/powerpc/mm/slb.c | 20 +++++++- arch/powerpc/mm/slice.c | 29 ++++-------- 8 files changed, 40 insertions(+), 102 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index b3520b549cba..39764214aef5 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -496,6 +496,7 @@ static inline void hpte_init_pseries(void) { } extern void hpte_init_native(void); extern void slb_initialize(void); +extern void core_flush_all_slbs(struct mm_struct *mm); extern void slb_flush_and_rebolt(void); void slb_flush_all_realmode(void); void __slb_restore_bolted_realmode(void); diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h index 4331295db0f7..8c258a057207 100644 --- a/arch/powerpc/include/asm/paca.h +++ b/arch/powerpc/include/asm/paca.h @@ -143,18 +143,6 @@ struct paca_struct { struct tlb_core_data tcd; #endif /* CONFIG_PPC_BOOK3E */ -#ifdef CONFIG_PPC_BOOK3S - mm_context_id_t mm_ctx_id; -#ifdef CONFIG_PPC_MM_SLICES - unsigned char mm_ctx_low_slices_psize[BITS_PER_LONG / BITS_PER_BYTE]; - unsigned char mm_ctx_high_slices_psize[SLICE_ARRAY_SIZE]; - unsigned long mm_ctx_slb_addr_limit; -#else - u16 mm_ctx_user_psize; - u16 mm_ctx_sllp; -#endif -#endif - /* * then miscellaneous read-write fields */ @@ -256,7 +244,6 @@ struct paca_struct { #endif /* CONFIG_PPC_PSERIES */ } ____cacheline_aligned; -extern void copy_mm_to_paca(struct mm_struct *mm); extern struct paca_struct **paca_ptrs; extern void initialise_paca(struct paca_struct *new_paca, int cpu); extern void setup_paca(struct paca_struct *new_paca); diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index 7834256585f1..43b67ead5b97 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -181,15 +181,6 @@ int main(void) OFFSET(PACAIRQSOFTMASK, paca_struct, irq_soft_mask); OFFSET(PACAIRQHAPPENED, paca_struct, irq_happened); OFFSET(PACA_FTRACE_ENABLED, paca_struct, ftrace_enabled); -#ifdef CONFIG_PPC_BOOK3S - OFFSET(PACACONTEXTID, paca_struct, mm_ctx_id); -#ifdef CONFIG_PPC_MM_SLICES - OFFSET(PACALOWSLICESPSIZE, paca_struct, mm_ctx_low_slices_psize); - OFFSET(PACAHIGHSLICEPSIZE, paca_struct, mm_ctx_high_slices_psize); - OFFSET(PACA_SLB_ADDR_LIMIT, paca_struct, mm_ctx_slb_addr_limit); - DEFINE(MMUPSIZEDEFSIZE, sizeof(struct mmu_psize_def)); -#endif /* CONFIG_PPC_MM_SLICES */ -#endif #ifdef CONFIG_PPC_BOOK3E OFFSET(PACAPGD, paca_struct, pgd); diff --git a/arch/powerpc/kernel/paca.c b/arch/powerpc/kernel/paca.c index 0ee3e6d50f28..6752e17f0281 100644 --- a/arch/powerpc/kernel/paca.c +++ b/arch/powerpc/kernel/paca.c @@ -259,24 +259,3 @@ void __init free_unused_pacas(void) paca_ptrs_size + paca_struct_size, nr_cpu_ids); } -void copy_mm_to_paca(struct mm_struct *mm) -{ -#ifdef CONFIG_PPC_BOOK3S - mm_context_t *context = &mm->context; - - get_paca()->mm_ctx_id = context->id; -#ifdef CONFIG_PPC_MM_SLICES - VM_BUG_ON(!mm->context.slb_addr_limit); - get_paca()->mm_ctx_slb_addr_limit = mm->context.slb_addr_limit; - memcpy(&get_paca()->mm_ctx_low_slices_psize, - &context->low_slices_psize, sizeof(context->low_slices_psize)); - memcpy(&get_paca()->mm_ctx_high_slices_psize, - &context->high_slices_psize, TASK_SLICE_ARRAY_SZ(mm)); -#else /* CONFIG_PPC_MM_SLICES */ - get_paca()->mm_ctx_user_psize = context->user_psize; - get_paca()->mm_ctx_sllp = context->sllp; -#endif -#else /* !CONFIG_PPC_BOOK3S */ - return; -#endif -} diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c index f23a89d8e4ce..88c95dc8b141 100644 --- a/arch/powerpc/mm/hash_utils_64.c +++ b/arch/powerpc/mm/hash_utils_64.c @@ -1088,16 +1088,16 @@ unsigned int hash_page_do_lazy_icache(unsigned int pp, pte_t pte, int trap) } #ifdef CONFIG_PPC_MM_SLICES -static unsigned int get_paca_psize(unsigned long addr) +static unsigned int get_psize(struct mm_struct *mm, unsigned long addr) { unsigned char *psizes; unsigned long index, mask_index; if (addr < SLICE_LOW_TOP) { - psizes = get_paca()->mm_ctx_low_slices_psize; + psizes = mm->context.low_slices_psize; index = GET_LOW_SLICE_INDEX(addr); } else { - psizes = get_paca()->mm_ctx_high_slices_psize; + psizes = mm->context.high_slices_psize; index = GET_HIGH_SLICE_INDEX(addr); } mask_index = index & 0x1; @@ -1105,9 +1105,9 @@ static unsigned int get_paca_psize(unsigned long addr) } #else -unsigned int get_paca_psize(unsigned long addr) +unsigned int get_psize(struct mm_struct *mm, unsigned long addr) { - return get_paca()->mm_ctx_user_psize; + return mm->context.user_psize; } #endif @@ -1118,15 +1118,11 @@ unsigned int get_paca_psize(unsigned long addr) #ifdef CONFIG_PPC_64K_PAGES void demote_segment_4k(struct mm_struct *mm, unsigned long addr) { - if (get_slice_psize(mm, addr) == MMU_PAGE_4K) + if (get_psize(mm, addr) == MMU_PAGE_4K) return; slice_set_range_psize(mm, addr, 1, MMU_PAGE_4K); copro_flush_all_slbs(mm); - if ((get_paca_psize(addr) != MMU_PAGE_4K) && (current->mm == mm)) { - - copy_mm_to_paca(mm); - slb_flush_and_rebolt(); - } + core_flush_all_slbs(mm); } #endif /* CONFIG_PPC_64K_PAGES */ @@ -1191,22 +1187,6 @@ void hash_failure_debug(unsigned long ea, unsigned long access, trap, vsid, ssize, psize, lpsize, pte); } -static void check_paca_psize(unsigned long ea, struct mm_struct *mm, - int psize, bool user_region) -{ - if (user_region) { - if (psize != get_paca_psize(ea)) { - copy_mm_to_paca(mm); - slb_flush_and_rebolt(); - } - } else if (get_paca()->vmalloc_sllp != - mmu_psize_defs[mmu_vmalloc_psize].sllp) { - get_paca()->vmalloc_sllp = - mmu_psize_defs[mmu_vmalloc_psize].sllp; - slb_vmalloc_update(); - } -} - /* Result code is: * 0 - handled * 1 - normal page fault @@ -1239,7 +1219,7 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, rc = 1; goto bail; } - psize = get_slice_psize(mm, ea); + psize = get_psize(mm, ea); ssize = user_segment_size(ea); vsid = get_user_vsid(&mm->context, ea, ssize); break; @@ -1327,9 +1307,6 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, WARN_ON(1); } #endif - if (current->mm == mm) - check_paca_psize(ea, mm, psize, user_region); - goto bail; } @@ -1364,15 +1341,14 @@ int hash_page_mm(struct mm_struct *mm, unsigned long ea, "to 4kB pages because of " "non-cacheable mapping\n"); psize = mmu_vmalloc_psize = MMU_PAGE_4K; + slb_vmalloc_update(); copro_flush_all_slbs(mm); + core_flush_all_slbs(mm); } } #endif /* CONFIG_PPC_64K_PAGES */ - if (current->mm == mm) - check_paca_psize(ea, mm, psize, user_region); - #ifdef CONFIG_PPC_64K_PAGES if (psize == MMU_PAGE_64K) rc = __hash_page_64K(ea, access, vsid, ptep, trap, @@ -1460,7 +1436,7 @@ int __hash_page(unsigned long ea, unsigned long msr, unsigned long trap, #ifdef CONFIG_PPC_MM_SLICES static bool should_hash_preload(struct mm_struct *mm, unsigned long ea) { - int psize = get_slice_psize(mm, ea); + int psize = get_psize(mm, ea); /* We only prefault standard pages for now */ if (unlikely(psize != mm->context.user_psize)) diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index f84e14f23e50..28ae2835db3d 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -54,8 +54,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * MMU context id, which is then moved to SPRN_PID. * * For the hash MMU it is either the first load from slb_cache - * in switch_slb(), and/or the store of paca->mm_ctx_id in - * copy_mm_to_paca(). + * in switch_slb(), and/or load of MMU context id. * * On the other side, the barrier is in mm/tlb-radix.c for * radix which orders earlier stores to clear the PTEs vs diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 5643879fa2b1..221d94b4f9cf 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -274,8 +274,6 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm) get_paca()->slb_cache_ptr = 0; } - copy_mm_to_paca(mm); - /* * preload some userspace segments into the SLB. * Almost all 32 and 64bit PowerPC executables are linked at @@ -302,6 +300,24 @@ void slb_set_size(u16 size) mmu_slb_size = size; } +static void cpu_flush_slb(void *parm) +{ + struct mm_struct *mm = parm; + unsigned long flags; + + if (mm != current->active_mm) + return; + + local_irq_save(flags); + slb_flush_and_rebolt(); + local_irq_restore(flags); +} + +void core_flush_all_slbs(struct mm_struct *mm) +{ + on_each_cpu(cpu_flush_slb, mm, 1); +} + void slb_initialize(void) { unsigned long linear_llp, vmalloc_llp, io_llp; diff --git a/arch/powerpc/mm/slice.c b/arch/powerpc/mm/slice.c index 205fe557ca10..606f424aac47 100644 --- a/arch/powerpc/mm/slice.c +++ b/arch/powerpc/mm/slice.c @@ -207,23 +207,6 @@ static bool slice_check_range_fits(struct mm_struct *mm, return true; } -static void slice_flush_segments(void *parm) -{ -#ifdef CONFIG_PPC64 - struct mm_struct *mm = parm; - unsigned long flags; - - if (mm != current->active_mm) - return; - - copy_mm_to_paca(current->active_mm); - - local_irq_save(flags); - slb_flush_and_rebolt(); - local_irq_restore(flags); -#endif -} - static void slice_convert(struct mm_struct *mm, const struct slice_mask *mask, int psize) { @@ -289,6 +272,9 @@ static void slice_convert(struct mm_struct *mm, spin_unlock_irqrestore(&slice_convert_lock, flags); copro_flush_all_slbs(mm); +#ifdef CONFIG_PPC64 + core_flush_all_slbs(mm); +#endif } /* @@ -502,8 +488,9 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len, * be already initialised beyond the old address limit. */ mm->context.slb_addr_limit = high_limit; - - on_each_cpu(slice_flush_segments, mm, 1); +#ifdef CONFIG_PPC64 + core_flush_all_slbs(mm); +#endif } /* Sanity checks */ @@ -665,8 +652,10 @@ unsigned long slice_get_unmapped_area(unsigned long addr, unsigned long len, (SLICE_NUM_HIGH && !bitmap_empty(potential_mask.high_slices, SLICE_NUM_HIGH))) { slice_convert(mm, &potential_mask, psize); +#ifdef CONFIG_PPC64 if (psize > MMU_PAGE_BASE) - on_each_cpu(slice_flush_segments, mm, 1); + core_flush_all_slbs(mm); +#endif } return newaddr; From patchwork Mon Aug 20 09:41:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 959551 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41v8Hh2qcSz9s3Z for ; Mon, 20 Aug 2018 19:52:48 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HrNguRsf"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41v8Hh1NMPzF0yB for ; Mon, 20 Aug 2018 19:52:48 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HrNguRsf"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::442; helo=mail-pf1-x442.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HrNguRsf"; dkim-atps=neutral Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41v83f4P4ZzF162 for ; Mon, 20 Aug 2018 19:42:22 +1000 (AEST) Received: by mail-pf1-x442.google.com with SMTP id k21-v6so6479996pff.11 for ; Mon, 20 Aug 2018 02:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZvciuPOIX2Cx4otQ9jaj30UarhYsRC6tKvZMn934L9M=; b=HrNguRsfY3F0BgfFYYqHSgPUiZOGK3WOqMg90Ssh5pRWq6BKPzQlvxRXGlDgMqtoua tmwi+ztGpnGEfis4hYqjRsj3cJ8AN2dcDWM2LJJbBaGa0fTO+5RdDSySUTtL03rED7P/ XOo7QGUfhuabMb2dBi2YRMHC5ssKzyWb2vv3VAXfAU02U6WI/xYJ6jebvTL8saClLRHn H/xD9xc3UcmYiaWAjkg+czjJj7zyZtp3SnsFfqDb5fmHqDT5e6oqsRKUmMnak88G/ezL /K2dG+I1lZCYvof6U9ERaLrAo7hlTQaDkPARET8eQk5cXLA0tUOb3kfYn9oDgdDFItAD jNyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZvciuPOIX2Cx4otQ9jaj30UarhYsRC6tKvZMn934L9M=; b=pa6P2wqYK1kM3RUlxI1gfPK+YhBoMMUTUigr8oojqYU0e2x0d5BSjzl+XHFvAt/+L+ TC7pNOHhpg0+an5ESlVN60e32u84MVlZKD9ZBqQfP3WPWQ0JOTmx80WvZVPOznErT+fM tF049V3/VMncURiLhFU398HDieW+8I6GyM324aseQg37WwfmpzgYliTIv7o0/6dR9ef0 xsZILMTM2iDqelVEsY5CVdaMOhxgs4m/Wnt6q3ZKgcpqIbSCEB1YLOjGBwuISloGTdUu NHqqPB7MeOWhT6bHH5io+FcHWk1tyyHLc6m/metTAKNAYlF2ZmCQMC6/MuGKdHIDeSlF 4uVw== X-Gm-Message-State: AOUpUlEjL/TCgHZgmueNZaB4g2Gk1O60xipx2hakj7tvkWmYyyHRxCF2 Imu80DXVKvadbmLS+beOOXOE40oc X-Google-Smtp-Source: AA+uWPy+cWPvGEfr8j4cyxqrK9u2IOvBZ+iGhvVsnpy7KO9Xqp7Sa705vtLLrjf+shZ4DQfnCfmfIA== X-Received: by 2002:a62:b604:: with SMTP id j4-v6mr47742831pff.199.1534758140503; Mon, 20 Aug 2018 02:42:20 -0700 (PDT) Received: from roar.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id x87-v6sm14793281pfa.143.2018.08.20.02.42.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 02:42:20 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/5] powerpc/64s/hash: remove the first vmalloc segment from the bolted SLB Date: Mon, 20 Aug 2018 19:41:58 +1000 Message-Id: <20180820094200.13003-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com> References: <20180820094200.13003-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Remove the first vmalloc segment from bolted SLBEs. This is not required to be bolted, and seems like it was added to help pre-load the SLB on context switch. However there are now other segments like the vmemmap segment that often take misses after a context switch, so it is better to solve this a different way and save a bolted entry. --- arch/powerpc/include/asm/book3s/64/mmu-hash.h | 2 +- arch/powerpc/mm/slb.c | 16 ++++------------ 2 files changed, 5 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu-hash.h b/arch/powerpc/include/asm/book3s/64/mmu-hash.h index 39764214aef5..4c8d413ce99a 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu-hash.h +++ b/arch/powerpc/include/asm/book3s/64/mmu-hash.h @@ -30,7 +30,7 @@ * SLB */ -#define SLB_NUM_BOLTED 3 +#define SLB_NUM_BOLTED 2 #define SLB_CACHE_ENTRIES 8 #define SLB_MIN_SIZE 32 diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 221d94b4f9cf..6e595d75d997 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -133,13 +133,11 @@ static void __slb_flush_and_rebolt(void) { /* If you change this make sure you change SLB_NUM_BOLTED * and PR KVM appropriately too. */ - unsigned long linear_llp, vmalloc_llp, lflags, vflags; + unsigned long linear_llp, lflags; unsigned long ksp_esid_data, ksp_vsid_data; linear_llp = mmu_psize_defs[mmu_linear_psize].sllp; - vmalloc_llp = mmu_psize_defs[mmu_vmalloc_psize].sllp; lflags = SLB_VSID_KERNEL | linear_llp; - vflags = SLB_VSID_KERNEL | vmalloc_llp; ksp_esid_data = mk_esid_data(get_paca()->kstack, mmu_kernel_ssize, KSTACK_INDEX); if ((ksp_esid_data & ~0xfffffffUL) <= PAGE_OFFSET) { @@ -157,14 +155,10 @@ static void __slb_flush_and_rebolt(void) * the stack between the slbia and rebolting it. */ asm volatile("isync\n" "slbia\n" - /* Slot 1 - first VMALLOC segment */ + /* Slot 1 - kernel stack */ "slbmte %0,%1\n" - /* Slot 2 - kernel stack */ - "slbmte %2,%3\n" "isync" - :: "r"(mk_vsid_data(VMALLOC_START, mmu_kernel_ssize, vflags)), - "r"(mk_esid_data(VMALLOC_START, mmu_kernel_ssize, VMALLOC_INDEX)), - "r"(ksp_vsid_data), + :: "r"(ksp_vsid_data), "r"(ksp_esid_data) : "memory"); } @@ -321,7 +315,7 @@ void core_flush_all_slbs(struct mm_struct *mm) void slb_initialize(void) { unsigned long linear_llp, vmalloc_llp, io_llp; - unsigned long lflags, vflags; + unsigned long lflags; static int slb_encoding_inited; #ifdef CONFIG_SPARSEMEM_VMEMMAP unsigned long vmemmap_llp; @@ -347,14 +341,12 @@ void slb_initialize(void) get_paca()->stab_rr = SLB_NUM_BOLTED - 1; lflags = SLB_VSID_KERNEL | linear_llp; - vflags = SLB_VSID_KERNEL | vmalloc_llp; /* Invalidate the entire SLB (even entry 0) & all the ERATS */ asm volatile("isync":::"memory"); asm volatile("slbmte %0,%0"::"r" (0) : "memory"); asm volatile("isync; slbia; isync":::"memory"); create_shadowed_slbe(PAGE_OFFSET, mmu_kernel_ssize, lflags, LINEAR_INDEX); - create_shadowed_slbe(VMALLOC_START, mmu_kernel_ssize, vflags, VMALLOC_INDEX); /* For the boot cpu, we're running on the stack in init_thread_union, * which is in the first segment of the linear mapping, and also From patchwork Mon Aug 20 09:41:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 959552 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41v8LD67nvz9s3Z for ; Mon, 20 Aug 2018 19:55:00 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aYX3m4vs"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41v8LD4XKVzF1Rx for ; Mon, 20 Aug 2018 19:55:00 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aYX3m4vs"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52c; helo=mail-pg1-x52c.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aYX3m4vs"; dkim-atps=neutral Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41v83h6ZkYzF1Rh for ; Mon, 20 Aug 2018 19:42:24 +1000 (AEST) Received: by mail-pg1-x52c.google.com with SMTP id z25-v6so1814507pgu.7 for ; Mon, 20 Aug 2018 02:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=252lYkcjOqPTLw6OndLhHYDgsIa+3AUCIk39crBAMOQ=; b=aYX3m4vsq/VFHnDftPBNrWrZMJQ8dT/20+gPD3BMAhio36skc0lEGoONR6y3C+B99U P/LwI5SBe0N0hL4Kv02ymUwYEux3EJI7dlti044Y2a2L4RV4Z14KH0YO3hhCnP+PQHKT vRghF627a1sTr3oYg8y7aAAm1tCrglzrWoad7Gmp5eT05SnkQB7KrPQAmxyf5OPYMawc bD4HUaS4CZxa8O/xDUwruEL6+0qWDiEBwTotQ3bStxBX8eZnBkavA7Z598iSuv/D1D0z xEecbIFqzdSf5v6hKFhF25H51aGbrbvVJI8fV+DSLAO4wjL0SZgHBpx8FNeIxcY7Pae5 Y4zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=252lYkcjOqPTLw6OndLhHYDgsIa+3AUCIk39crBAMOQ=; b=SXQU6CqafyKtJAFmdgZuDzgrxLJv1DF6DPLY2mspzFDQ6uMtnCrNOP4cLE4uyZoUtQ +V4j1mSzgq0qX208Ebgf5TB8tLvtH8s5nwzm2r6OZJO4IwNxZdkWl2u+FSI5RkohTtB+ Coc/Pc9cVQu6P5aHqUUzdLT5zaIFclIMlgqUhkZWHkuVmJKQfx+38x0hmtsBBLyFVGLi 4+Y0DJchb7uu8jAS/MCGij6XnllhdPtQ5rlAyqw4g7XqmzlI1oxJ4LmRkkLAC3htYgVP E5mujdgQkkx3zKisXlb3QB9nFo2AOK5dCGAQ9LGwSEcE1ElxmW2PVLHLhCnaMsDkDx95 qJfQ== X-Gm-Message-State: AOUpUlHMJGVm3HMoXL3U0T/wON4CCK9vjaNcuqkJ0ZtfDa0uqT+5l0qT a18cSpYki+srr2NTJbRkSsuWnkkP X-Google-Smtp-Source: AA+uWPxTaKzBWQRxHBWCn9PACI/3sVgd97CuEBcHljn7cw4+3MRDlzHFHI7xqf2s1chysqjFR6t6Fw== X-Received: by 2002:a62:8d7:: with SMTP id 84-v6mr47862783pfi.172.1534758142737; Mon, 20 Aug 2018 02:42:22 -0700 (PDT) Received: from roar.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id x87-v6sm14793281pfa.143.2018.08.20.02.42.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 02:42:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/5] powerpc/64s/hash: Add SLB allocation bitmaps Date: Mon, 20 Aug 2018 19:41:59 +1000 Message-Id: <20180820094200.13003-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com> References: <20180820094200.13003-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add 32-entry bitmaps to track the allocation status of the first 32 SLB entries, and whether they are user or kernel entries. These are used to prevent context switches rolling the SLB round robin allocator and evicting important kernel SLBEs when there are obvious free entries. --- arch/powerpc/include/asm/paca.h | 6 +++-- arch/powerpc/mm/slb.c | 42 +++++++++++++++++++++++---------- arch/powerpc/xmon/xmon.c | 2 +- 3 files changed, 35 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/include/asm/paca.h b/arch/powerpc/include/asm/paca.h index 8c258a057207..bf7ab59be3b8 100644 --- a/arch/powerpc/include/asm/paca.h +++ b/arch/powerpc/include/asm/paca.h @@ -113,7 +113,10 @@ struct paca_struct { * on the linear mapping */ /* SLB related definitions */ u16 vmalloc_sllp; - u16 slb_cache_ptr; + u8 slb_cache_ptr; + u8 stab_rr; /* stab/slb round-robin counter */ + u32 slb_used_bitmap; /* Bitmaps for first 32 SLB entries. */ + u32 slb_kern_bitmap; u32 slb_cache[SLB_CACHE_ENTRIES]; #endif /* CONFIG_PPC_BOOK3S_64 */ @@ -148,7 +151,6 @@ struct paca_struct { */ struct task_struct *__current; /* Pointer to current */ u64 kstack; /* Saved Kernel stack addr */ - u64 stab_rr; /* stab/slb round-robin counter */ u64 saved_r1; /* r1 save for RTAS calls or PM or EE=0 */ u64 saved_msr; /* MSR saved here by enter_rtas */ u16 trap_save; /* Used when bad stack is encountered */ diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 6e595d75d997..3de63598f7c4 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -267,6 +267,7 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm) get_paca()->slb_cache_ptr = 0; } + get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap; /* * preload some userspace segments into the SLB. @@ -339,6 +340,8 @@ void slb_initialize(void) } get_paca()->stab_rr = SLB_NUM_BOLTED - 1; + get_paca()->slb_kern_bitmap |= (1U << SLB_NUM_BOLTED) - 1; + get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap; lflags = SLB_VSID_KERNEL | linear_llp; @@ -390,27 +393,42 @@ static void slb_cache_update(unsigned long esid_data) } } -static enum slb_index alloc_slb_index(void) +static enum slb_index alloc_slb_index(bool kernel) { enum slb_index index; - /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */ - index = get_paca()->stab_rr; - if (index < (mmu_slb_size - 1)) - index++; - else - index = SLB_NUM_BOLTED; - get_paca()->stab_rr = index; + /* + * SLBs beyond 32 entries are allocated with stab_rr only + * POWER7/8/9 have 32 SLB entries, this could be expanded if a + * future CPU has more. + */ + if (get_paca()->slb_used_bitmap != U32_MAX) { + index = ffz(get_paca()->slb_used_bitmap); + get_paca()->slb_used_bitmap |= 1U << index; + if (kernel) + get_paca()->slb_kern_bitmap |= 1U << index; + } else { + /* round-robin replacement of slb starting at SLB_NUM_BOLTED. */ + index = get_paca()->stab_rr; + if (index < (mmu_slb_size - 1)) + index++; + else + index = SLB_NUM_BOLTED; + get_paca()->stab_rr = index; + if (kernel && index < 32) + get_paca()->slb_kern_bitmap |= 1U << index; + } + BUG_ON(index < SLB_NUM_BOLTED); return index; } static void slb_insert_entry(unsigned long ea, unsigned long context, - unsigned long flags, int ssize) + unsigned long flags, int ssize, bool kernel) { unsigned long vsid; unsigned long vsid_data, esid_data; - enum slb_index index = alloc_slb_index(); + enum slb_index index = alloc_slb_index(kernel); vsid = get_vsid(context, ea, ssize); vsid_data = (vsid << slb_vsid_shift(ssize)) | flags | @@ -454,7 +472,7 @@ static long slb_allocate_kernel(unsigned long ea, unsigned long id) context = id - KERNEL_REGION_CONTEXT_OFFSET; - slb_insert_entry(ea, context, flags, ssize); + slb_insert_entry(ea, context, flags, ssize, true); return 0; } @@ -487,7 +505,7 @@ static long slb_allocate_user(struct mm_struct *mm, unsigned long ea) bpsize = get_slice_psize(mm, ea); flags = SLB_VSID_USER | mmu_psize_defs[bpsize].sllp; - slb_insert_entry(ea, context, flags, ssize); + slb_insert_entry(ea, context, flags, ssize, false); return 0; } diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c index 8c1d8b4bf6d5..44e16675c7f9 100644 --- a/arch/powerpc/xmon/xmon.c +++ b/arch/powerpc/xmon/xmon.c @@ -2415,7 +2415,7 @@ static void dump_one_paca(int cpu) DUMP(p, __current, "%-*px"); DUMP(p, kstack, "%#-*llx"); printf(" %-*s = 0x%016llx\n", 25, "kstack_base", p->kstack & ~(THREAD_SIZE - 1)); - DUMP(p, stab_rr, "%#-*llx"); + DUMP(p, stab_rr, "%#-*x"); DUMP(p, saved_r1, "%#-*llx"); DUMP(p, trap_save, "%#-*x"); DUMP(p, irq_soft_mask, "%#-*x"); From patchwork Mon Aug 20 09:42:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 959553 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41v8PL714Xz9s3Z for ; Mon, 20 Aug 2018 19:57:42 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Heua0+6+"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41v8PL5QdszF0xR for ; Mon, 20 Aug 2018 19:57:42 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Heua0+6+"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::444; helo=mail-pf1-x444.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Heua0+6+"; dkim-atps=neutral Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41v83l2yWWzF1d7 for ; Mon, 20 Aug 2018 19:42:27 +1000 (AEST) Received: by mail-pf1-x444.google.com with SMTP id i26-v6so6473739pfo.12 for ; Mon, 20 Aug 2018 02:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=b5s1dsTwzDjtm/nNsxxcVS1LkDV2JDZ+M8j/iDgwdEg=; b=Heua0+6+Rbz3qBZWwzTjxUJwpq4Essx//43j6R6JWDTUeLuPyyiYJKskQ3sjR4e4MS IZpE2GcYJj/vERE0j3FR3/tN0ytI9KX6IOb8IgPG9Pr73NbWlsbGQOW3svFC+vjyKiir +85Lkf8ZfStrKQTb5Fs0f1OYusYWu/8CS/xmlVcsiRTNUFdnLs1fpaD8gUVlIZ5dIElL ceIqidOqXIE1h9Z2dA09SxMhpkE6alhYWNhrpPim2khguBQPtAGRmKI31YlEgiBE9MzN TQljdzq3js7JiBx870NEXxQwl6twTPExgE+gvqtttF/4iGH7xcTSe7bnEXjPIj/WmAzt tnAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=b5s1dsTwzDjtm/nNsxxcVS1LkDV2JDZ+M8j/iDgwdEg=; b=F5V7lMjg8YJyj9TTxhTeAu3H/VExZHAIiNKay8TPdbiY7waRyMXj+6SEkSX/HivimO Q8kKTM+JDDsrt9+a/5OphnW2zlP7WmPV2p0bbN3QqRt9pFc8pPHrwljAKvLawBj7aOmM LQiFNKYo9vt8f2GA+zl5MgqxxiEUgPzRuIWxO2/zFDvNK7bnsFYdwEi3kKZ+73x4DpvY hgo+O3/dJ2qaIAB7xKqqWg09qOuXkChuDWT6wq6PF8oysqnE5/VYSq5AMLBPhB56KXZc LsG1Xk5PHrBDddcAFk6+LPrEKvTSiWKg0oZIa5fmCaB6rbjfDaXFJJT9G6YwVQzLun1J 1rVA== X-Gm-Message-State: AOUpUlHccHeV9gpMQkRfm0JQ4/yuwCEAqLjD7VbS6N+/DGVjdsYZUN/B l8mNaMy5R5JhDnpElAiyCvsHoWwE X-Google-Smtp-Source: AA+uWPzfFLtywsQYMe70P7EDby1TIwL/ju44hTT1iWugZ+3QTJXTIrmJYnzWLOFqNr1MHqzmiKRYAQ== X-Received: by 2002:a62:54c7:: with SMTP id i190-v6mr47292806pfb.155.1534758145004; Mon, 20 Aug 2018 02:42:25 -0700 (PDT) Received: from roar.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id x87-v6sm14793281pfa.143.2018.08.20.02.42.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 20 Aug 2018 02:42:24 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 5/5] powerpc/64s/hash: Add a SLB preload cache Date: Mon, 20 Aug 2018 19:42:00 +1000 Message-Id: <20180820094200.13003-6-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180820094200.13003-1-npiggin@gmail.com> References: <20180820094200.13003-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When switching processes, currently all user SLBEs are cleared, and a few (exec_base, pc, and stack) are preloaded. In trivial testing with small apps, this tends to miss the heap and low 256MB segments, and it will also miss commonly accessed segments on large memory workloads. Add a simple round-robin preload cache that just inserts the last SLB miss into the head of the cache and preloads those at context switch time. Much more could go into this, including into the SLB entry reclaim side to track some LRU information etc, which would require a study of large memory workloads. But this is a simple thing we can do now that is an obvious win for common workloads. This plus the previous patch reduces SLB misses of a bare bones boot to busybox from 945 to 180 when using 256MB segments, and 900 to 100 when using 1T segments. These could almost all be eliminated by preloading a bit more carefully with ELF binary loading. --- arch/powerpc/include/asm/thread_info.h | 4 + arch/powerpc/kernel/process.c | 6 ++ arch/powerpc/mm/mmu_context_book3s64.c | 10 ++- arch/powerpc/mm/slb.c | 107 ++++++++++++++++++++----- 4 files changed, 102 insertions(+), 25 deletions(-) diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/asm/thread_info.h index 3c0002044bc9..ee5e49ec12c7 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -29,6 +29,7 @@ #include #include +#define SLB_PRELOAD_NR 8U /* * low level task data. */ @@ -44,6 +45,9 @@ struct thread_info { #if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) && defined(CONFIG_PPC32) struct cpu_accounting_data accounting; #endif + unsigned int slb_preload_nr; + unsigned long slb_preload_ea[SLB_PRELOAD_NR]; + /* low level flags - has atomic operations done on it */ unsigned long flags ____cacheline_aligned_in_smp; }; diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index 913c5725cdb2..678a2c668270 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -1710,6 +1710,8 @@ int copy_thread(unsigned long clone_flags, unsigned long usp, return 0; } +void preload_new_slb_context(unsigned long start, unsigned long sp); + /* * Set up a thread for executing a new program */ @@ -1717,6 +1719,10 @@ void start_thread(struct pt_regs *regs, unsigned long start, unsigned long sp) { #ifdef CONFIG_PPC64 unsigned long load_addr = regs->gpr[2]; /* saved by ELF_PLAT_INIT */ + +#ifdef CONFIG_PPC_BOOK3S_64 + preload_new_slb_context(start, sp); +#endif #endif /* diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c index 4a892d894a0f..3671a32141e2 100644 --- a/arch/powerpc/mm/mmu_context_book3s64.c +++ b/arch/powerpc/mm/mmu_context_book3s64.c @@ -85,7 +85,9 @@ int hash__alloc_context_id(void) } EXPORT_SYMBOL_GPL(hash__alloc_context_id); -static int hash__init_new_context(struct mm_struct *mm) +void init_new_slb_context(struct task_struct *tsk, struct mm_struct *mm); + +static int hash__init_new_context(struct task_struct *tsk, struct mm_struct *mm) { int index; @@ -107,8 +109,10 @@ static int hash__init_new_context(struct mm_struct *mm) * We should not be calling init_new_context() on init_mm. Hence a * check against 0 is OK. */ - if (mm->context.id == 0) + if (mm->context.id == 0) { slice_init_new_context_exec(mm); + init_new_slb_context(tsk, mm); + } subpage_prot_init_new_context(mm); @@ -152,7 +156,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm) if (radix_enabled()) index = radix__init_new_context(mm); else - index = hash__init_new_context(mm); + index = hash__init_new_context(tsk, mm); if (index < 0) return index; diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c index 3de63598f7c4..e53846d4e474 100644 --- a/arch/powerpc/mm/slb.c +++ b/arch/powerpc/mm/slb.c @@ -216,14 +216,85 @@ static inline int esids_match(unsigned long addr1, unsigned long addr2) return (GET_ESID_1T(addr1) == GET_ESID_1T(addr2)); } +static bool preload_hit(struct thread_info *ti, unsigned long ea) +{ + int i; + + for (i = 0; i < min(SLB_PRELOAD_NR, ti->slb_preload_nr); i++) + if (esids_match(ti->slb_preload_ea[i], ea)) + return true; + return false; +} + +static bool preload_add(struct thread_info *ti, unsigned long ea) +{ + if (preload_hit(ti, ea)) + return false; + + ti->slb_preload_ea[ti->slb_preload_nr % SLB_PRELOAD_NR] = ea; + ti->slb_preload_nr++; + + return true; +} + +void preload_new_slb_context(unsigned long start, unsigned long sp) +{ + struct thread_info *ti = current_thread_info(); + struct mm_struct *mm = current->mm; + unsigned long heap = mm->start_brk; + + // printk("preload new slb context tsk:%s pc:%lx heap:%lx stack:%lx\n", current->comm, start, heap, sp); + + if (!is_kernel_addr(start)) { + if (preload_add(ti, start)) + slb_allocate_user(mm, start); + } + + if (!is_kernel_addr(sp)) { + if (preload_add(ti, sp)) + slb_allocate_user(mm, sp); + } + + if (heap && !is_kernel_addr(heap)) { + if (preload_add(ti, heap)) + slb_allocate_user(mm, heap); + } + + // preload mm->mmap_base is too late at this point +} + +void init_new_slb_context(struct task_struct *tsk, struct mm_struct *mm) +{ + struct thread_info *ti = task_thread_info(tsk); + + // printk("init new slb context tsk:%s\n", tsk->comm); + + /* + * Clear out previous cache because new exec will put addresses + * at different places. Preloading is still generally a win here + * because we don't have all ELF information yet and take several + * misses in kernel space on the user addresss when loading binary + * and libraries. However we don't want to insert more slb entries + * than we need for small processes. Probably should look at aging + * out the preload cache slowly at context switch time. + */ + ti->slb_preload_nr = 0; + + /* + * preload some userspace segments into the SLB. + * Almost all 32 and 64bit PowerPC executables are linked at + * 0x10000000 so it makes sense to preload this segment. + */ + preload_add(ti, 0x10000000); +} + /* Flush all user entries from the segment table of the current processor. */ void switch_slb(struct task_struct *tsk, struct mm_struct *mm) { unsigned long offset; unsigned long slbie_data = 0; - unsigned long pc = KSTK_EIP(tsk); - unsigned long stack = KSTK_ESP(tsk); - unsigned long exec_base; + struct thread_info *ti = task_thread_info(tsk); + int i; /* * We need interrupts hard-disabled here, not just soft-disabled, @@ -269,25 +340,12 @@ void switch_slb(struct task_struct *tsk, struct mm_struct *mm) } get_paca()->slb_used_bitmap = get_paca()->slb_kern_bitmap; - /* - * preload some userspace segments into the SLB. - * Almost all 32 and 64bit PowerPC executables are linked at - * 0x10000000 so it makes sense to preload this segment. + /* XXX: should we gradually age out SLBs after a number of context + * switches to reduce reload overhead of unused entries (like we do + * with FP/VEC reload)? */ - exec_base = 0x10000000; - - if (is_kernel_addr(pc) || is_kernel_addr(stack) || - is_kernel_addr(exec_base)) - return; - - slb_allocate_user(mm, pc); - - if (!esids_match(pc, stack)) - slb_allocate_user(mm, stack); - - if (!esids_match(pc, exec_base) && - !esids_match(stack, exec_base)) - slb_allocate_user(mm, exec_base); + for (i = 0; i < min(SLB_PRELOAD_NR, ti->slb_preload_nr); i++) + slb_allocate_user(mm, ti->slb_preload_ea[i]); } void slb_set_size(u16 size) @@ -536,11 +594,16 @@ long do_slb_fault(struct pt_regs *regs, unsigned long ea) return slb_allocate_kernel(ea, id); } else { struct mm_struct *mm = current->mm; + long err; if (unlikely(!mm)) return -EFAULT; - return slb_allocate_user(mm, ea); + err = slb_allocate_user(mm, ea); + if (!err) + preload_add(current_thread_info(), ea); + + return err; } }