From patchwork Sun May 20 00:43:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916962 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNYZ6lDKz9s3x for ; Sun, 20 May 2018 10:48:02 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aygoVXrm"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNYZ4GFpzDrTf for ; Sun, 20 May 2018 10:48:02 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aygoVXrm"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::241; helo=mail-pl0-x241.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aygoVXrm"; dkim-atps=neutral Received: from mail-pl0-x241.google.com (mail-pl0-x241.google.com [IPv6:2607:f8b0:400e:c01::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNSx5wgBzDqDv for ; Sun, 20 May 2018 10:44:01 +1000 (AEST) Received: by mail-pl0-x241.google.com with SMTP id c19-v6so6682915pls.6 for ; Sat, 19 May 2018 17:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v7GDsvzqwHiYS2IzEkQzIn+ohPI9+PPIU3WCj89n0Fk=; b=aygoVXrma5X6jxv/L3OFbviHu0rpk+QdxKmqqs4m7EWG2N+w9lVxKrwGYrgS7SGLeJ Ep9/mdBPYKCAWiPyTCJ4+rPEi+YIC46ZhOcn6SWWYh2q3YRHi8LxvxAKFo5ikX7G8w+0 NsC66i9PfxQpXhCj+9J7PcJ/sufvRaq6/ajO0cuBP3wMQ0dZwgb40k/SUt3r9vfzicC8 BSmNci4bUwn95aNInazJ9WaAzpAfEdKVo2hJnOr2IaDFAhWb+UJ4Qbb01Yw4o3FdMB6Z s1swNjyoO5XZksbl/PxMdUoT3IiOfoPOJ3tBbHTwiAMe4FB7H034za/IXZl6xsrO6CWk M5mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v7GDsvzqwHiYS2IzEkQzIn+ohPI9+PPIU3WCj89n0Fk=; b=fG/LnIDgZ4SsOJEGV9Zz8QlEpvSAFDpgCFD4R4mxkX0U+IPd1a2Wm2i7B788y9pLR6 4kHg6NteRBnXBs5jE2atCepdsdYjayzqIVXihtKiaEM8NSPVEoc1+aU3HFmSiy//C7kG ChgSSuz1ZBZTjbv6iUcgaSTWO7ftgtzlRqET84Gra16xHE4y5aHVSTd7GzA1XIkrq83P QZfXNu/3LLZcKoCybLdSpFYFHAmkw6cq3TftO9Zsq1O2iG0IqoKyT++UaJIviki0oVH4 IxDOj1jJ0g5JEXWUn68NwiGhI4mtF741nxLZEUgkNn4JGEZ9T+OZ7EGWOgDAIAfFSKpg duMw== X-Gm-Message-State: ALKqPwcKcz/fq7+O/07tmF7gPP6np/5uvVvPY6dREzZZjR/hNRKboXLm 3JXLzsKH2IMPOe1O8tnDxfQObA== X-Google-Smtp-Source: AB8JxZrsIWoLt96Egc6o5VakEQVwmSZl+akfMNxN1uBD3PsikPbUSjve1aKCKhl/P9cejyjGoNd07w== X-Received: by 2002:a17:902:8305:: with SMTP id bd5-v6mr4280293plb.13.1526777039520; Sat, 19 May 2018 17:43:59 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.43.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:43:59 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 1/7] powerpc/64s/radix: do not flush TLB on spurious fault Date: Sun, 20 May 2018 10:43:41 +1000 Message-Id: <20180520004347.19508-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" In the case of a spurious fault (which can happen due to a race with another thread that changes the page table), the default Linux mm code calls flush_tlb_page for that address. This is not required because the pte will be re-fetched. Hash does not wire this up to a hardware TLB flush for this reason. This patch avoids the flush for radix. From Power ISA v3.0B, p.1090: Setting a Reference or Change Bit or Upgrading Access Authority (PTE Subject to Atomic Hardware Updates) If the only change being made to a valid PTE that is subject to atomic hardware updates is to set the Refer- ence or Change bit to 1 or to add access authorities, a simpler sequence suffices because the translation hardware will refetch the PTE if an access is attempted for which the only problems were reference and/or change bits needing to be set or insufficient access authority. The nest MMU on POWER9 does not re-fetch the PTE after such an access attempt before faulting, so address spaces with a coprocessor attached will continue to flush in these cases. This reduces tlbies for a kernel compile workload from 0.95M to 0.90M. fork --fork --exec benchmark improved 0.5% (12300->12400). Signed-off-by: Nicholas Piggin Reviewed-by: Aneesh Kumar K.V --- Since v1: - Added NMMU handling arch/powerpc/include/asm/book3s/64/tlbflush.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 0cac17253513..ebf572ea621e 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -4,7 +4,7 @@ #define MMU_NO_CONTEXT ~0UL - +#include #include #include @@ -137,6 +137,16 @@ static inline void flush_all_mm(struct mm_struct *mm) #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #define flush_all_mm(mm) local_flush_all_mm(mm) #endif /* CONFIG_SMP */ + +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault +static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address) +{ + /* See ptep_set_access_flags comment */ + if (atomic_read(&vma->vm_mm->context.copros) > 0) + flush_tlb_page(vma, address); +} + /* * flush the page walk cache for the address */ From patchwork Sun May 20 00:43:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916963 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNcm1jDsz9s3x for ; Sun, 20 May 2018 10:50:48 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="maOH4Sg1"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNcl70dBzDrWY for ; Sun, 20 May 2018 10:50:47 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="maOH4Sg1"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="maOH4Sg1"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT01XFyzDrKQ for ; Sun, 20 May 2018 10:44:04 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id bi12-v6so6669838plb.12 for ; Sat, 19 May 2018 17:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=roXOSc5EZTH7bJwC37TDo8u04yWmpWGbslij326U2Fk=; b=maOH4Sg15bhkpR5SzN6RuemyiZK8Zus6dpUl7JLmwzvr0w192EalTa7E0sALKzJIfY C3pSX2wQAO5OrF3xjZkHAC46KRb0MSCkW9BwdXurgkNVIpNLEho7cUHLzKdTG0AKDCkN 2CW2GqcWd/H1bZc2YzEQwBSICT/crmFbsusE5v9gCAdGX1WhGilDIEAaGCe/xkOvHGrc wqvaXWUdS2gcOpQ68KXuOG3/RDOiJ8ozf+B5DwoVrfOecg+KfpiSEE4yWp1pk8LDPx3k o/gR1kkfVRHAw6jtBFTE2bYyg9Omw4RLl5fOfdQBvWRsSnWupokk4zQX4Whp5pmZuoBQ qkmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=roXOSc5EZTH7bJwC37TDo8u04yWmpWGbslij326U2Fk=; b=leQwLiZEWy9yzbXPfPZNuwTXImWJGGpm3dyrIZNmH/Om2KHGRchNgknaHgj/TVK+H6 92lfkhSOMp0L/eONklnaiZeVBflOv5fraY/AplmFMT/smQZS7SEXqQgyKmSsSvbU68f7 6aWl5oEdKoIh6ud2T77PanvIwivMaf5BnKIs4s8mjXN+c6QT3fGGT2ozQgI11JAaw7dC oYnSCmXCEgp5dTrfpE1/II214jwmLJidy2FsK89R5sGRXHWeBUpyS+rHQ5sdtyHnSOxB dX7Ape0Gb8X3eHqraqInY1natFgKnm6l7U1PpasoTBQkErQXN/08/jsH2vgpp4epbdYJ /fOg== X-Gm-Message-State: ALKqPweSQlLMKQc/nb3Tj3mf0rKvw1vIFHTCLzHygDq04pd4OzBKV/q8 41ukQgsMLzWVW9hj6IW9m6U7Gg== X-Google-Smtp-Source: AB8JxZrvXtmTFnulEByFhfFAfBEqObJd9FXP7WF38Cqhq+ifORBkxNw4K7HgjkIkIKBarMb5hoGm6g== X-Received: by 2002:a17:902:b58e:: with SMTP id a14-v6mr15236211pls.261.1526777041808; Sat, 19 May 2018 17:44:01 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.43.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:01 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 2/7] powerpc/64s/radix: reset mm_cpumask for single thread process when possible Date: Sun, 20 May 2018 10:43:42 +1000 Message-Id: <20180520004347.19508-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When a single-threaded process has a non-local mm_cpumask and requires a full PID tlbie invalidation, use that as an opportunity to reset the cpumask back to the current CPU we're running on. No other thread can concurrently switch to this mm, because it must have been given a reference to mm_users by the current thread before it can use_mm. mm_users can be asynchronously incremented (by mmget_not_zero), but those users must not do a use_mm, because existing convention already disallows it (sparc64 has reset mm_cpumask using this condition since the start of history, see arch/sparc/kernel/smp_64.c). This reduces tlbies for a kernel compile workload from 0.90M to 0.40M, tlbiels are increased from 20M to 22.5M because local pid flushes take 128 tlbiels vs 1 for global pid flush. This slows down context switch benchmark ping-ponging on different CPUs by 7.5%, because switching to the idle thread and back now requires a PID switch. There are ways this could be avoided, but for now it's simpler not to allow lazy PID switching. Reviewed-by: Balbir Singh Signed-off-by: Nicholas Piggin --- Since v1: - Fix switch_mm to be irq-safe (thanks Balbir) arch/powerpc/include/asm/mmu_context.h | 19 +++++++++ arch/powerpc/include/asm/tlb.h | 8 ++++ arch/powerpc/mm/tlb-radix.c | 57 +++++++++++++++++++------- 3 files changed, 70 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 1835ca1505d6..f26704371fdc 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -201,6 +202,24 @@ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next) static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { +#ifdef CONFIG_PPC_BOOK3S_64 + /* + * Under radix, we do not want to keep lazy PIDs around because + * even if the CPU does not access userspace, it can still bring + * in translations through speculation and prefetching. + * + * Switching away here allows us to trim back the mm_cpumask in + * cases where we know the process is not running on some CPUs + * (see mm/tlb-radix.c). + */ + if (radix_enabled() && mm != &init_mm) { + mmgrab(&init_mm); + tsk->active_mm = &init_mm; + switch_mm(mm, tsk->active_mm, tsk); + mmdrop(mm); + } +#endif + /* 64-bit Book3E keeps track of current PGD in the PACA */ #ifdef CONFIG_PPC_BOOK3E_64 get_paca()->pgd = NULL; diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index a7eabff27a0f..006fce98c403 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -76,6 +76,14 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } +static inline void mm_reset_thread_local(struct mm_struct *mm) +{ + WARN_ON(atomic_read(&mm->context.copros) > 0); + WARN_ON(!(atomic_read(&mm->mm_users) == 1 && current->mm == mm)); + atomic_set(&mm->context.active_cpus, 1); + cpumask_clear(mm_cpumask(mm)); + cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); +} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 5ac3206c51cc..d5593a78702a 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -504,6 +504,15 @@ void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmadd } EXPORT_SYMBOL(radix__local_flush_tlb_page); +static bool mm_is_singlethreaded(struct mm_struct *mm) +{ + if (atomic_read(&mm->context.copros) > 0) + return false; + if (atomic_read(&mm->mm_users) == 1 && current->mm == mm) + return true; + return false; +} + static bool mm_needs_flush_escalation(struct mm_struct *mm) { /* @@ -511,7 +520,9 @@ static bool mm_needs_flush_escalation(struct mm_struct *mm) * caching PTEs and not flushing them properly when * RIC = 0 for a PID/LPID invalidate */ - return atomic_read(&mm->context.copros) != 0; + if (atomic_read(&mm->context.copros) > 0) + return true; + return false; } #ifdef CONFIG_SMP @@ -525,12 +536,17 @@ void radix__flush_tlb_mm(struct mm_struct *mm) preempt_disable(); if (!mm_is_thread_local(mm)) { - if (mm_needs_flush_escalation(mm)) + if (mm_is_singlethreaded(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); - else + mm_reset_thread_local(mm); + } else if (mm_needs_flush_escalation(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + } else { _tlbie_pid(pid, RIC_FLUSH_TLB); - } else + } + } else { _tlbiel_pid(pid, RIC_FLUSH_TLB); + } preempt_enable(); } EXPORT_SYMBOL(radix__flush_tlb_mm); @@ -544,10 +560,13 @@ void radix__flush_all_mm(struct mm_struct *mm) return; preempt_disable(); - if (!mm_is_thread_local(mm)) + if (!mm_is_thread_local(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); - else + if (mm_is_singlethreaded(mm)) + mm_reset_thread_local(mm); + } else { _tlbiel_pid(pid, RIC_FLUSH_ALL); + } preempt_enable(); } EXPORT_SYMBOL(radix__flush_all_mm); @@ -644,10 +663,14 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, if (local) { _tlbiel_pid(pid, RIC_FLUSH_TLB); } else { - if (mm_needs_flush_escalation(mm)) + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else if (mm_needs_flush_escalation(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); - else + } else { _tlbie_pid(pid, RIC_FLUSH_TLB); + } } } else { bool hflush = false; @@ -802,13 +825,19 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, } if (full) { - if (!local && mm_needs_flush_escalation(mm)) - also_pwc = true; - - if (local) + if (local) { _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); - else - _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL: RIC_FLUSH_TLB); + } else { + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else { + if (mm_needs_flush_escalation(mm)) + also_pwc = true; + + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); + } + } } else { if (local) _tlbiel_va_range(start, end, pid, page_size, psize, also_pwc); From patchwork Sun May 20 00:43:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916964 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNgT2J5Hz9s3x for ; Sun, 20 May 2018 10:53:09 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNgT0dzGzDqlg for ; Sun, 20 May 2018 10:53:09 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::243; helo=mail-pl0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jKTfakWR"; dkim-atps=neutral Received: from mail-pl0-x243.google.com (mail-pl0-x243.google.com [IPv6:2607:f8b0:400e:c01::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT20mlVzDrTn for ; Sun, 20 May 2018 10:44:06 +1000 (AEST) Received: by mail-pl0-x243.google.com with SMTP id c41-v6so6671531plj.10 for ; Sat, 19 May 2018 17:44:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vixH0jmn3eXEDpLEJlRrdy4hsil6LS4CrZDFf+QRtD4=; b=jKTfakWRNPqUkfWKsaozy/aDlKI83/yz05tYuJzVihwzmSPRnoEdK4s1QfVGPH0L1e PmaHVbv0aaTVtCzV7HctzqFhgfxKYGwo9oBGVqlJnQx74Ris6EFi/btcjx5jqv9fqbOn qtBtRwtjQrsu5epacLO1DOdzgw3wf5ajwFCpKbzwHTsqzmVLv+dikFwJkwx/xr3d1jdg wlT/XFy+iCwKhsdAC5RdZC6s8qzGuuKKVkoRiaubBMnnvvLjcDphVflnKBcXNnFNlX8g GK0axXBU+ff/Mq+pOi99tpu8jNv06gZIbrkIKgH4iCE2aJ+FvpYPtw8cdeuC/5bdVd67 uHKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vixH0jmn3eXEDpLEJlRrdy4hsil6LS4CrZDFf+QRtD4=; b=L2izcuRcAUrSTjkta/uVRM70agxm15Jma2GY1mK+YFEn5Xj7KfcYjstQ/fHwu6iFGp 5z1Is0Z9w5AsSH26GFggDDX18J4r80i0vmnS9zvEQjWZkIL2NiRmCir0EuDCMcwAxL2O LrxXwTpd8AEBbDRt1IroYXVv2li2pnzoxlmUuirg6JzvN/r3Fy5prphHf7eC6I8tAbcQ WweDSF7LOtQ2eB6wa/B4v+0aWv9ZPQH5N6yz2Y2yM8kyc2S/ZDF0YXtero/r32EmRjYO W2Kn7IQNNAMxUi/FLxvYTKEAq4YUemfFpZOt3vadq/lJXGor8yHDIQeAACPTQSXz2L33 CgaA== X-Gm-Message-State: ALKqPwc8nm34BXgyE5Ayiubnh7mB6NaGjncohDM5H5N5Qqveqi6711ex UMSMpgLfyoXXjMKLE5wALkOO0A== X-Google-Smtp-Source: AB8JxZrc8Pw7xytXevMRmrC4mwqha0d04ml+vv2/BLs2V4kFBfXOC4wEaFQjc2daO5GzuBU3qm0sJQ== X-Received: by 2002:a17:902:206:: with SMTP id 6-v6mr7134626plc.294.1526777044067; Sat, 19 May 2018 17:44:04 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:03 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 3/7] powerpc/64s/radix: make single threaded mms always flush all translations from non-local CPUs Date: Sun, 20 May 2018 10:43:43 +1000 Message-Id: <20180520004347.19508-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Go one step further, if we're going to put a tlbie on the bus at all, make it count. Make any global invalidation from a single threaded mm do a full PID flush so the mm_cpumask can be reset. The tradeoff is that it will over-flush one time the local CPU's TLB if there was a small number of pages to flush that could be done with specific address tlbies. If the workload is invalidate-heavy enough for this to be a concern, this should be outweighed by the benefit that it can subsequently avoid the global flush. This reduces tlbies for a kernel compile workload from 0.40M to 0.18M, tlbiels are increased from 22.5M to 23.8M because local pid flushes take 128 tlbiels vs 1 for global pid flush. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 45 ++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index d5593a78702a..55f93d66c8d2 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -587,10 +587,16 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, return; preempt_disable(); - if (!mm_is_thread_local(mm)) - _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); - else + if (mm_is_thread_local(mm)) { _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } else { + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else { + _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); + } + } preempt_enable(); } @@ -659,14 +665,14 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, nr_pages > tlb_single_page_flush_ceiling); } - if (full) { + if (!local && mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else if (full) { if (local) { _tlbiel_pid(pid, RIC_FLUSH_TLB); } else { - if (mm_is_singlethreaded(mm)) { - _tlbie_pid(pid, RIC_FLUSH_ALL); - mm_reset_thread_local(mm); - } else if (mm_needs_flush_escalation(mm)) { + if (mm_needs_flush_escalation(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); } else { _tlbie_pid(pid, RIC_FLUSH_TLB); @@ -824,19 +830,17 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, nr_pages > tlb_single_page_flush_ceiling); } - if (full) { + if (!local && mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else if (full) { if (local) { _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } else { - if (mm_is_singlethreaded(mm)) { - _tlbie_pid(pid, RIC_FLUSH_ALL); - mm_reset_thread_local(mm); - } else { - if (mm_needs_flush_escalation(mm)) - also_pwc = true; + if (mm_needs_flush_escalation(mm)) + also_pwc = true; - _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); - } + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } } else { if (local) @@ -882,7 +886,12 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) if (mm_is_thread_local(mm)) { _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } else { - _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + if (mm_is_singlethreaded(mm)) { + _tlbie_pid(pid, RIC_FLUSH_ALL); + mm_reset_thread_local(mm); + } else { + _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); + } } preempt_enable(); From patchwork Sun May 20 00:43:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916968 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNkD1czLz9s3x for ; Sun, 20 May 2018 10:55:32 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F5LE34Bk"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNkC5RbLzDrTf for ; Sun, 20 May 2018 10:55:31 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F5LE34Bk"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c01::241; helo=mail-pl0-x241.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F5LE34Bk"; dkim-atps=neutral Received: from mail-pl0-x241.google.com (mail-pl0-x241.google.com [IPv6:2607:f8b0:400e:c01::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT42glmzDrWR for ; Sun, 20 May 2018 10:44:08 +1000 (AEST) Received: by mail-pl0-x241.google.com with SMTP id az12-v6so6678987plb.8 for ; Sat, 19 May 2018 17:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hvag+NSnjD+bOjB7jAHzwIebQMpDE3JgxbO5+IjbYFQ=; b=F5LE34BkGqMDsKDbjIRqecYiNSFc0QoFJBlZoilDoF7CbQNUWomFx6GU8Wx0KJWdcp /fmMsNRw0WdYCGjAOsRguzKfJjaRv96udoBpv+4x+/oUFB7zUvQV9hoFmba4f3OG8tHS X44UgOLmyxr61en5MgXw30vQhLHLHQuU3uitrjhKtnOwbbL2JbORPL6oBhWIgv68u694 Tc22m5ZX3Qfr/gzZVHejUpUuZqdHk7x9Ul0uO9XtEP20/CAtDcS6bBnjD2JqxD/qgALy NeBcvzK4QBXY4CdO9HBfVnIOpyr8uY/JSn9YQ8K5xWAlDY8EEMX3HrI8qLv2jBzm8RYg knVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hvag+NSnjD+bOjB7jAHzwIebQMpDE3JgxbO5+IjbYFQ=; b=OrHKQHTQ5eUUOnFJMn1X1w8xa8YZc9ZE1WvTieuDUlJqArBBPYn2xcEMb3yqF3s++j TDkrkTJI+ilvl3jzNgBTJHzueqc2otAHIPvJ3ijsZU/cdSYZGvFZN0lOnvcwIoMgAK/M h/SrRH88TGHzzKb0DcNUm5zLLO3xpzx1VQPHQ/6RI1jM5n549gv8JXN1xZzYrIlzuRf3 dGW+oP0R7oglo/33hiQ7f6zlVab87GvFLyGFG/ruAPuVJkNSPbQtejF/EAFpNuZbvyiy UR9/BkgKV+RjwVyjFMLYv1ZrKNWrvtYh3PHjxYkLcCwaZhTlV9Rk2PvXDs5PCAWdOEDU NSwA== X-Gm-Message-State: ALKqPwctZtnhKOXUEKNz34lLlHvuS2GIJWc+AZHFZ9BWOLz7DmBAGaAC An8IqRwQdvSrMaI+/ce8b6AYYw== X-Google-Smtp-Source: AB8JxZplPyBMeCpk/opJIKz8austG3jWrlCFP7ij1KJ8vA/Cb+kUbiVSuZr/KXkAlSBulu3ilvxkXQ== X-Received: by 2002:a17:902:a4:: with SMTP id a33-v6mr14937049pla.346.1526777046505; Sat, 19 May 2018 17:44:06 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:05 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 4/7] powerpc/64s/radix: make ptep_get_and_clear_full non-atomic for the full case Date: Sun, 20 May 2018 10:43:44 +1000 Message-Id: <20180520004347.19508-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This matches other architectures, when we know there will be no further accesses to the address (e.g., for teardown), page table entries can be cleared non-atomically. The comments about NMMU are bogus: all MMU notifiers (including NMMU) are released at this point, with their TLBs flushed. An NMMU access at this point would be a bug. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 705193e7192f..fcd92f9b6ec0 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -176,14 +176,8 @@ static inline pte_t radix__ptep_get_and_clear_full(struct mm_struct *mm, unsigned long old_pte; if (full) { - /* - * If we are trying to clear the pte, we can skip - * the DD1 pte update sequence and batch the tlb flush. The - * tlb flush batching is done by mmu gather code. We - * still keep the cmp_xchg update to make sure we get - * correct R/C bit which might be updated via Nest MMU. - */ - old_pte = __radix_pte_update(ptep, ~0ul, 0); + old_pte = pte_val(*ptep); + *ptep = __pte(0); } else old_pte = radix__pte_update(mm, addr, ptep, ~0ul, 0, 0); From patchwork Sun May 20 00:43:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916969 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNn33Km2z9s3x for ; Sun, 20 May 2018 10:57:59 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="koDRr6dY"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNn318nrzDqL3 for ; Sun, 20 May 2018 10:57:59 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="koDRr6dY"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::244; helo=mail-pf0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="koDRr6dY"; dkim-atps=neutral Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT71T8RzDrWN for ; Sun, 20 May 2018 10:44:11 +1000 (AEST) Received: by mail-pf0-x244.google.com with SMTP id c10-v6so5392936pfi.12 for ; Sat, 19 May 2018 17:44:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UY9HrfPTXR1psieAxhqjlNapy7ff6XVJ6CqmVXYfkJs=; b=koDRr6dYpYa6hJbZVc7kBLAHBD+jNj24eSkiLMHrfh6w5DXLfCwFSz0qEMgzayMfuf c3o/gVTm4x8DO4DvhhS7+goHBaHuOL1vwE2el4MOIDZWvUhFila6U8EzLZDFeUJO4ksk cfBxVbLzagQ3crIV6Qo0vW4bamTIE436RGmdnh2aS27is1VKfx0kclrMn0KNIf8KTXGe +00mvbmtqnnYd5w6yI989Ug6H89Ro7WrrGtVuf/NTlDPZeMuGREOIEzpo+YkYRbyTMKT 4dmgCa4glSkX7Fh7DcTOKI6OtaCg9c+VWRXMTD1m0LViihzE73Dtb2Sf4QqzSWyNbN5L YA2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UY9HrfPTXR1psieAxhqjlNapy7ff6XVJ6CqmVXYfkJs=; b=Nan0QYioz2S8/MMtutnbePlPv4f1HnCi8gvp9hOJ+zj7RH2ItwAKsC4c2mw5hdANdC Ay4BblOg8utH2Yd8puXoFJTrijiNu6ulUFIIm309xqC7a7G9v+d6jei6Y9ygjRQOs4yw I1KXtFvQ31gd+A9ahIZcCzAvZyxy2+7bR+wmPlsHwpRj3sgw94ybaNOt9B1EsDEkwPgr Y6m/ZXbWSHqgoVGdh06s5TH1ok8kjTHszq7OD//TihX1T7flyxaDjTrARFJhAyzBnUk7 5l6PMpd05zrQGJu/gd55HJnaoS22ylZD2OWF9W1MBgW0SuT8WGRXp3tMiBFXDvEmjp9o 8N7A== X-Gm-Message-State: ALKqPwcehb3irTVg5+VzNXfozgz8FWL1Q6/+RB5DJHUnrRiXZ1nFBTnM p/lMP4xCr8fVjpMXphgee1hLBw== X-Google-Smtp-Source: AB8JxZpe6ypH/feSSxPb0Y6DlcJCxk/z+/rHHAo8pYw+wtHB3A9fjmjugGgRtfLaH8eD386Qx9jWSA== X-Received: by 2002:a62:23d7:: with SMTP id q84-v6mr14704079pfj.31.1526777048789; Sat, 19 May 2018 17:44:08 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:08 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 5/7] powerpc/64s/radix: optimise pte_update Date: Sun, 20 May 2018 10:43:45 +1000 Message-Id: <20180520004347.19508-6-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implementing pte_update with pte_xchg (which uses cmpxchg) is inefficient. A single larx/stcx. works fine, no need for the less efficient cmpxchg sequence. Then remove the memory barriers from the operation. There is a requirement for TLB flushing to load mm_cpumask after the store that reduces pte permissions, which is moved into the TLB flush code. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 25 +++++++++++----------- arch/powerpc/mm/mmu_context.c | 6 ++++-- arch/powerpc/mm/tlb-radix.c | 11 +++++++++- 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index fcd92f9b6ec0..ab4ed61a74fa 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -127,20 +127,21 @@ extern void radix__mark_initmem_nx(void); static inline unsigned long __radix_pte_update(pte_t *ptep, unsigned long clr, unsigned long set) { - pte_t pte; - unsigned long old_pte, new_pte; - - do { - pte = READ_ONCE(*ptep); - old_pte = pte_val(pte); - new_pte = (old_pte | set) & ~clr; - - } while (!pte_xchg(ptep, __pte(old_pte), __pte(new_pte))); - - return old_pte; + __be64 old_be, tmp_be; + + __asm__ __volatile__( + "1: ldarx %0,0,%3 # pte_update\n" + " andc %1,%0,%5 \n" + " or %1,%1,%4 \n" + " stdcx. %1,0,%3 \n" + " bne- 1b" + : "=&r" (old_be), "=&r" (tmp_be), "=m" (*ptep) + : "r" (ptep), "r" (cpu_to_be64(set)), "r" (cpu_to_be64(clr)) + : "cc" ); + + return be64_to_cpu(old_be); } - static inline unsigned long radix__pte_update(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long clr, diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 0ab297c4cfad..f84e14f23e50 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -57,8 +57,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * in switch_slb(), and/or the store of paca->mm_ctx_id in * copy_mm_to_paca(). * - * On the read side the barrier is in pte_xchg(), which orders - * the store to the PTE vs the load of mm_cpumask. + * On the other side, the barrier is in mm/tlb-radix.c for + * radix which orders earlier stores to clear the PTEs vs + * the load of mm_cpumask. And pte_xchg which does the same + * thing for hash. * * This full barrier is needed by membarrier when switching * between processes after store to rq->curr, before user-space diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 55f93d66c8d2..b419702b1ba6 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -535,6 +535,11 @@ void radix__flush_tlb_mm(struct mm_struct *mm) return; preempt_disable(); + /* + * Order loads of mm_cpumask vs previous stores to clear ptes before + * the invalidate. See barrier in switch_mm_irqs_off + */ + smp_mb(); if (!mm_is_thread_local(mm)) { if (mm_is_singlethreaded(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); @@ -560,6 +565,7 @@ void radix__flush_all_mm(struct mm_struct *mm) return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (!mm_is_thread_local(mm)) { _tlbie_pid(pid, RIC_FLUSH_ALL); if (mm_is_singlethreaded(mm)) @@ -587,6 +593,7 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); } else { @@ -655,6 +662,7 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { local = true; full = (end == TLB_FLUSH_ALL || @@ -820,6 +828,7 @@ static inline void __radix__flush_tlb_range_psize(struct mm_struct *mm, return; preempt_disable(); + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { local = true; full = (end == TLB_FLUSH_ALL || @@ -882,7 +891,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) /* Otherwise first do the PWC, then iterate the pages. */ preempt_disable(); - + smp_mb(); /* see radix__flush_tlb_mm */ if (mm_is_thread_local(mm)) { _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true); } else { From patchwork Sun May 20 00:43:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916970 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNrL6YNsz9s3x for ; Sun, 20 May 2018 11:00:50 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AQyaioUr"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNrL4qzmzDrWQ for ; Sun, 20 May 2018 11:00:50 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AQyaioUr"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::244; helo=mail-pg0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AQyaioUr"; dkim-atps=neutral Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNT93dVDzDrWR for ; Sun, 20 May 2018 10:44:13 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id 11-v6so3193982pge.2 for ; Sat, 19 May 2018 17:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=R+lFYoroCbtw3G0UsBZurh+Z8+Vpx4wQpt9kugh/0Lo=; b=AQyaioUrz9bKPkf6vIt5oDF8GDkLI/8pyCU9WjZBSpkVlgxcLbeSzxlusDQhIwV0RH mFGKPwzgdG7Cj4QLwXfBwTeNI/YhHM2qlkTL30W1jPyCYEegS4cmbZAaIDMc0ThueTEQ s+ubGNEfYJeOMtrFmFfA/Y5iYteFE1lYibCpRMmLX9+wFxaiiZc3CZAykXLmgUGAYYrU ORjlNvWndbdSEv5L0IXmyXyDpFplpUGGOpHclyUApAmgAYvSPYJSXb4G7uawtY32S6DB kGpPbUAOe2imAAahRcbrQd3nTPkQe0jx4AEFhLLN45nfqmgVhvlsjCw3ogaGxY04oQyB qSlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=R+lFYoroCbtw3G0UsBZurh+Z8+Vpx4wQpt9kugh/0Lo=; b=VBTruXIdWMZd0cnaOwOr880MIVPG82JvCm9eshcg40U4uNb6/2Qvn0EN5GmKusGhSy 0hDIf5K0WOmPOhLwNfe3t3gyHSbdx1AsMgm711wb2/UNNVFDWOH3MF9dm/bvEMp7zc32 35Cp2CixeU12j9Wy3eNQ/Q7bzDARortbwoP7KVJm1vghuIK9k4afPDwpyZgfDhw1gxBQ iycrKkpYNjT0Fm0CKISD8qVBeT63mpFoKf8zHCqdt5Si8F8HlWPeW+TJ2lTxhQ20Pd3u 3TpSDgKzqmRB7dOL06Tvkl+yNk893UB4CpSE9RttX659puRYKaHMLdfTZkOU1wd9w5zL XmhA== X-Gm-Message-State: ALKqPwfQ4dW3u4vVh3Bw/jKRCl3S6ppFL0D7yB0exAJXxwiJ2BD+4MFN ap3B+X/WWzdKv3iu0S9/WH5/yw== X-Google-Smtp-Source: AB8JxZoFjK1SBR1O/Cf7GRz2Jpwa8J7s+aFFHd3G2s9sQajStul7wsE6JdY+jHuTFpZ/+eE4QhCH3Q== X-Received: by 2002:a62:1f03:: with SMTP id f3-v6mr14897126pff.213.1526777051019; Sat, 19 May 2018 17:44:11 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:10 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 6/7] powerpc/64s/radix: prefetch user address in update_mmu_cache Date: Sun, 20 May 2018 10:43:46 +1000 Message-Id: <20180520004347.19508-7-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Prefetch the faulting address in update_mmu_cache to give the page table walker perhaps 100 cycles head start as locks are dropped and the interrupt completed. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/mem.c | 4 +++- arch/powerpc/mm/pgtable-book3s64.c | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index c3c39b02b2ba..8cecda4bd66a 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -509,8 +509,10 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, */ unsigned long access, trap; - if (radix_enabled()) + if (radix_enabled()) { + prefetch((void *)address); return; + } /* We only want HPTEs for linux PTEs that have _PAGE_ACCESSED set */ if (!pte_young(*ptep) || address >= TASK_SIZE) diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 994492453f0e..7ce889a7e5ce 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -145,7 +145,8 @@ pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd) { - return; + if (radix_enabled()) + prefetch((void *)addr); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Sun May 20 00:43:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 916971 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40pNvC4gwCz9s3x for ; Sun, 20 May 2018 11:03:19 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LR34MSbm"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40pNvC2xdFzDqlg for ; Sun, 20 May 2018 11:03:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LR34MSbm"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::244; helo=mail-pf0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LR34MSbm"; dkim-atps=neutral Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40pNTC2yCdzDrWN for ; Sun, 20 May 2018 10:44:15 +1000 (AEST) Received: by mail-pf0-x244.google.com with SMTP id a14-v6so5409072pfi.1 for ; Sat, 19 May 2018 17:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SgRg8q3b2CEBO5sWSm132MAp5jWP3BXhwBnhRmDiWlo=; b=LR34MSbmEynET5OKEI1Ugyj9RhMWH7EldThCfDw+p7oib28pxkFg6+bZj3FnvBJbXj bs2Lrxc58Hdge0kL/z1RiTx6ZQR7XZEAP+A+LRxO5ffdEkSkDijOV3V3ONNHaGFiOv3p XIBY0VJRizlQaSmoj1GkjLts6gOnWVaU/aFql11UxD+Lu0mCLCPU8h1Cug0UXTQWCAsq kY6cXJRcntg2Ab5K2Bk0xnguEiIbW8AeRGXWsujv4MtVuDaGZh+tEmGWF5hjjzsRd8ZH tvazC55RNuXPyzdYcgKpTsGsnyq1QLv71Ce2QBVAh21xPMarVRL9Oh0V1ATExsShr5Ci JSmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SgRg8q3b2CEBO5sWSm132MAp5jWP3BXhwBnhRmDiWlo=; b=Jae7Y3bZ8rznOuMjIxfrmyTJmwYH1K+PM6casbu4SbhQB9kpeEVlToRf3URgLb2BZ3 XSmfoDuQ+8U61x19BQhtRpV4MRZmMJB6fMebJ7i5KlAQ4zPXsPWaXJ3JFtRCpLX979Zu LJQWdOE6b1DtDj/0nZEkZLDWxgfp5vhrxxWlWYR9lw30k7F5JbSUJEj/144aiALFDeiZ TBclSUQ1PmLseVLIhbeG2uOVdOzLT93OSGz1tIcCPeBXfpd+TcHafHrfN/mcmNDchMFM bWtYZM2qHaOk38grhvfsyh2vHyx3NptjEfXR7i21fu86BIiQ1knDGZuuWd/PvyCceOhT R1qg== X-Gm-Message-State: ALKqPwfAcumoHjJyh2BvGz7DPSvUtkI985b6uKUQByJXKR8aCrK5D/MF j2cc7W3oHgZBesZelGjfXUZlPw== X-Google-Smtp-Source: AB8JxZoIJS/c2Kmaww087lxRnJ8M/Kd3lgHYInETRue0PIrKfFmWw+rGOUgEAi7xKhoQ4nhZOFzgEQ== X-Received: by 2002:a62:f55b:: with SMTP id n88-v6mr14884038pfh.208.1526777053362; Sat, 19 May 2018 17:44:13 -0700 (PDT) Received: from roar.au.ibm.com (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id x25-v6sm16356062pfn.11.2018.05.19.17.44.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 May 2018 17:44:12 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 7/7] powerpc/64s/radix: avoid ptesync after set_pte and ptep_set_access_flags Date: Sun, 20 May 2018 10:43:47 +1000 Message-Id: <20180520004347.19508-8-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180520004347.19508-1-npiggin@gmail.com> References: <20180520004347.19508-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The ISA suggests ptesync after setting a pte, to prevent a table walk initiated by a subsequent access from missing that store and causing a spurious fault. This is an architectual allowance that allows an implementation's page table walker to be incoherent with the store queue. However there is no correctness problem in taking a spurious fault in userspace -- the kernel copes with these at any time, so the updated pte will be found eventually. Spurious kernel faults on vmap memory must be avoided, so a ptesync is put into flush_cache_vmap. On POWER9 so far I have not found a measurable window where this can result in more minor faults, so as an optimisation, remove the costly ptesync from pte updates. If an implementation benefits from ptesync, it would be better to add it back in update_mmu_cache, so it's not done for things like fork(2). fork --fork --exec benchmark improved 5.2% (12400->13100). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/radix.h | 2 -- arch/powerpc/include/asm/cacheflush.h | 13 +++++++++++++ 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index ab4ed61a74fa..cc9437a542cc 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -210,7 +210,6 @@ static inline void radix__ptep_set_access_flags(struct mm_struct *mm, __radix_pte_update(ptep, 0, new_pte); } else __radix_pte_update(ptep, 0, set); - asm volatile("ptesync" : : : "memory"); } static inline int radix__pte_same(pte_t pte_a, pte_t pte_b) @@ -227,7 +226,6 @@ static inline void radix__set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, int percpu) { *ptep = pte; - asm volatile("ptesync" : : : "memory"); } static inline int radix__pmd_bad(pmd_t pmd) diff --git a/arch/powerpc/include/asm/cacheflush.h b/arch/powerpc/include/asm/cacheflush.h index 11843e37d9cf..e9662648e72d 100644 --- a/arch/powerpc/include/asm/cacheflush.h +++ b/arch/powerpc/include/asm/cacheflush.h @@ -26,6 +26,19 @@ #define flush_cache_vmap(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) +#ifdef CONFIG_BOOK3S_64 +/* + * Book3s has no ptesync after setting a pte, so without this ptesync it's + * possible for a kernel virtual mapping access to return a spurious fault + * if it's accessed right after the pte is set. The page fault handler does + * not expect this type of fault. flush_cache_vmap is not exactly the right + * place to put this, but it seems to work well enough. + */ +#define flush_cache_vmap(start, end) do { asm volatile("ptesync"); } while (0) +#else +#define flush_cache_vmap(start, end) do { } while (0) +#endif + #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 extern void flush_dcache_page(struct page *page); #define flush_dcache_mmap_lock(mapping) do { } while (0)