From patchwork Thu Sep 7 14:51:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811060 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3R91sm1z9s81 for ; Fri, 8 Sep 2017 00:55:33 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Cjo61LHJ"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3R90WKvzDrWs for ; Fri, 8 Sep 2017 00:55:33 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Cjo61LHJ"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::242; helo=mail-pf0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Cjo61LHJ"; dkim-atps=neutral Received: from mail-pf0-x242.google.com (mail-pf0-x242.google.com [IPv6:2607:f8b0:400e:c00::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3MP6hDFzDrWb for ; Fri, 8 Sep 2017 00:52:17 +1000 (AEST) Received: by mail-pf0-x242.google.com with SMTP id a2so4609843pfj.4 for ; Thu, 07 Sep 2017 07:52:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wUW++PagQ4UzHZDXfLcVZsqs8tvSNWgy0gDgIcHJ170=; b=Cjo61LHJyZ39KXo/j8Cbhc+yk5KV9OFWv10xuGgcnIBHHK7iqYR/ka8rmjzofm8nyQ 4Pzd/RNUSzYf+pW982Fo90DYrsdPI0xGs+4S/9JvWPlmRKKGwx3b7/CEStms9d+gkn/G MEd6m2379bJLROY/ThNQ70ETWfTXUjo3NvD13cmTNi0xeQn20pTGcUMJYfwQd8y3Nxgw XTvJVad7UkT7XegnJAR5P8nKfndegRtkGn481pUfq6CtO2XjuJ6f9UDLre0o/m0WH3yI TarR1O5d3rUTe/PA8oY5ukNtiQvjPqzNfOBuOUh2n6L7yuard+s2ypXh1RWml6QxdJvn wUwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wUW++PagQ4UzHZDXfLcVZsqs8tvSNWgy0gDgIcHJ170=; b=IH/SC5UIXXmH5JwkN74ZlHliWT6lp27FeHGMgvH2rE6eqltNGu6udGG8Ca4jCo/fsw YkQje/tPVjAS+/nnlW5iAVQ2muAyBMXRYY+3Cjd/dRvb0/FZvP2UYzRZgh7vwoAvqBoH gaUkDq9JDJF/WexyOfQvakrHOKK/y2FJPyx7MvENah2g5Fk1nNRM5xLII3Q0Q7nAXhb4 HwaQJCC+EyzrzvEViiD0mi3NkAtL7cFJo+3+rDADnVuhksWNbsfYnhWNU0V799RlAGTL HE4/fAZXRCKjOAbZxacyFb67lDG0YagGYTrv1yjs7iHnUvxxJQEqDpUklziTHRGFwlVS rhDA== X-Gm-Message-State: AHPjjUg3FQILEyuA5SsWXeGM1cV54Yt1dvGpla5L9OhjPZZ5bN0F621U 3WGIwgKdztn4M519 X-Google-Smtp-Source: ADKCNb52WyOKqlK2d1rujZMpAwWvHFoFWNB7+9lsSvGA3Kj4HpI0KB6DpOjN/emSVBDxXyJ2y5zVXw== X-Received: by 10.101.73.198 with SMTP id t6mr3033508pgs.382.1504795935594; Thu, 07 Sep 2017 07:52:15 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:14 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/8] powerpc/64s/radix: Fix theoretical process table entry cache invalidation Date: Fri, 8 Sep 2017 00:51:41 +1000 Message-Id: <20170907145148.24398-2-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" According to the architecture, the process table entry cache must be flushed with RIC=2 tlbies. This problem doesn't hit in existing implementations that do not cache process table entries over mtpid. The PID is only destroyed and re-used after all CPUs have switched away from the mm, guaranteeing its entry is not cached anywhere. But this is not generally safe according to the ISA. Fix this by clearing the process table entry before the final flush (which is always a RIC=2 flush that invalidates the process table entry cache). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/mmu_context.h | 4 ++++ arch/powerpc/mm/mmu_context_book3s64.c | 23 ++++++++++++++++++----- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 309592589e30..0a70221adcf7 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -118,9 +118,13 @@ static inline void arch_dup_mmap(struct mm_struct *oldmm, { } +#ifndef CONFIG_PPC_BOOK3S_64 static inline void arch_exit_mmap(struct mm_struct *mm) { } +#else +extern void arch_exit_mmap(struct mm_struct *mm); +#endif static inline void arch_unmap(struct mm_struct *mm, struct vm_area_struct *vma, diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c index 05e15386d4cb..feb3f43195c2 100644 --- a/arch/powerpc/mm/mmu_context_book3s64.c +++ b/arch/powerpc/mm/mmu_context_book3s64.c @@ -216,19 +216,32 @@ void destroy_context(struct mm_struct *mm) #ifdef CONFIG_SPAPR_TCE_IOMMU WARN_ON_ONCE(!list_empty(&mm->context.iommu_group_mem_list)); #endif + if (radix_enabled()) + WARN_ON(process_tb[mm->context.id].prtb0 != 0); + else + subpage_prot_free(mm); + destroy_pagetable_page(mm); + __destroy_context(mm->context.id); + mm->context.id = MMU_NO_CONTEXT; +} + +void arch_exit_mmap(struct mm_struct *mm) +{ if (radix_enabled()) { /* * Radix doesn't have a valid bit in the process table * entries. However we know that at least P9 implementation * will avoid caching an entry with an invalid RTS field, * and 0 is invalid. So this will do. + * + * This runs before the "fullmm" tlb flush in exit_mmap, + * which does a RIC_FLUSH_ALL to clear the process table + * entry. No barrier required here after the store because + * this process will do the invalidate, which starts with + * ptesync. */ process_tb[mm->context.id].prtb0 = 0; - } else - subpage_prot_free(mm); - destroy_pagetable_page(mm); - __destroy_context(mm->context.id); - mm->context.id = MMU_NO_CONTEXT; + } } #ifdef CONFIG_PPC_RADIX_MMU From patchwork Thu Sep 7 14:51:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811061 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3TY5HLCz9s81 for ; Fri, 8 Sep 2017 00:57:37 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NmjH8Wfz"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3TY47NdzDrWl for ; Fri, 8 Sep 2017 00:57:37 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NmjH8Wfz"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::242; helo=mail-pf0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="NmjH8Wfz"; dkim-atps=neutral Received: from mail-pf0-x242.google.com (mail-pf0-x242.google.com [IPv6:2607:f8b0:400e:c00::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3MT298KzDrXg for ; Fri, 8 Sep 2017 00:52:21 +1000 (AEST) Received: by mail-pf0-x242.google.com with SMTP id h4so1228396pfk.0 for ; Thu, 07 Sep 2017 07:52:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+ZPfAMcqXapAjldd/laEzBZTdN/wbTO5QIQ9HBhCkQI=; b=NmjH8Wfz668NU1Lr7uFplWcr79NM3opLoKSoVOJMOp6aMGj5YotGFNf1Il8rDLy+oe TYvNpj2ACXQEX15LJ/cPk+18QYiWv2GUilKUdUbPfBckdr88TNwnDJEZUM1SpDAWG0Uz 6eNuDtnygZvr6iusF3nV2BYkYWX40FxLjxfw+O7IR7f7aThaN9mJ2VrYI9bt4zTet4tc JgyWN68Ew6FL8ixQHJVYy4syM1Jnd4zfJeeGoydo30hF5+/NsaKJKr1Hgdq8Oo6sd/aJ Onu3rEKdpRES2R8lfMjudL9V7iIalRFjxPvQpT/6fpyQGHr/1WaKpA+rcLAgR1yI124o jzBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+ZPfAMcqXapAjldd/laEzBZTdN/wbTO5QIQ9HBhCkQI=; b=QCH9kTpEhRkcx7LhAOGBTGjCNwgmSU3gq7hsp1SWCOAERBgMt2fJWfT1HOo6PRaPmj 4YSFfeOunEqQuV1ys+MrIJB0X8asW+68O1gKBxoSl6pzO5DB7isGZGCOuaE5R4l1yjk2 xKx54Zjve1S5MlaNfU6KzkFl+0CYahhpe4M4pPA3ZnSWsApdYVczsWth9Ty0akj2aRgk pU4/jkp6CccpjsEBJhDmxYF5+r4lxgBazIoq5wMQpMPxdPq04i01k2I0qx34YxaqxDQL foyfwCms8oqkHzDy/t3H9n99wN9pxCI848oZzhzcmWAcTZb2mvScrsCab7fNR8cKTnW7 xPMQ== X-Gm-Message-State: AHPjjUiEtVNUH3ix0EctYKIcpkwERXrS+CPzTWJ8ZrDtpGeL/X6i+rmE MhUDfiZiBdrEcgA2 X-Google-Smtp-Source: ADKCNb40y624lVjDeMQfzwvn34BN+OckAcxTgwcUsBubehQzcKsSTPmwLPV4D+jsVFk6WCWphN/qpQ== X-Received: by 10.84.133.73 with SMTP id 67mr3289301plf.378.1504795939200; Thu, 07 Sep 2017 07:52:19 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:18 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/8] powerpc/64s/radix: tlbie improve preempt handling Date: Fri, 8 Sep 2017 00:51:42 +1000 Message-Id: <20170907145148.24398-3-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Preempt should be consistently disabled for mm_is_thread_local tests, so bring the rest of these under preempt_disable(). Preempt does not need to be disabled for the mm->context.id tests, which allows simplification and removal of gotos. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 47 +++++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 25 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index b3e849c4886e..1ed61baf58da 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -186,16 +186,15 @@ void radix__flush_tlb_mm(struct mm_struct *mm) { unsigned long pid; - preempt_disable(); pid = mm->context.id; if (unlikely(pid == MMU_NO_CONTEXT)) - goto no_context; + return; + preempt_disable(); if (!mm_is_thread_local(mm)) _tlbie_pid(pid, RIC_FLUSH_TLB); else _tlbiel_pid(pid, RIC_FLUSH_TLB); -no_context: preempt_enable(); } EXPORT_SYMBOL(radix__flush_tlb_mm); @@ -204,16 +203,15 @@ static void radix__flush_all_mm(struct mm_struct *mm) { unsigned long pid; - preempt_disable(); pid = mm->context.id; if (unlikely(pid == MMU_NO_CONTEXT)) - goto no_context; + return; + preempt_disable(); if (!mm_is_thread_local(mm)) _tlbie_pid(pid, RIC_FLUSH_ALL); else _tlbiel_pid(pid, RIC_FLUSH_ALL); -no_context: preempt_enable(); } @@ -229,15 +227,14 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, unsigned long pid; unsigned long ap = mmu_get_ap(psize); - preempt_disable(); pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) - goto bail; + return; + preempt_disable(); if (!mm_is_thread_local(mm)) _tlbie_va(vmaddr, pid, ap, RIC_FLUSH_TLB); else _tlbiel_va(vmaddr, pid, ap, RIC_FLUSH_TLB); -bail: preempt_enable(); } @@ -322,46 +319,44 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, { unsigned long pid; unsigned long addr; - int local = mm_is_thread_local(mm); + bool local; unsigned long ap = mmu_get_ap(psize); unsigned long page_size = 1UL << mmu_psize_defs[psize].shift; - - preempt_disable(); pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) - goto err_out; + return; + preempt_disable(); + local = mm_is_thread_local(mm); if (end == TLB_FLUSH_ALL || (end - start) > tlb_single_page_flush_ceiling * page_size) { if (local) _tlbiel_pid(pid, RIC_FLUSH_TLB); else _tlbie_pid(pid, RIC_FLUSH_TLB); - goto err_out; - } - for (addr = start; addr < end; addr += page_size) { + } else { + for (addr = start; addr < end; addr += page_size) { - if (local) - _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); - else - _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + if (local) + _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); + else + _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + } } -err_out: preempt_enable(); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) { - int local = mm_is_thread_local(mm); unsigned long ap = mmu_get_ap(mmu_virtual_psize); unsigned long pid, end; - + bool local; pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) - goto no_context; + return; /* 4k page size, just blow the world */ if (PAGE_SIZE == 0x1000) { @@ -369,6 +364,8 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) return; } + preempt_disable(); + local = mm_is_thread_local(mm); /* Otherwise first do the PWC */ if (local) _tlbiel_pid(pid, RIC_FLUSH_PWC); @@ -383,7 +380,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) else _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); } -no_context: + preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Thu Sep 7 14:51:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811062 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3WW2DsTz9s81 for ; Fri, 8 Sep 2017 00:59:19 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3WW14c6zDrYt for ; Fri, 8 Sep 2017 00:59:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::244; helo=mail-pg0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3MX6qLlzDrWX for ; Fri, 8 Sep 2017 00:52:24 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id v82so4426355pgb.1 for ; Thu, 07 Sep 2017 07:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UDIibrf762Cv/CDNL2CMxskCyimmje6rG63nwqMD0cM=; b=DUJcPvvfIi+z/bhswggtfW25oJ8iip+b12WI+OblQSZsRN00ECbW++iuqqKpPo83L8 s0Zzeh6T/cHxcYJuyXxn3B8UIaccvqFcyrgYgS7fRCHTGqEszD5fNqlTpxP+V97lUrsW Fv1w9VMXCBZ5WicqLTUjV0RHms2gsgxlN9RUB6dpF1mtwVDjaiHgM1YullATrOaWZR6a AK11j6DWmoOtm9bIBF0HNMJUQSTNwTz9jLYPR2hiFe8ZG1OffiY4VChaAFmX58MLWtRV 56zhpIFekuDGYnH2JxYr7/E/WpbxIrcOx61n2VSTYHo2Y5qdwlyjEZ1dFWNe+GnodB7b wajg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UDIibrf762Cv/CDNL2CMxskCyimmje6rG63nwqMD0cM=; b=e0DPCKT4RIV8F9d2CvpIOtRIT7LMwfyqiFeU4F7bS2MM0lYycMCIxnWrntj2gSXSbR S8yME58TKAk1qcmoQY+kAOzPF3ltZigU+U11Kv7bQe2QANfPBiDzyJKtiLHXamgvQY4z ZUCuVBjtDYesmDZn2myViZeN6FRthg711aEPn3qqzFRTfXJDW/d0IKYDg7uqIUaYuaha +J6yh3dG8e7V/+hBnZRVSHmtIl9oKVNHJ9rPyFMJebSj4UoFPCXAEZtmcB9aGFxb92w8 oDAu7mYiYB93OBmpAGdbjQPwDSi4bihmOKn+oJDMjqLtTgnt2D01h/e382m0o6k9MtWY fMVw== X-Gm-Message-State: AHPjjUjk7zBy9YXMYJAL/gfS6pGm1UI+kJ6QuzQ/TZwfG6bP0q+eEAEG nsFE+CzdY/7wPP3e X-Google-Smtp-Source: ADKCNb5+915ru8ATJqdPREt1F8oNvuGSsUWO6cjlzgrAsiKraVHXDYaCd/5Kv5HcCHPjHHMHu1b1sA== X-Received: by 10.84.233.66 with SMTP id k2mr3241999plt.57.1504795942944; Thu, 07 Sep 2017 07:52:22 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/8] powerpc/64s/radix: optimize TLB range flush barriers Date: Fri, 8 Sep 2017 00:51:43 +1000 Message-Id: <20170907145148.24398-4-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Short range flushes issue a sequences of tlbie(l) instructions for individual effective addresses. These do not all require individual barrier sequences, only one set around all instructions. Commit f7327e0ba3 ("powerpc/mm/radix: Remove unnecessary ptesync") made a similar optimization for tlbiel for PID flushing. For tlbie, the ISA says: The tlbsync instruction provides an ordering function for the effects of all tlbie instructions executed by the thread executing the tlbsync instruction, with respect to the memory barrier created by a subsequent ptesync instruction executed by the same thread. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1ed61baf58da..c30f3faf5356 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -84,7 +84,7 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) trace_tlbie(0, 0, rb, rs, ric, prs, r); } -static inline void _tlbiel_va(unsigned long va, unsigned long pid, +static inline void __tlbiel_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -95,14 +95,20 @@ static inline void _tlbiel_va(unsigned long va, unsigned long pid, prs = 1; /* process scoped */ r = 1; /* raidx format */ - asm volatile("ptesync": : :"memory"); asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); - asm volatile("ptesync": : :"memory"); trace_tlbie(0, 1, rb, rs, ric, prs, r); } -static inline void _tlbie_va(unsigned long va, unsigned long pid, +static inline void _tlbiel_va(unsigned long va, unsigned long pid, + unsigned long ap, unsigned long ric) +{ + asm volatile("ptesync": : :"memory"); + __tlbiel_va(va, pid, ap, ric); + asm volatile("ptesync": : :"memory"); +} + +static inline void __tlbie_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -113,13 +119,20 @@ static inline void _tlbie_va(unsigned long va, unsigned long pid, prs = 1; /* process scoped */ r = 1; /* raidx format */ - asm volatile("ptesync": : :"memory"); asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); - asm volatile("eieio; tlbsync; ptesync": : :"memory"); trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void _tlbie_va(unsigned long va, unsigned long pid, + unsigned long ap, unsigned long ric) +{ + asm volatile("ptesync": : :"memory"); + __tlbie_va(va, pid, ap, ric); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + + /* * Base TLB flushing operations: * @@ -335,14 +348,20 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, _tlbiel_pid(pid, RIC_FLUSH_TLB); else _tlbie_pid(pid, RIC_FLUSH_TLB); + } else { + asm volatile("ptesync": : :"memory"); for (addr = start; addr < end; addr += page_size) { if (local) - _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); + __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); else - _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); } + if (local) + asm volatile("ptesync": : :"memory"); + else + asm volatile("eieio; tlbsync; ptesync": : :"memory"); } preempt_enable(); } @@ -373,6 +392,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) _tlbie_pid(pid, RIC_FLUSH_PWC); /* Then iterate the pages */ + asm volatile("ptesync": : :"memory"); end = addr + HPAGE_PMD_SIZE; for (; addr < end; addr += PAGE_SIZE) { if (local) @@ -380,7 +400,10 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) else _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); } - + if (local) + asm volatile("ptesync": : :"memory"); + else + asm volatile("eieio; tlbsync; ptesync": : :"memory"); preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Thu Sep 7 14:51:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811063 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3Yt39qZz9s81 for ; Fri, 8 Sep 2017 01:01:22 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3Yt22LMzDrWY for ; Fri, 8 Sep 2017 01:01:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::244; helo=mail-pg0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mc5STtzDrXs for ; Fri, 8 Sep 2017 00:52:28 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id m9so5058072pgd.0 for ; Thu, 07 Sep 2017 07:52:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aCLrXLAokfH0FfTlXyiOfylUgTFq5cU4rg+EtS7+X+E=; b=LmzE0vMqaFiJuey7gskzAnk89CCqrF4s93MTQhQvatqzfWVx0rks7Jk452Njz5doKU GTODCaHN15lIBd9uGNvxmCak4lrwJlioDhmlRtcGqHpI6ypm4ByOtfA7UvkhMk0D9jDs dq4ny6OZYkG5PnpuKhAS8pbZ1m3/FlJ4Z/c4bzP/s7420gHGV4Nm85TlDaXQILaEN8zc 2MER0JUWNzGI3qrVOq/VtoUjPiISBQyu6a+WMXTIgnRARhJfGkJsNahEQHjGGYXmHjwq OCiTiUwlZTkgCDKL6OMXn3q6JDEGYZs3603BviRaB7ar6eRwMiA3Kvw0CQDvZTTFBGCz AQIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aCLrXLAokfH0FfTlXyiOfylUgTFq5cU4rg+EtS7+X+E=; b=cU7dZb907TLJcP6oyMtKnam1DoEsDZvVBy1DcwTMf3xChRHfvV1AxXQOFWLOHW53WU V5hFSqkQjukDSpisCCwDpGXLFHxHqdObvLZtcH3S7xg5M4005SgSEQcH2+w0Nc5T9kWJ PonUq2n8pTTTNm2DMz0Wc5N2rOg32GZRFN3flRPVWBPd2G9WVMxj9UgNC/RsBUpNXorl gIrDHoNyVOxwoi76LhsklCYtKHa7GooBf+GEp69xOnAQzS0X/13Mzynhaf5lo4IHt9zU EEGD4p5+2g4im79cA2dJ462Rn/SCJCvy+aPyOKU2tRhORkFqJu12Ls/NtuZJpMPIUEVM cnPg== X-Gm-Message-State: AHPjjUhUUxNhLzacFs/Xxefov2LVLnZk83k//pzrwZTzVzBnm0XT57In kuq56uSlPZ1aC2sE X-Google-Smtp-Source: ADKCNb6YYiOX7Jr73SAW3BwI25+ebUrDuHlloHEO+ldZBj2ov5XfuA2hBUoxZ2qJKNkP8Krx4+Y/gQ== X-Received: by 10.84.211.100 with SMTP id b91mr3410603pli.126.1504795946812; Thu, 07 Sep 2017 07:52:26 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:25 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/8] powerpc/64s/radix: Implement _tlbie(l)_va_range flush functions Date: Fri, 8 Sep 2017 00:51:44 +1000 Message-Id: <20170907145148.24398-5-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Move the barriers and range iteration down into the _tlbie* level, which improves readability. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 70 ++++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index c30f3faf5356..1d3cbc01596d 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -85,7 +85,7 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) } static inline void __tlbiel_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -101,13 +101,28 @@ static inline void __tlbiel_va(unsigned long va, unsigned long pid, } static inline void _tlbiel_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long psize, unsigned long ric) { + unsigned long ap = mmu_get_ap(psize); + asm volatile("ptesync": : :"memory"); __tlbiel_va(va, pid, ap, ric); asm volatile("ptesync": : :"memory"); } +static inline void _tlbiel_va_range(unsigned long start, unsigned long end, + unsigned long pid, unsigned long page_size, + unsigned long psize) +{ + unsigned long addr; + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + for (addr = start; addr < end; addr += page_size) + __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); + asm volatile("ptesync": : :"memory"); +} + static inline void __tlbie_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { @@ -125,13 +140,27 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid, } static inline void _tlbie_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long psize, unsigned long ric) { + unsigned long ap = mmu_get_ap(psize); + asm volatile("ptesync": : :"memory"); __tlbie_va(va, pid, ap, ric); asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbie_va_range(unsigned long start, unsigned long end, + unsigned long pid, unsigned long page_size, + unsigned long psize) +{ + unsigned long addr; + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + for (addr = start; addr < end; addr += page_size) + __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} /* * Base TLB flushing operations: @@ -173,12 +202,11 @@ void radix__local_flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmadd int psize) { unsigned long pid; - unsigned long ap = mmu_get_ap(psize); preempt_disable(); pid = mm ? mm->context.id : 0; if (pid != MMU_NO_CONTEXT) - _tlbiel_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); preempt_enable(); } @@ -238,16 +266,15 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, int psize) { unsigned long pid; - unsigned long ap = mmu_get_ap(psize); pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) return; preempt_disable(); if (!mm_is_thread_local(mm)) - _tlbie_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); else - _tlbiel_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); preempt_enable(); } @@ -331,9 +358,7 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize) { unsigned long pid; - unsigned long addr; bool local; - unsigned long ap = mmu_get_ap(psize); unsigned long page_size = 1UL << mmu_psize_defs[psize].shift; pid = mm ? mm->context.id : 0; @@ -350,18 +375,10 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, _tlbie_pid(pid, RIC_FLUSH_TLB); } else { - asm volatile("ptesync": : :"memory"); - for (addr = start; addr < end; addr += page_size) { - - if (local) - __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); - else - __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); - } if (local) - asm volatile("ptesync": : :"memory"); + _tlbiel_va_range(start, end, pid, page_size, psize); else - asm volatile("eieio; tlbsync; ptesync": : :"memory"); + _tlbie_va_range(start, end, pid, page_size, psize); } preempt_enable(); } @@ -369,7 +386,6 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, #ifdef CONFIG_TRANSPARENT_HUGEPAGE void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) { - unsigned long ap = mmu_get_ap(mmu_virtual_psize); unsigned long pid, end; bool local; @@ -392,18 +408,12 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) _tlbie_pid(pid, RIC_FLUSH_PWC); /* Then iterate the pages */ - asm volatile("ptesync": : :"memory"); end = addr + HPAGE_PMD_SIZE; - for (; addr < end; addr += PAGE_SIZE) { - if (local) - _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); - else - _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); - } + if (local) - asm volatile("ptesync": : :"memory"); + _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); else - asm volatile("eieio; tlbsync; ptesync": : :"memory"); + _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Thu Sep 7 14:51:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811064 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3c10vSWz9t2R for ; Fri, 8 Sep 2017 01:03:13 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sibHLZrc"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3c06nclzDqF4 for ; Fri, 8 Sep 2017 01:03:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sibHLZrc"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::244; helo=mail-pf0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sibHLZrc"; dkim-atps=neutral Received: from mail-pf0-x244.google.com (mail-pf0-x244.google.com [IPv6:2607:f8b0:400e:c00::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mh6KRzzDrX5 for ; Fri, 8 Sep 2017 00:52:32 +1000 (AEST) Received: by mail-pf0-x244.google.com with SMTP id g13so4277569pfm.2 for ; Thu, 07 Sep 2017 07:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=KLOK9UO01Ygd9O7yKf9WUdV9tD7wOX+INybDZQKY+qo=; b=sibHLZrch5mcd0iuPro+7SoUVc+ErbIDZPMUOGIokuOlBQWfqqBlbOEKhvFXJ6j3Mu tjdXXBNi5W4iVVd/trEG6fl+iqAaplTcG1zs5Lp99DHEp36NpjrtWO1lvwnq36VjwmcR HwqMR90r+46zLMz07Lsb5hKgEZXHXivg329jB5Su2PCa/BEKZz5Zk4UiuEEcSRB7I5uO fn7l8ofMEOyxIWwzUhLkbDJggivZtDFuiEeNnSLQmJojjfIBx981DFqRZyTysJ6eutsR rJBIvOi16ezMwd8lk6BUcY2UFS2gSc165rY5qnQGeVx5JGSL3/fNOYo9FMZ0jtxTbS2Y K54A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=KLOK9UO01Ygd9O7yKf9WUdV9tD7wOX+INybDZQKY+qo=; b=QOI92BHtZ3g9421YAmBKWUPaQmp96fDU1YKmOprppa6zsBcx8CCsZhdZJdEWsINB7P 6MOJGrLZvhzGX/LLejluBB/33QcjCn9B0fyySRTCp07fwzvoTMpTyu6EMG6kXU0yH7Gn vMD5xvrQFhoyOkYIdOKl2QjgNiW4bhUVlDFU+ecPKkqSfm62EEAdvGBLNIEw0rM+55Zd WeAp5ASmqMBz1xmdqve2YJDIiL9UGnS+SemwTqRolucgYvXxkM9x9r4Ybzz30ezwD8mm O6zMH8zcHREoeTw2ci+sUydsm4ZKSYhbs9EYqTqmKT4+br/Cj7awOUKTunFEB5j3cMwU 8RJw== X-Gm-Message-State: AHPjjUgSI62Q7skS7T/QDJ4TfW68d+L7F4WVjI8ycn9dd9N2urskT1E3 FrtJasV4OnQZjbrA X-Google-Smtp-Source: ADKCNb7Jq0c8DRArLtQPG7nw/XQ/TC+scT8c2OrnZyAjgijlKIaRGKyQbRspSgMoVrB+5YRYfES75g== X-Received: by 10.84.194.131 with SMTP id h3mr3251495pld.250.1504795950668; Thu, 07 Sep 2017 07:52:30 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:29 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 5/8] powerpc/64s/radix: Introduce local single page ceiling for TLB range flush Date: Fri, 8 Sep 2017 00:51:45 +1000 Message-Id: <20170907145148.24398-6-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The single page flush ceiling is the cut-off point at which we switch from invalidating individual pages, to invalidating the entire process address space in response to a range flush. Introduce a local variant of this heuristic because local and global tlbie have significantly different properties: - Local tlbiel requires 128 instructions to invalidate a PID, global tlbie only 1 instruction. - Global tlbie instructions are expensive broadcast operations. The local ceiling has been made much higher, 2x the number of instructions required to invalidate the entire PID (this has not yet been benchmarked in detail). --- arch/powerpc/mm/tlb-radix.c | 49 +++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1d3cbc01596d..8ec59b57d46c 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -348,35 +348,41 @@ void radix__tlb_flush(struct mmu_gather *tlb) } #define TLB_FLUSH_ALL -1UL + /* - * Number of pages above which we will do a bcast tlbie. Just a - * number at this point copied from x86 + * Number of pages above which we invalidate the entire PID rather than + * flush individual pages, for local and global flushes respectively. + * + * tlbie goes out to the interconnect and individual ops are more costly. + * It also does not iterate over sets like the local tlbiel variant when + * invalidating a full PID, so it has a far lower threshold to change from + * individual page flushes to full-pid flushes. */ static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33; +static unsigned long tlb_local_single_page_flush_ceiling __read_mostly = POWER9_TLB_SETS_RADIX * 2; void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize) { unsigned long pid; - bool local; - unsigned long page_size = 1UL << mmu_psize_defs[psize].shift; + unsigned int page_shift = mmu_psize_defs[psize].shift; + unsigned long page_size = 1UL << page_shift; pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) return; preempt_disable(); - local = mm_is_thread_local(mm); - if (end == TLB_FLUSH_ALL || - (end - start) > tlb_single_page_flush_ceiling * page_size) { - if (local) + if (mm_is_thread_local(mm)) { + if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > + tlb_local_single_page_flush_ceiling) _tlbiel_pid(pid, RIC_FLUSH_TLB); else - _tlbie_pid(pid, RIC_FLUSH_TLB); - - } else { - if (local) _tlbiel_va_range(start, end, pid, page_size, psize); + } else { + if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > + tlb_single_page_flush_ceiling) + _tlbie_pid(pid, RIC_FLUSH_TLB); else _tlbie_va_range(start, end, pid, page_size, psize); } @@ -387,7 +393,6 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) { unsigned long pid, end; - bool local; pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) @@ -399,21 +404,17 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) return; } - preempt_disable(); - local = mm_is_thread_local(mm); - /* Otherwise first do the PWC */ - if (local) - _tlbiel_pid(pid, RIC_FLUSH_PWC); - else - _tlbie_pid(pid, RIC_FLUSH_PWC); - - /* Then iterate the pages */ end = addr + HPAGE_PMD_SIZE; - if (local) + /* Otherwise first do the PWC, then iterate the pages. */ + preempt_disable(); + if (mm_is_thread_local(mm)) { + _tlbiel_pid(pid, RIC_FLUSH_PWC); _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); - else + } else { + _tlbie_pid(pid, RIC_FLUSH_PWC); _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); + } preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ From patchwork Thu Sep 7 14:51:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811065 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3fM43cqz9s7c for ; Fri, 8 Sep 2017 01:05:15 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3fM2WHpzDrWZ for ; Fri, 8 Sep 2017 01:05:15 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::242; helo=mail-pg0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral Received: from mail-pg0-x242.google.com (mail-pg0-x242.google.com [IPv6:2607:f8b0:400e:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mm4CdWzDrXN for ; Fri, 8 Sep 2017 00:52:36 +1000 (AEST) Received: by mail-pg0-x242.google.com with SMTP id m9so5058194pgd.0 for ; Thu, 07 Sep 2017 07:52:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iYSP55WzqfkovJb0arC7l1Ra5u10EnLIgc6ntlZmHLw=; b=tRw+04Oz6SPlAA2J/BRUBrT37UBzN7DQaPZn+Cr4VkqiTUkPY8p/l2zG6yQZIIHdbx qhFGALxdrUjVlpZAqMLn0FVRKg/JeIvVbcQox+2C1ibPTmXLN9v9alj5OG/TfCHAEyJZ 1T/i6JNIfc8b6c01j70j7UZG+XoTssJt9OFMBLJVPwPx2/vwnyXQXAGDDc9PbXHjulai 0BD9z3dlq3ChEMzjZTS2DdXapBn/IoyziaKuDfj5ya1RibTIKG3KxyyIrpgaqxHuqWhA Pxh1ZCX+++5ggF++ieKN9qFWxBQvlimODnTj6gqbgmW0k81PBSYC+LXKdz7PAPUGOtXc 4L2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iYSP55WzqfkovJb0arC7l1Ra5u10EnLIgc6ntlZmHLw=; b=TSEatdqDZxBvlVWkS1sPBNHNgMwBdKyoNM+Z7cWc7GjnxeizFc7fGx3zS1/ZUpYYOH WC79z0mR0/45PMi2Nl3TjlL0bsOl4kxilzSdlvE/jmpB7smcWo4mi0UDiY9wd7+UkL6O pB3RcELn0yKNsHGZ91Mi4995unI3npKHOl1KDJxNZeqfAUO0rOCfFN1y9CbU7XH3DLPh xXVmBPa7vSTRyWebIODgMxl+OAcXMLiAAE0EzwPt1t4CD9lAAbF1+lD3h6TwYjK8ad4i Sd8KZq0Jyzmw689p7cWntHqvo8VXjVZLiBO/+Yo9jNTc1UVP+yGtt6zHZ5x8d0yWdBG6 QYww== X-Gm-Message-State: AHPjjUhdc+m7J3t/rgYQOjY3Tz4Xe0hIN3cJOHt7GkrlNX8QMlvKyrdg 2r2sw61dAC9saqjR X-Google-Smtp-Source: ADKCNb4/og7gX1hz0BEtiY3AyhWi2zOu/IqFScLeU64tRLPWaVSHbYW5GA/kD+rd/DejxTLoWAL8dg== X-Received: by 10.99.127.78 with SMTP id p14mr2932428pgn.159.1504795954348; Thu, 07 Sep 2017 07:52:34 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:33 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 6/8] powerpc/64s/radix: Optimize flush_tlb_range Date: Fri, 8 Sep 2017 00:51:46 +1000 Message-Id: <20170907145148.24398-7-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently for radix, flush_tlb_range flushes the entire PID, because we don't know about THP vs regular pages. This is quite sub-optimal for small mremap/mprotect/change_protection. Instead, implement this with two range flush passes, one for each page size. If the small page range flush ended up doing the full PID invalidation, then avoid the second flush. If not, the second flush is an order of magnitude or two fewer operations than the first, so it's relatively insignificant. There is still room for improvement here with some changes to generic APIs, particularly if there are a lot of huge pages in place. Signed-off-by: Nicholas Piggin --- .../powerpc/include/asm/book3s/64/tlbflush-radix.h | 2 +- arch/powerpc/mm/tlb-radix.c | 52 +++++++++++++++++----- 2 files changed, 42 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 9b433a624bf3..b12460b306a7 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -12,7 +12,7 @@ static inline int mmu_get_ap(int psize) extern void radix__flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); -extern void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, +extern bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 8ec59b57d46c..1b0cac656680 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -299,17 +299,40 @@ void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end) } EXPORT_SYMBOL(radix__flush_tlb_kernel_range); -/* - * Currently, for range flushing, we just do a full mm flush. Because - * we use this in code path where we don' track the page size. - */ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { struct mm_struct *mm = vma->vm_mm; + bool full; - radix__flush_tlb_mm(mm); +#ifdef CONFIG_HUGETLB_PAGE + if (is_vm_hugetlb_page(vma)) + return radix__flush_hugetlb_tlb_range(vma, start, end); +#endif + full = radix__flush_tlb_range_psize(mm, start, end, mmu_virtual_psize); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (!full) { + /* + * If the small page flush was not a full PID flush, we have + * to do a second pass to flush transparent huge pages. This + * will be a far smaller number of invalidates, so it's not + * worth calculating. + * + * Range flushes are still sub-optimal for cases of all or + * no hugepages (moreso the former), which should be improved + * by changing the flush API. + */ + unsigned long hstart, hend; + hstart = (start + HPAGE_PMD_SIZE - 1) >> HPAGE_PMD_SHIFT; + hend = end >> HPAGE_PMD_SHIFT; + if (hstart != hend) { + hstart <<= HPAGE_PMD_SHIFT; + hend <<= HPAGE_PMD_SHIFT; + radix__flush_tlb_range_psize(mm, hstart, hend, MMU_PAGE_2M); + } + } +#endif } EXPORT_SYMBOL(radix__flush_tlb_range); @@ -361,32 +384,39 @@ void radix__tlb_flush(struct mmu_gather *tlb) static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33; static unsigned long tlb_local_single_page_flush_ceiling __read_mostly = POWER9_TLB_SETS_RADIX * 2; -void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, +bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize) { unsigned long pid; unsigned int page_shift = mmu_psize_defs[psize].shift; unsigned long page_size = 1UL << page_shift; + bool full = false; pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) - return; + return full; preempt_disable(); if (mm_is_thread_local(mm)) { if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > - tlb_local_single_page_flush_ceiling) + tlb_local_single_page_flush_ceiling) { + full = true; _tlbiel_pid(pid, RIC_FLUSH_TLB); - else + } else { _tlbiel_va_range(start, end, pid, page_size, psize); + } } else { if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > - tlb_single_page_flush_ceiling) + tlb_single_page_flush_ceiling) { + full = true; _tlbie_pid(pid, RIC_FLUSH_TLB); - else + } else { _tlbie_va_range(start, end, pid, page_size, psize); + } } preempt_enable(); + + return full; } #ifdef CONFIG_TRANSPARENT_HUGEPAGE From patchwork Thu Sep 7 14:51:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811066 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3hb1y4Nz9ryr for ; Fri, 8 Sep 2017 01:07:11 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C7OShDLY"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3hb0mSczDrW9 for ; Fri, 8 Sep 2017 01:07:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C7OShDLY"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::241; helo=mail-pf0-x241.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="C7OShDLY"; dkim-atps=neutral Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mr1X0bzDrWY for ; Fri, 8 Sep 2017 00:52:40 +1000 (AEST) Received: by mail-pf0-x241.google.com with SMTP id a2so4610220pfj.4 for ; Thu, 07 Sep 2017 07:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=k7jtoLrGnc5cupJ5VkPsvyaqyNbZ8IwAu+HiJhw7pnE=; b=C7OShDLYqGCgdah8QOHU/++KAFNwqX4We2CiILc5HOkvVSMkl+ffI6Z8WY2J8pI/uH NIc4BzkejnjOX157irnsXL4KUH2L6dmuflO8tfPT+Jvq983wwc4JtBGcAcIwgoaQEC8B pnd51gp7DL6P3hkuNZiwOps9ZTNXObuGA8gRARFhzqgIsCWmHCdjDm0TYnDlXN3NTToX tN/+5ZGzSrdgRLViQeCnRjnfbY03J9ii1Sfxpz4aG5XOJavA+zzlw3078j7MI/uynqSb d6P5UXq5PqxgTkeSbzBOhl7gnA8coxlr4Z+tCt41uf8vtVBm62DrGXp5hTWGfJqz416X 56+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=k7jtoLrGnc5cupJ5VkPsvyaqyNbZ8IwAu+HiJhw7pnE=; b=q3U0+loSOuwjCgm08Fl6KZekgai+l3CXVG8MZIW/VYOVjVUep4LxlNO65ovadHDLlM 4WnXBf6qgYMVJZJoDGpQa8InkPeDQv6T9AlrrcbJLIV5/g3Ju1sfeUUFVHv8ROD0L0HN kKqdBPjIADe6y7aQbdlh/xzLlU1bmDUcztIyXbV+2z39LLVOS7nbf5cLcon1BQocpkJO NxNFgJRhEE8MlWIJq8D5mfjQXvjMfQb3R0ZHG3zrDd4cMmZfnYTt9nlKrgV6usXVBJcp r7B1a5A8a5kcJxrxaqdwoX/FY4pE/YX/Gk5qf9POJSlk3uTXbXwfsTDNm5hpJiIHhhsT 0O4g== X-Gm-Message-State: AHPjjUh/RkSAl9aBOf9zyVda9m6QdCRu90DRnLXW2tzFpV/iLU2KahRD zUC+S4nFQyCFwoTJ X-Google-Smtp-Source: ADKCNb5zcDX8yyNnH4RGvDtDuaPyLHSU6eLBDfo0R3QgEo4P7jqpoIY99zWouAhqw/DRYKRp6kSOfQ== X-Received: by 10.99.126.84 with SMTP id o20mr2989166pgn.133.1504795958009; Thu, 07 Sep 2017 07:52:38 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:37 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 7/8] powerpc/64s/radix: Improve TLB flushing for unmaps that free a page table Date: Fri, 8 Sep 2017 00:51:47 +1000 Message-Id: <20170907145148.24398-8-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Unmaps that free page tables always flush the PID, which is sub optimal. Allow those to do TLB range flushes with separate PWC flush. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 51 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1b0cac656680..7452e1f4aa3c 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -351,23 +351,35 @@ static int radix_get_mmu_psize(int page_size) return psize; } +static void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, + unsigned long end, int psize); + void radix__tlb_flush(struct mmu_gather *tlb) { int psize = 0; struct mm_struct *mm = tlb->mm; int page_size = tlb->page_size; - psize = radix_get_mmu_psize(page_size); /* * if page size is not something we understand, do a full mm flush */ - if (psize != -1 && !tlb->fullmm && !tlb->need_flush_all) - radix__flush_tlb_range_psize(mm, tlb->start, tlb->end, psize); - else if (tlb->need_flush_all) { - tlb->need_flush_all = 0; + if (tlb->fullmm) { radix__flush_all_mm(mm); - } else - radix__flush_tlb_mm(mm); + } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { + if (!tlb->need_flush_all) + radix__flush_tlb_mm(mm); + else + radix__flush_all_mm(mm); + } else { + unsigned long start = tlb->start; + unsigned long end = tlb->end; + + if (!tlb->need_flush_all) + radix__flush_tlb_range_psize(mm, start, end, psize); + else + radix__flush_tlb_pwc_range_psize(mm, start, end, psize); + } + tlb->need_flush_all = 0; } #define TLB_FLUSH_ALL -1UL @@ -384,8 +396,9 @@ void radix__tlb_flush(struct mmu_gather *tlb) static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33; static unsigned long tlb_local_single_page_flush_ceiling __read_mostly = POWER9_TLB_SETS_RADIX * 2; -bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, - unsigned long end, int psize) +static bool __radix__flush_tlb_range_psize(struct mm_struct *mm, + unsigned long start, unsigned long end, + int psize, bool also_pwc) { unsigned long pid; unsigned int page_shift = mmu_psize_defs[psize].shift; @@ -401,17 +414,21 @@ bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > tlb_local_single_page_flush_ceiling) { full = true; - _tlbiel_pid(pid, RIC_FLUSH_TLB); + _tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } else { _tlbiel_va_range(start, end, pid, page_size, psize); + if (also_pwc) + _tlbiel_pid(pid, RIC_FLUSH_PWC); } } else { if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > tlb_single_page_flush_ceiling) { full = true; - _tlbie_pid(pid, RIC_FLUSH_TLB); + _tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB); } else { _tlbie_va_range(start, end, pid, page_size, psize); + if (also_pwc) + _tlbie_pid(pid, RIC_FLUSH_PWC); } } preempt_enable(); @@ -419,6 +436,18 @@ bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, return full; } +bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, + unsigned long end, int psize) +{ + return __radix__flush_tlb_range_psize(mm, start, end, psize, false); +} + +static void radix__flush_tlb_pwc_range_psize(struct mm_struct *mm, unsigned long start, + unsigned long end, int psize) +{ + __radix__flush_tlb_range_psize(mm, start, end, psize, true); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) { From patchwork Thu Sep 7 14:51:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811068 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3kg6rf3z9t2v for ; Fri, 8 Sep 2017 01:08:59 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="PYmamQlV"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3kg5CF8zDrSL for ; Fri, 8 Sep 2017 01:08:59 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="PYmamQlV"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c00::243; helo=mail-pf0-x243.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="PYmamQlV"; dkim-atps=neutral Received: from mail-pf0-x243.google.com (mail-pf0-x243.google.com [IPv6:2607:f8b0:400e:c00::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mv6wFDzDrXG for ; Fri, 8 Sep 2017 00:52:43 +1000 (AEST) Received: by mail-pf0-x243.google.com with SMTP id q76so4607007pfq.5 for ; Thu, 07 Sep 2017 07:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Z0H1SH2RamIuKZq2U/Q2NxEWB2Eqo3rTC3mDm145w1g=; b=PYmamQlVk2t9azWD+b/7XTI7CwPban1bT2lx6vXP0OA6QtpM8PpbF590V+2ci2fH+Q S3pMdfulp5Y3Rp/l3NkScmTilwjAZbDJ0EAMOINbli0XaV5EkGMNkYkPKQ1/MIyASR/k /wYBD8gk8E+wNZ+EFrLOpKM2V0ri6MSvvlSxWPSZNYuW9GjRr9ubGxzt6LtEm1FKmSK0 lp5xwWvqcS4bmC2QYR+sZM4Hd2LeBBTClJl3VYT0IHaa0fmmQ3TwX0I32P7ldPgxGlF/ pssCt+LzaIQwoKzAnkqdnyO5P1Y8ctJgxMX8sB8pYvxwWNhdzPo2WOSMCAWXsD1XgsJX W8cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Z0H1SH2RamIuKZq2U/Q2NxEWB2Eqo3rTC3mDm145w1g=; b=RyjW/NDeS+xZJqMFCFe7oUBU6Pv2eDCf9EHpBFQ8BOY4KnQbzSVEgGgUGEAL7YJMwS BvFqdi8aozxxt0xWwkWtrLPfG866eyEWh5VdOmW65A5J/vVtusDH2Z6RhM6AYIbICEcc kiTRF3L98blbFKkD+JQ1O38+rbVIHjD5X5deEwBtfWiCZofjffq1blafgWCE/5K+MuQa Ac4qCq0S4gvk5/eWQ7zVJr+mNO+9FuTh2yVTRB4jfgcY8m4eZMk5oOp1B9UfOwvN/tkq xiKTwM3+qLxXsxXBXdmT01KBMUO87/eX+MztXOQXxA6q8dQUo2qXbcJeyEJkPPBe2kwR CNow== X-Gm-Message-State: AHPjjUhFau/+8H5ca7mOtZpZ49XEa5hnVmLiM1cakdo6rH4ugaNgCVSF j9mqv2pz7y8x0mDq X-Google-Smtp-Source: ADKCNb70/ElXq4JdaOvflOP88bBJZqFckwlD6g8mE1Vuk7Jrq4p/0H27j/MM1YOEs/uRq76zbci5Yg== X-Received: by 10.99.38.4 with SMTP id m4mr2942539pgm.53.1504795961849; Thu, 07 Sep 2017 07:52:41 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:40 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 8/8] powerpc/64s/radix: Only flush local TLB for spurious fault flushes Date: Fri, 8 Sep 2017 00:51:48 +1000 Message-Id: <20170907145148.24398-9-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When permissiveness is relaxed, or found to have been relaxed by another thread, we flush that address out of the TLB to avoid a future fault or micro-fault due to a stale TLB entry. Currently for processes with TLBs on other CPUs, this flush is always done with a global tlbie. Although that could reduce faults on remote CPUs, a broadcast operation seems to be wasteful for something that can be handled in-core by the remote CPU if it comes to it. This is not benchmarked yet. It does seem cut some tlbie operations from the bus. Signed-off-by: Nicholas Piggin --- .../powerpc/include/asm/book3s/64/tlbflush-radix.h | 5 ++++ arch/powerpc/include/asm/book3s/64/tlbflush.h | 11 +++++++++ arch/powerpc/mm/pgtable-book3s64.c | 5 +++- arch/powerpc/mm/pgtable.c | 2 +- arch/powerpc/mm/tlb-radix.c | 27 ++++++++++++++++++++++ 5 files changed, 48 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index b12460b306a7..34cd864b8fc1 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -16,6 +16,8 @@ extern bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long sta unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +extern void radix__local_flush_pmd_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end); extern void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end); @@ -24,6 +26,9 @@ extern void radix__local_flush_tlb_mm(struct mm_struct *mm); extern void radix__local_flush_tlb_page(struct vm_area_struct *vma, unsigned long vmaddr); extern void radix__local_flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, int psize); +extern void radix__local_flush_tlb_range_psize(struct mm_struct *mm, + unsigned long start, unsigned long end, + int psize); extern void radix__tlb_flush(struct mmu_gather *tlb); #ifdef CONFIG_SMP extern void radix__flush_tlb_mm(struct mm_struct *mm); diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 72b925f97bab..8a8b3e11a28e 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -83,6 +83,17 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, #define flush_tlb_mm(mm) local_flush_tlb_mm(mm) #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #endif /* CONFIG_SMP */ + +#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault +static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, + unsigned long address) +{ + if (radix_enabled()) + radix__local_flush_tlb_page(vma, address); + else + flush_tlb_page(vma, address); +} + /* * flush the page walk cache for the address */ diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 3b65917785a5..e46f346388d6 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -40,7 +40,10 @@ int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, if (changed) { __ptep_set_access_flags(vma->vm_mm, pmdp_ptep(pmdp), pmd_pte(entry), address); - flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); + if (radix_enabled()) + radix__local_flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); + else + flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); } return changed; } diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index a03ff3d99e0c..acd6ae8062ce 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -223,7 +223,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, if (!is_vm_hugetlb_page(vma)) assert_pte_locked(vma->vm_mm, address); __ptep_set_access_flags(vma->vm_mm, ptep, entry, address); - flush_tlb_page(vma, address); + flush_tlb_fix_spurious_fault(vma, address); } return changed; } diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 7452e1f4aa3c..bcb41d037593 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -396,6 +396,27 @@ void radix__tlb_flush(struct mmu_gather *tlb) static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33; static unsigned long tlb_local_single_page_flush_ceiling __read_mostly = POWER9_TLB_SETS_RADIX * 2; +void radix__local_flush_tlb_range_psize(struct mm_struct *mm, + unsigned long start, unsigned long end, + int psize) +{ + unsigned long pid; + unsigned int page_shift = mmu_psize_defs[psize].shift; + unsigned long page_size = 1UL << page_shift; + + pid = mm ? mm->context.id : 0; + if (unlikely(pid == MMU_NO_CONTEXT)) + return; + + preempt_disable(); + if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > + tlb_local_single_page_flush_ceiling) + _tlbiel_pid(pid, RIC_FLUSH_TLB); + else + _tlbiel_va_range(start, end, pid, page_size, psize); + preempt_enable(); +} + static bool __radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize, bool also_pwc) @@ -518,6 +539,12 @@ void radix__flush_tlb_lpid(unsigned long lpid) } EXPORT_SYMBOL(radix__flush_tlb_lpid); +void radix__local_flush_pmd_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + radix__local_flush_tlb_range_psize(vma->vm_mm, start, end, MMU_PAGE_2M); +} + void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) {