From patchwork Thu Sep 7 14:51:43 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811062 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3WW2DsTz9s81 for ; Fri, 8 Sep 2017 00:59:19 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3WW14c6zDrYt for ; Fri, 8 Sep 2017 00:59:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::244; helo=mail-pg0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DUJcPvvf"; dkim-atps=neutral Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3MX6qLlzDrWX for ; Fri, 8 Sep 2017 00:52:24 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id v82so4426355pgb.1 for ; Thu, 07 Sep 2017 07:52:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UDIibrf762Cv/CDNL2CMxskCyimmje6rG63nwqMD0cM=; b=DUJcPvvfIi+z/bhswggtfW25oJ8iip+b12WI+OblQSZsRN00ECbW++iuqqKpPo83L8 s0Zzeh6T/cHxcYJuyXxn3B8UIaccvqFcyrgYgS7fRCHTGqEszD5fNqlTpxP+V97lUrsW Fv1w9VMXCBZ5WicqLTUjV0RHms2gsgxlN9RUB6dpF1mtwVDjaiHgM1YullATrOaWZR6a AK11j6DWmoOtm9bIBF0HNMJUQSTNwTz9jLYPR2hiFe8ZG1OffiY4VChaAFmX58MLWtRV 56zhpIFekuDGYnH2JxYr7/E/WpbxIrcOx61n2VSTYHo2Y5qdwlyjEZ1dFWNe+GnodB7b wajg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UDIibrf762Cv/CDNL2CMxskCyimmje6rG63nwqMD0cM=; b=e0DPCKT4RIV8F9d2CvpIOtRIT7LMwfyqiFeU4F7bS2MM0lYycMCIxnWrntj2gSXSbR S8yME58TKAk1qcmoQY+kAOzPF3ltZigU+U11Kv7bQe2QANfPBiDzyJKtiLHXamgvQY4z ZUCuVBjtDYesmDZn2myViZeN6FRthg711aEPn3qqzFRTfXJDW/d0IKYDg7uqIUaYuaha +J6yh3dG8e7V/+hBnZRVSHmtIl9oKVNHJ9rPyFMJebSj4UoFPCXAEZtmcB9aGFxb92w8 oDAu7mYiYB93OBmpAGdbjQPwDSi4bihmOKn+oJDMjqLtTgnt2D01h/e382m0o6k9MtWY fMVw== X-Gm-Message-State: AHPjjUjk7zBy9YXMYJAL/gfS6pGm1UI+kJ6QuzQ/TZwfG6bP0q+eEAEG nsFE+CzdY/7wPP3e X-Google-Smtp-Source: ADKCNb5+915ru8ATJqdPREt1F8oNvuGSsUWO6cjlzgrAsiKraVHXDYaCd/5Kv5HcCHPjHHMHu1b1sA== X-Received: by 10.84.233.66 with SMTP id k2mr3241999plt.57.1504795942944; Thu, 07 Sep 2017 07:52:22 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/8] powerpc/64s/radix: optimize TLB range flush barriers Date: Fri, 8 Sep 2017 00:51:43 +1000 Message-Id: <20170907145148.24398-4-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Short range flushes issue a sequences of tlbie(l) instructions for individual effective addresses. These do not all require individual barrier sequences, only one set around all instructions. Commit f7327e0ba3 ("powerpc/mm/radix: Remove unnecessary ptesync") made a similar optimization for tlbiel for PID flushing. For tlbie, the ISA says: The tlbsync instruction provides an ordering function for the effects of all tlbie instructions executed by the thread executing the tlbsync instruction, with respect to the memory barrier created by a subsequent ptesync instruction executed by the same thread. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1ed61baf58da..c30f3faf5356 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -84,7 +84,7 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) trace_tlbie(0, 0, rb, rs, ric, prs, r); } -static inline void _tlbiel_va(unsigned long va, unsigned long pid, +static inline void __tlbiel_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -95,14 +95,20 @@ static inline void _tlbiel_va(unsigned long va, unsigned long pid, prs = 1; /* process scoped */ r = 1; /* raidx format */ - asm volatile("ptesync": : :"memory"); asm volatile(PPC_TLBIEL(%0, %4, %3, %2, %1) : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); - asm volatile("ptesync": : :"memory"); trace_tlbie(0, 1, rb, rs, ric, prs, r); } -static inline void _tlbie_va(unsigned long va, unsigned long pid, +static inline void _tlbiel_va(unsigned long va, unsigned long pid, + unsigned long ap, unsigned long ric) +{ + asm volatile("ptesync": : :"memory"); + __tlbiel_va(va, pid, ap, ric); + asm volatile("ptesync": : :"memory"); +} + +static inline void __tlbie_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -113,13 +119,20 @@ static inline void _tlbie_va(unsigned long va, unsigned long pid, prs = 1; /* process scoped */ r = 1; /* raidx format */ - asm volatile("ptesync": : :"memory"); asm volatile(PPC_TLBIE_5(%0, %4, %3, %2, %1) : : "r"(rb), "i"(r), "i"(prs), "i"(ric), "r"(rs) : "memory"); - asm volatile("eieio; tlbsync; ptesync": : :"memory"); trace_tlbie(0, 0, rb, rs, ric, prs, r); } +static inline void _tlbie_va(unsigned long va, unsigned long pid, + unsigned long ap, unsigned long ric) +{ + asm volatile("ptesync": : :"memory"); + __tlbie_va(va, pid, ap, ric); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} + + /* * Base TLB flushing operations: * @@ -335,14 +348,20 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, _tlbiel_pid(pid, RIC_FLUSH_TLB); else _tlbie_pid(pid, RIC_FLUSH_TLB); + } else { + asm volatile("ptesync": : :"memory"); for (addr = start; addr < end; addr += page_size) { if (local) - _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); + __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); else - _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); } + if (local) + asm volatile("ptesync": : :"memory"); + else + asm volatile("eieio; tlbsync; ptesync": : :"memory"); } preempt_enable(); } @@ -373,6 +392,7 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) _tlbie_pid(pid, RIC_FLUSH_PWC); /* Then iterate the pages */ + asm volatile("ptesync": : :"memory"); end = addr + HPAGE_PMD_SIZE; for (; addr < end; addr += PAGE_SIZE) { if (local) @@ -380,7 +400,10 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) else _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); } - + if (local) + asm volatile("ptesync": : :"memory"); + else + asm volatile("eieio; tlbsync; ptesync": : :"memory"); preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */