From patchwork Thu Sep 7 14:51:44 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811063 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3Yt39qZz9s81 for ; Fri, 8 Sep 2017 01:01:22 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3Yt22LMzDrWY for ; Fri, 8 Sep 2017 01:01:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::244; helo=mail-pg0-x244.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LmzE0vMq"; dkim-atps=neutral Received: from mail-pg0-x244.google.com (mail-pg0-x244.google.com [IPv6:2607:f8b0:400e:c05::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mc5STtzDrXs for ; Fri, 8 Sep 2017 00:52:28 +1000 (AEST) Received: by mail-pg0-x244.google.com with SMTP id m9so5058072pgd.0 for ; Thu, 07 Sep 2017 07:52:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=aCLrXLAokfH0FfTlXyiOfylUgTFq5cU4rg+EtS7+X+E=; b=LmzE0vMqaFiJuey7gskzAnk89CCqrF4s93MTQhQvatqzfWVx0rks7Jk452Njz5doKU GTODCaHN15lIBd9uGNvxmCak4lrwJlioDhmlRtcGqHpI6ypm4ByOtfA7UvkhMk0D9jDs dq4ny6OZYkG5PnpuKhAS8pbZ1m3/FlJ4Z/c4bzP/s7420gHGV4Nm85TlDaXQILaEN8zc 2MER0JUWNzGI3qrVOq/VtoUjPiISBQyu6a+WMXTIgnRARhJfGkJsNahEQHjGGYXmHjwq OCiTiUwlZTkgCDKL6OMXn3q6JDEGYZs3603BviRaB7ar6eRwMiA3Kvw0CQDvZTTFBGCz AQIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=aCLrXLAokfH0FfTlXyiOfylUgTFq5cU4rg+EtS7+X+E=; b=cU7dZb907TLJcP6oyMtKnam1DoEsDZvVBy1DcwTMf3xChRHfvV1AxXQOFWLOHW53WU V5hFSqkQjukDSpisCCwDpGXLFHxHqdObvLZtcH3S7xg5M4005SgSEQcH2+w0Nc5T9kWJ PonUq2n8pTTTNm2DMz0Wc5N2rOg32GZRFN3flRPVWBPd2G9WVMxj9UgNC/RsBUpNXorl gIrDHoNyVOxwoi76LhsklCYtKHa7GooBf+GEp69xOnAQzS0X/13Mzynhaf5lo4IHt9zU EEGD4p5+2g4im79cA2dJ462Rn/SCJCvy+aPyOKU2tRhORkFqJu12Ls/NtuZJpMPIUEVM cnPg== X-Gm-Message-State: AHPjjUhUUxNhLzacFs/Xxefov2LVLnZk83k//pzrwZTzVzBnm0XT57In kuq56uSlPZ1aC2sE X-Google-Smtp-Source: ADKCNb6YYiOX7Jr73SAW3BwI25+ebUrDuHlloHEO+ldZBj2ov5XfuA2hBUoxZ2qJKNkP8Krx4+Y/gQ== X-Received: by 10.84.211.100 with SMTP id b91mr3410603pli.126.1504795946812; Thu, 07 Sep 2017 07:52:26 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:25 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/8] powerpc/64s/radix: Implement _tlbie(l)_va_range flush functions Date: Fri, 8 Sep 2017 00:51:44 +1000 Message-Id: <20170907145148.24398-5-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Move the barriers and range iteration down into the _tlbie* level, which improves readability. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/tlb-radix.c | 70 ++++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index c30f3faf5356..1d3cbc01596d 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -85,7 +85,7 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric) } static inline void __tlbiel_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long ap, unsigned long ric) { unsigned long rb,rs,prs,r; @@ -101,13 +101,28 @@ static inline void __tlbiel_va(unsigned long va, unsigned long pid, } static inline void _tlbiel_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long psize, unsigned long ric) { + unsigned long ap = mmu_get_ap(psize); + asm volatile("ptesync": : :"memory"); __tlbiel_va(va, pid, ap, ric); asm volatile("ptesync": : :"memory"); } +static inline void _tlbiel_va_range(unsigned long start, unsigned long end, + unsigned long pid, unsigned long page_size, + unsigned long psize) +{ + unsigned long addr; + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + for (addr = start; addr < end; addr += page_size) + __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); + asm volatile("ptesync": : :"memory"); +} + static inline void __tlbie_va(unsigned long va, unsigned long pid, unsigned long ap, unsigned long ric) { @@ -125,13 +140,27 @@ static inline void __tlbie_va(unsigned long va, unsigned long pid, } static inline void _tlbie_va(unsigned long va, unsigned long pid, - unsigned long ap, unsigned long ric) + unsigned long psize, unsigned long ric) { + unsigned long ap = mmu_get_ap(psize); + asm volatile("ptesync": : :"memory"); __tlbie_va(va, pid, ap, ric); asm volatile("eieio; tlbsync; ptesync": : :"memory"); } +static inline void _tlbie_va_range(unsigned long start, unsigned long end, + unsigned long pid, unsigned long page_size, + unsigned long psize) +{ + unsigned long addr; + unsigned long ap = mmu_get_ap(psize); + + asm volatile("ptesync": : :"memory"); + for (addr = start; addr < end; addr += page_size) + __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); + asm volatile("eieio; tlbsync; ptesync": : :"memory"); +} /* * Base TLB flushing operations: @@ -173,12 +202,11 @@ void radix__local_flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmadd int psize) { unsigned long pid; - unsigned long ap = mmu_get_ap(psize); preempt_disable(); pid = mm ? mm->context.id : 0; if (pid != MMU_NO_CONTEXT) - _tlbiel_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); preempt_enable(); } @@ -238,16 +266,15 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr, int psize) { unsigned long pid; - unsigned long ap = mmu_get_ap(psize); pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) return; preempt_disable(); if (!mm_is_thread_local(mm)) - _tlbie_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB); else - _tlbiel_va(vmaddr, pid, ap, RIC_FLUSH_TLB); + _tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB); preempt_enable(); } @@ -331,9 +358,7 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize) { unsigned long pid; - unsigned long addr; bool local; - unsigned long ap = mmu_get_ap(psize); unsigned long page_size = 1UL << mmu_psize_defs[psize].shift; pid = mm ? mm->context.id : 0; @@ -350,18 +375,10 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, _tlbie_pid(pid, RIC_FLUSH_TLB); } else { - asm volatile("ptesync": : :"memory"); - for (addr = start; addr < end; addr += page_size) { - - if (local) - __tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); - else - __tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); - } if (local) - asm volatile("ptesync": : :"memory"); + _tlbiel_va_range(start, end, pid, page_size, psize); else - asm volatile("eieio; tlbsync; ptesync": : :"memory"); + _tlbie_va_range(start, end, pid, page_size, psize); } preempt_enable(); } @@ -369,7 +386,6 @@ void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, #ifdef CONFIG_TRANSPARENT_HUGEPAGE void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) { - unsigned long ap = mmu_get_ap(mmu_virtual_psize); unsigned long pid, end; bool local; @@ -392,18 +408,12 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr) _tlbie_pid(pid, RIC_FLUSH_PWC); /* Then iterate the pages */ - asm volatile("ptesync": : :"memory"); end = addr + HPAGE_PMD_SIZE; - for (; addr < end; addr += PAGE_SIZE) { - if (local) - _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB); - else - _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB); - } + if (local) - asm volatile("ptesync": : :"memory"); + _tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); else - asm volatile("eieio; tlbsync; ptesync": : :"memory"); + _tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize); preempt_enable(); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */