From patchwork Thu Sep 7 14:51:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 811065 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xp3fM43cqz9s7c for ; Fri, 8 Sep 2017 01:05:15 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3xp3fM2WHpzDrWZ for ; Fri, 8 Sep 2017 01:05:15 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:400e:c05::242; helo=mail-pg0-x242.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tRw+04Oz"; dkim-atps=neutral Received: from mail-pg0-x242.google.com (mail-pg0-x242.google.com [IPv6:2607:f8b0:400e:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3xp3Mm4CdWzDrXN for ; Fri, 8 Sep 2017 00:52:36 +1000 (AEST) Received: by mail-pg0-x242.google.com with SMTP id m9so5058194pgd.0 for ; Thu, 07 Sep 2017 07:52:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iYSP55WzqfkovJb0arC7l1Ra5u10EnLIgc6ntlZmHLw=; b=tRw+04Oz6SPlAA2J/BRUBrT37UBzN7DQaPZn+Cr4VkqiTUkPY8p/l2zG6yQZIIHdbx qhFGALxdrUjVlpZAqMLn0FVRKg/JeIvVbcQox+2C1ibPTmXLN9v9alj5OG/TfCHAEyJZ 1T/i6JNIfc8b6c01j70j7UZG+XoTssJt9OFMBLJVPwPx2/vwnyXQXAGDDc9PbXHjulai 0BD9z3dlq3ChEMzjZTS2DdXapBn/IoyziaKuDfj5ya1RibTIKG3KxyyIrpgaqxHuqWhA Pxh1ZCX+++5ggF++ieKN9qFWxBQvlimODnTj6gqbgmW0k81PBSYC+LXKdz7PAPUGOtXc 4L2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iYSP55WzqfkovJb0arC7l1Ra5u10EnLIgc6ntlZmHLw=; b=TSEatdqDZxBvlVWkS1sPBNHNgMwBdKyoNM+Z7cWc7GjnxeizFc7fGx3zS1/ZUpYYOH WC79z0mR0/45PMi2Nl3TjlL0bsOl4kxilzSdlvE/jmpB7smcWo4mi0UDiY9wd7+UkL6O pB3RcELn0yKNsHGZ91Mi4995unI3npKHOl1KDJxNZeqfAUO0rOCfFN1y9CbU7XH3DLPh xXVmBPa7vSTRyWebIODgMxl+OAcXMLiAAE0EzwPt1t4CD9lAAbF1+lD3h6TwYjK8ad4i Sd8KZq0Jyzmw689p7cWntHqvo8VXjVZLiBO/+Yo9jNTc1UVP+yGtt6zHZ5x8d0yWdBG6 QYww== X-Gm-Message-State: AHPjjUhdc+m7J3t/rgYQOjY3Tz4Xe0hIN3cJOHt7GkrlNX8QMlvKyrdg 2r2sw61dAC9saqjR X-Google-Smtp-Source: ADKCNb4/og7gX1hz0BEtiY3AyhWi2zOu/IqFScLeU64tRLPWaVSHbYW5GA/kD+rd/DejxTLoWAL8dg== X-Received: by 10.99.127.78 with SMTP id p14mr2932428pgn.159.1504795954348; Thu, 07 Sep 2017 07:52:34 -0700 (PDT) Received: from roar.au.ibm.com (203-219-56-202.tpgi.com.au. [203.219.56.202]) by smtp.gmail.com with ESMTPSA id a6sm4642791pfa.76.2017.09.07.07.52.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Sep 2017 07:52:33 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 6/8] powerpc/64s/radix: Optimize flush_tlb_range Date: Fri, 8 Sep 2017 00:51:46 +1000 Message-Id: <20170907145148.24398-7-npiggin@gmail.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20170907145148.24398-1-npiggin@gmail.com> References: <20170907145148.24398-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Aneesh Kumar K . V" , Nicholas Piggin , Anton Blanchard Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently for radix, flush_tlb_range flushes the entire PID, because we don't know about THP vs regular pages. This is quite sub-optimal for small mremap/mprotect/change_protection. Instead, implement this with two range flush passes, one for each page size. If the small page range flush ended up doing the full PID invalidation, then avoid the second flush. If not, the second flush is an order of magnitude or two fewer operations than the first, so it's relatively insignificant. There is still room for improvement here with some changes to generic APIs, particularly if there are a lot of huge pages in place. Signed-off-by: Nicholas Piggin --- .../powerpc/include/asm/book3s/64/tlbflush-radix.h | 2 +- arch/powerpc/mm/tlb-radix.c | 52 +++++++++++++++++----- 2 files changed, 42 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h index 9b433a624bf3..b12460b306a7 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h @@ -12,7 +12,7 @@ static inline int mmu_get_ap(int psize) extern void radix__flush_hugetlb_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); -extern void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, +extern bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize); extern void radix__flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 8ec59b57d46c..1b0cac656680 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -299,17 +299,40 @@ void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end) } EXPORT_SYMBOL(radix__flush_tlb_kernel_range); -/* - * Currently, for range flushing, we just do a full mm flush. Because - * we use this in code path where we don' track the page size. - */ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { struct mm_struct *mm = vma->vm_mm; + bool full; - radix__flush_tlb_mm(mm); +#ifdef CONFIG_HUGETLB_PAGE + if (is_vm_hugetlb_page(vma)) + return radix__flush_hugetlb_tlb_range(vma, start, end); +#endif + full = radix__flush_tlb_range_psize(mm, start, end, mmu_virtual_psize); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (!full) { + /* + * If the small page flush was not a full PID flush, we have + * to do a second pass to flush transparent huge pages. This + * will be a far smaller number of invalidates, so it's not + * worth calculating. + * + * Range flushes are still sub-optimal for cases of all or + * no hugepages (moreso the former), which should be improved + * by changing the flush API. + */ + unsigned long hstart, hend; + hstart = (start + HPAGE_PMD_SIZE - 1) >> HPAGE_PMD_SHIFT; + hend = end >> HPAGE_PMD_SHIFT; + if (hstart != hend) { + hstart <<= HPAGE_PMD_SHIFT; + hend <<= HPAGE_PMD_SHIFT; + radix__flush_tlb_range_psize(mm, hstart, hend, MMU_PAGE_2M); + } + } +#endif } EXPORT_SYMBOL(radix__flush_tlb_range); @@ -361,32 +384,39 @@ void radix__tlb_flush(struct mmu_gather *tlb) static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33; static unsigned long tlb_local_single_page_flush_ceiling __read_mostly = POWER9_TLB_SETS_RADIX * 2; -void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, +bool radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start, unsigned long end, int psize) { unsigned long pid; unsigned int page_shift = mmu_psize_defs[psize].shift; unsigned long page_size = 1UL << page_shift; + bool full = false; pid = mm ? mm->context.id : 0; if (unlikely(pid == MMU_NO_CONTEXT)) - return; + return full; preempt_disable(); if (mm_is_thread_local(mm)) { if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > - tlb_local_single_page_flush_ceiling) + tlb_local_single_page_flush_ceiling) { + full = true; _tlbiel_pid(pid, RIC_FLUSH_TLB); - else + } else { _tlbiel_va_range(start, end, pid, page_size, psize); + } } else { if (end == TLB_FLUSH_ALL || ((end - start) >> page_shift) > - tlb_single_page_flush_ceiling) + tlb_single_page_flush_ceiling) { + full = true; _tlbie_pid(pid, RIC_FLUSH_TLB); - else + } else { _tlbie_va_range(start, end, pid, page_size, psize); + } } preempt_enable(); + + return full; } #ifdef CONFIG_TRANSPARENT_HUGEPAGE