From patchwork Wed Nov 1 10:17:34 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 832906 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yRkkM6bQKz9sPs for ; Wed, 1 Nov 2017 21:20:27 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yRkkM5grGzDrJh for ; Wed, 1 Nov 2017 21:20:27 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=khandual@linux.vnet.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yRkgL5KWCzDr5Q for ; Wed, 1 Nov 2017 21:17:50 +1100 (AEDT) Received: from pps.filterd (m0098394.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vA1AFYcA128029 for ; Wed, 1 Nov 2017 06:17:48 -0400 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0a-001b2d01.pphosted.com with ESMTP id 2dyaq8cwkg-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 01 Nov 2017 06:17:47 -0400 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 1 Nov 2017 10:17:45 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 1 Nov 2017 10:17:42 -0000 Received: from d06av24.portsmouth.uk.ibm.com (d06av24.portsmouth.uk.ibm.com [9.149.105.60]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vA1AHg3n25755790; Wed, 1 Nov 2017 10:17:42 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DF28142041; Wed, 1 Nov 2017 10:12:55 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5FC6D4203F; Wed, 1 Nov 2017 10:12:54 +0000 (GMT) Received: from localhost.in.ibm.com (unknown [9.77.121.32]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 1 Nov 2017 10:12:54 +0000 (GMT) From: Anshuman Khandual To: linuxppc-dev@lists.ozlabs.org Subject: [RFC 1/2] mm/tlbbatch: Introduce arch_tlbbatch_should_defer() Date: Wed, 1 Nov 2017 15:47:34 +0530 X-Mailer: git-send-email 2.9.3 In-Reply-To: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> References: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17110110-0020-0000-0000-000003C7309C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17110110-0021-0000-0000-0000425C2AD0 Message-Id: <20171101101735.2318-2-khandual@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-11-01_02:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1711010146 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.vnet.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The entire scheme of deferred TLB flush in reclaim path rests on the fact that the cost to refill TLB entries is less than flushing out individual entries by sending IPI to remote CPUs. But architecture can have different ways to evaluate that. Hence apart from checking TTU_BATCH_FLUSH in the TTU flags, rest of the decision should be architecture specific. Signed-off-by: Anshuman Khandual --- arch/x86/include/asm/tlbflush.h | 12 ++++++++++++ mm/rmap.c | 9 +-------- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index c4aed0d..5875f2c 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -366,6 +366,18 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) void native_flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info); +static inline void arch_tlbbatch_should_defer(struct mm_struct *mm) +{ + bool should_defer = false; + + /* If remote CPUs need to be flushed then defer batch the flush */ + if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) + should_defer = true; + put_cpu(); + + return should_defer; +} + static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm) { diff --git a/mm/rmap.c b/mm/rmap.c index b874c47..bfbfe92 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -627,17 +627,10 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) */ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) { - bool should_defer = false; - if (!(flags & TTU_BATCH_FLUSH)) return false; - /* If remote CPUs need to be flushed then defer batch the flush */ - if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) - should_defer = true; - put_cpu(); - - return should_defer; + return arch_tlbbatch_should_defer(mm); } /* From patchwork Wed Nov 1 10:17:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 832907 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yRklw4GFBz9sPs for ; Wed, 1 Nov 2017 21:21:48 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yRklw3JZqzDrLB for ; Wed, 1 Nov 2017 21:21:48 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=khandual@linux.vnet.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yRkgN5VwLzDr4S for ; Wed, 1 Nov 2017 21:17:52 +1100 (AEDT) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vA1AHk3a123881 for ; Wed, 1 Nov 2017 06:17:50 -0400 Received: from e06smtp11.uk.ibm.com (e06smtp11.uk.ibm.com [195.75.94.107]) by mx0a-001b2d01.pphosted.com with ESMTP id 2dycb287be-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 01 Nov 2017 06:17:50 -0400 Received: from localhost by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 1 Nov 2017 10:17:48 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp11.uk.ibm.com (192.168.101.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 1 Nov 2017 10:17:46 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vA1AHjY523199790; Wed, 1 Nov 2017 10:17:45 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2DB3442041; Wed, 1 Nov 2017 10:12:59 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 892F94203F; Wed, 1 Nov 2017 10:12:57 +0000 (GMT) Received: from localhost.in.ibm.com (unknown [9.77.121.32]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 1 Nov 2017 10:12:57 +0000 (GMT) From: Anshuman Khandual To: linuxppc-dev@lists.ozlabs.org Subject: [RFC 2/2] powerpc/mm: Enable deferred flushing of TLB during reclaim Date: Wed, 1 Nov 2017 15:47:35 +0530 X-Mailer: git-send-email 2.9.3 In-Reply-To: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> References: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17110110-0040-0000-0000-000004093456 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17110110-0041-0000-0000-000020ABBBF6 Message-Id: <20171101101735.2318-3-khandual@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-11-01_02:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1711010146 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.vnet.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Deferred flushing can only be enabled on POWER9 DD2.0 processor onwards. Because prior versions of POWER9 and previous hash table based POWER processors will do TLB flushing in pte_get_and_clear() function itself which then prevents batching and eventual flush completion later on. Signed-off-by: Anshuman Khandual --- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/tlbbatch.h | 30 +++++++++++++++++++++++ arch/powerpc/include/asm/tlbflush.h | 3 +++ arch/powerpc/mm/tlb-radix.c | 49 +++++++++++++++++++++++++++++++++++++ 4 files changed, 83 insertions(+) create mode 100644 arch/powerpc/include/asm/tlbbatch.h diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 809c468..f06b565 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -230,6 +230,7 @@ config PPC select SPARSE_IRQ select SYSCTL_EXCEPTION_TRACE select VIRT_TO_BUS if !PPC64 + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if (PPC64 && PPC_BOOK3S) # # Please keep this list sorted alphabetically. # diff --git a/arch/powerpc/include/asm/tlbbatch.h b/arch/powerpc/include/asm/tlbbatch.h new file mode 100644 index 0000000..fc762ef --- /dev/null +++ b/arch/powerpc/include/asm/tlbbatch.h @@ -0,0 +1,30 @@ +#ifndef _ARCH_POWERPC_TLBBATCH_H +#define _ARCH_POWERPC_TLBBATCH_H + +#include + +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +#define MAX_BATCHED_MM 1024 + +struct arch_tlbflush_unmap_batch { + /* + * Each bit set is a CPU that potentially has a + * TLB entry for one of the PFN being flushed. + * This represents whether all deferred struct + * mm will be flushed for any given CPU. + */ + struct cpumask cpumask; + + /* All the deferred struct mm */ + struct mm_struct *mm[MAX_BATCHED_MM]; + unsigned long int nr_mm; + +}; + +extern bool arch_tlbbatch_should_defer(struct mm_struct *mm); +extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm); +#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +#endif /* _ARCH_POWERPC_TLBBATCH_H */ diff --git a/arch/powerpc/include/asm/tlbflush.h b/arch/powerpc/include/asm/tlbflush.h index 13dbcd4..2041923 100644 --- a/arch/powerpc/include/asm/tlbflush.h +++ b/arch/powerpc/include/asm/tlbflush.h @@ -20,6 +20,9 @@ */ #ifdef __KERNEL__ +#include +#include + #ifdef CONFIG_PPC_MMU_NOHASH /* * TLB flushing for software loaded TLB chips diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index b3e849c..506e7ed 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -12,6 +12,8 @@ #include #include #include +#include +#include #include #include @@ -519,3 +521,50 @@ extern void radix_kvm_prefetch_workaround(struct mm_struct *mm) } EXPORT_SYMBOL_GPL(radix_kvm_prefetch_workaround); #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ + +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static void clear_tlb(void *data) +{ + struct arch_tlbflush_unmap_batch *batch = data; + int i; + + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + for (i = 0; i < batch->nr_mm; i++) { + if (batch->mm[i]) + radix__local_flush_tlb_mm(batch->mm[i]); + } +} + +void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + smp_call_function_many(&batch->cpumask, (void *)clear_tlb, batch, 1); + batch->nr_mm = 0; + cpumask_clear(&batch->cpumask); +} + +void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm) +{ + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + ++batch->nr_mm; + if (batch->nr_mm != MAX_BATCHED_MM) + batch->mm[batch->nr_mm] = mm; + else + pr_err("Deferred TLB flush: missed a struct mm\n"); + cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); +} + +bool arch_tlbbatch_should_defer(struct mm_struct *mm) +{ + if (!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)) + return false; + + if (!mm_is_thread_local(mm)) + return true; + + return false; +} +#endif