From patchwork Wed Nov 1 10:17:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anshuman Khandual X-Patchwork-Id: 832907 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yRklw4GFBz9sPs for ; Wed, 1 Nov 2017 21:21:48 +1100 (AEDT) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3yRklw3JZqzDrLB for ; Wed, 1 Nov 2017 21:21:48 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=khandual@linux.vnet.ibm.com; receiver=) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3yRkgN5VwLzDr4S for ; Wed, 1 Nov 2017 21:17:52 +1100 (AEDT) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id vA1AHk3a123881 for ; Wed, 1 Nov 2017 06:17:50 -0400 Received: from e06smtp11.uk.ibm.com (e06smtp11.uk.ibm.com [195.75.94.107]) by mx0a-001b2d01.pphosted.com with ESMTP id 2dycb287be-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 01 Nov 2017 06:17:50 -0400 Received: from localhost by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 1 Nov 2017 10:17:48 -0000 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp11.uk.ibm.com (192.168.101.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 1 Nov 2017 10:17:46 -0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id vA1AHjY523199790; Wed, 1 Nov 2017 10:17:45 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2DB3442041; Wed, 1 Nov 2017 10:12:59 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 892F94203F; Wed, 1 Nov 2017 10:12:57 +0000 (GMT) Received: from localhost.in.ibm.com (unknown [9.77.121.32]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 1 Nov 2017 10:12:57 +0000 (GMT) From: Anshuman Khandual To: linuxppc-dev@lists.ozlabs.org Subject: [RFC 2/2] powerpc/mm: Enable deferred flushing of TLB during reclaim Date: Wed, 1 Nov 2017 15:47:35 +0530 X-Mailer: git-send-email 2.9.3 In-Reply-To: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> References: <20171101101735.2318-1-khandual@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 17110110-0040-0000-0000-000004093456 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17110110-0041-0000-0000-000020ABBBF6 Message-Id: <20171101101735.2318-3-khandual@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-11-01_02:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1711010146 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: aneesh.kumar@linux.vnet.ibm.com, npiggin@gmail.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Deferred flushing can only be enabled on POWER9 DD2.0 processor onwards. Because prior versions of POWER9 and previous hash table based POWER processors will do TLB flushing in pte_get_and_clear() function itself which then prevents batching and eventual flush completion later on. Signed-off-by: Anshuman Khandual --- arch/powerpc/Kconfig | 1 + arch/powerpc/include/asm/tlbbatch.h | 30 +++++++++++++++++++++++ arch/powerpc/include/asm/tlbflush.h | 3 +++ arch/powerpc/mm/tlb-radix.c | 49 +++++++++++++++++++++++++++++++++++++ 4 files changed, 83 insertions(+) create mode 100644 arch/powerpc/include/asm/tlbbatch.h diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 809c468..f06b565 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -230,6 +230,7 @@ config PPC select SPARSE_IRQ select SYSCTL_EXCEPTION_TRACE select VIRT_TO_BUS if !PPC64 + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH if (PPC64 && PPC_BOOK3S) # # Please keep this list sorted alphabetically. # diff --git a/arch/powerpc/include/asm/tlbbatch.h b/arch/powerpc/include/asm/tlbbatch.h new file mode 100644 index 0000000..fc762ef --- /dev/null +++ b/arch/powerpc/include/asm/tlbbatch.h @@ -0,0 +1,30 @@ +#ifndef _ARCH_POWERPC_TLBBATCH_H +#define _ARCH_POWERPC_TLBBATCH_H + +#include + +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +#define MAX_BATCHED_MM 1024 + +struct arch_tlbflush_unmap_batch { + /* + * Each bit set is a CPU that potentially has a + * TLB entry for one of the PFN being flushed. + * This represents whether all deferred struct + * mm will be flushed for any given CPU. + */ + struct cpumask cpumask; + + /* All the deferred struct mm */ + struct mm_struct *mm[MAX_BATCHED_MM]; + unsigned long int nr_mm; + +}; + +extern bool arch_tlbbatch_should_defer(struct mm_struct *mm); +extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm); +#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +#endif /* _ARCH_POWERPC_TLBBATCH_H */ diff --git a/arch/powerpc/include/asm/tlbflush.h b/arch/powerpc/include/asm/tlbflush.h index 13dbcd4..2041923 100644 --- a/arch/powerpc/include/asm/tlbflush.h +++ b/arch/powerpc/include/asm/tlbflush.h @@ -20,6 +20,9 @@ */ #ifdef __KERNEL__ +#include +#include + #ifdef CONFIG_PPC_MMU_NOHASH /* * TLB flushing for software loaded TLB chips diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index b3e849c..506e7ed 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -12,6 +12,8 @@ #include #include #include +#include +#include #include #include @@ -519,3 +521,50 @@ extern void radix_kvm_prefetch_workaround(struct mm_struct *mm) } EXPORT_SYMBOL_GPL(radix_kvm_prefetch_workaround); #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ + +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static void clear_tlb(void *data) +{ + struct arch_tlbflush_unmap_batch *batch = data; + int i; + + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + for (i = 0; i < batch->nr_mm; i++) { + if (batch->mm[i]) + radix__local_flush_tlb_mm(batch->mm[i]); + } +} + +void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + smp_call_function_many(&batch->cpumask, (void *)clear_tlb, batch, 1); + batch->nr_mm = 0; + cpumask_clear(&batch->cpumask); +} + +void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, struct mm_struct *mm) +{ + WARN_ON(!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)); + + ++batch->nr_mm; + if (batch->nr_mm != MAX_BATCHED_MM) + batch->mm[batch->nr_mm] = mm; + else + pr_err("Deferred TLB flush: missed a struct mm\n"); + cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); +} + +bool arch_tlbbatch_should_defer(struct mm_struct *mm) +{ + if (!radix_enabled() || cpu_has_feature(CPU_FTR_POWER9_DD1)) + return false; + + if (!mm_is_thread_local(mm)) + return true; + + return false; +} +#endif