From patchwork Tue Jun 1 06:23:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1485890 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=YlsXlVsD; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FvMYv6mQ4z9sSs for ; Tue, 1 Jun 2021 16:24:03 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4FvMYw11m1z3btd for ; Tue, 1 Jun 2021 16:24:04 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=YlsXlVsD; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::536; helo=mail-pg1-x536.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=YlsXlVsD; dkim-atps=neutral Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4FvMY23x8Vz2yWt for ; Tue, 1 Jun 2021 16:23:18 +1000 (AEST) Received: by mail-pg1-x536.google.com with SMTP id t8so3706131pgb.9 for ; Mon, 31 May 2021 23:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=YlsXlVsDWoYMGPxW+QyS762YHdFOjE7CxBcZfRMKkUnI2PFDKCy9EUcSexkxG0c6tc JEOxq2/lzF3ffdvVbpz5kWXSrVKFyifnzzZzYWyhirC+Wl3O2KuMjuZe1++feSbhj221 a7jXtO00cFZrq5ZPUvJy+EZIuhgvZs8ffjQUkcF3fs5vG0jR70dh7b3/nPaDuA5S3xTF IYMaQMFZcwU8LY4YhFmqsekoeC1QlNfSLRQlkxdl6VXaguYbigTo1wp/2pO3dXha7wM7 XnJrwdwqNReQ9QySyKo9rj2h0BQJIDEw9ISx0Rd4lrqILj5rW4LgeYMIdludaJaGFZyP FpBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ecwzSsruDCg5HQ2i3NCpkmbXtWN83qkbLVUmQqDCWlk=; b=GiP654b2AD2gB3DPUCyN/BzL/e3/5csho4+xW5MMiW7/lxNTKfkJ5G5UYFC18Yi3GI d9RSAs5jnlA/xEfxtWyo30rroqypdvfu1dHVVu/wy8Kchyu0C+yh+iQabD6x+HTgfwC8 AnY+FrcMWxX5h3T8HeCTjXS6yd6Oahe8zAKllGmQd1oZqUoXts1+OWCfeSUCPVq9uVIH rf0jwD1zQVhNMUdJSMi8roRnACjcKjGD6M7fiDu2gOERWpMoeyCtZfcukL8ib0IaKlGA iqE9kTgPkIhwDlzq7xFup887YpgxCg97VCOWFcBIYJ4XGrQSUodQCZWOCvT16stWOWCx PEbg== X-Gm-Message-State: AOAM531eXLEXFxSjnYpljAXGfb5+LLpMlV8emFRU4iB4mpAJVNnCJBJI x8M3DC4THSeIhXU1UsioNt5v/EfxhdM= X-Google-Smtp-Source: ABdhPJwCAsEtEKpT8K7Nr300NGB/ZvSukjJyFZA0oMZFNmXMIB8cLbka5YfgBrjsXWlgMYSF+dBsCQ== X-Received: by 2002:a63:da0a:: with SMTP id c10mr26585212pgh.255.1622528595531; Mon, 31 May 2021 23:23:15 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:15 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v3 1/4] lazy tlb: introduce lazy mm refcount helper functions Date: Tue, 1 Jun 2021 16:23:00 +1000 Message-Id: <20210601062303.3932513-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Randy Dunlap , linux-kernel@vger.kernel.org, Nicholas Piggin , linux-mm@kvack.org, Andy Lutomirski , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add explicit _lazy_tlb annotated functions for lazy mm refcounting. This makes lazy mm references more obvious, and allows explicit refcounting to be removed if it is not used. Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/kernel/smp.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 4 ++-- include/linux/sched/mm.h | 11 +++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 11 +++++++---- kernel/sched/core.c | 15 ++++++++------- 9 files changed, 34 insertions(+), 19 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 827b50f1c73e..1b4a41aad793 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,7 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 2e05c783440a..fb0bdfc67366 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1541,7 +1541,7 @@ void start_secondary(void *unused) { unsigned int cpu = raw_smp_processor_id(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 409e61210789..2962082787c0 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -663,10 +663,10 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } /* diff --git a/fs/exec.c b/fs/exec.c index 18594f11c31f..ca0f8b1af23a 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1029,9 +1029,9 @@ static int exec_mmap(struct mm_struct *mm) setmax_mm_hiwater_rss(&tsk->signal->maxrss, old_mm); mm_update_next_owner(old_mm); mmput(old_mm); - return 0; + } else { + mmdrop_lazy_tlb(active_mm); } - mmdrop(active_mm); return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index e24b1fe348e3..bfd1baca5266 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -49,6 +49,17 @@ static inline void mmdrop(struct mm_struct *mm) __mmdrop(mm); } +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index e538518556f4..e87a89824e6c 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -602,7 +602,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index fd1c04193e18..8e87ec5f6be2 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -476,7 +476,7 @@ static void exit_mm(void) __set_current_state(TASK_RUNNING); mmap_read_lock(mm); } - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index fe3f2a40d61e..b70e28431a01 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1314,14 +1314,14 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + mmgrab(mm); + task_lock(tsk); /* Hold off tlb flush IPIs while switching mm's */ local_irq_disable(); active_mm = tsk->active_mm; - if (active_mm != mm) { - mmgrab(mm); + if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm; membarrier_update_current_mm(mm); switch_mm_irqs_off(active_mm, mm, tsk); @@ -1341,7 +1341,7 @@ void kthread_use_mm(struct mm_struct *mm) * mmdrop(), or explicitly with smp_mb(). */ if (active_mm != mm) - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); else smp_mb(); @@ -1375,10 +1375,13 @@ void kthread_unuse_mm(struct mm_struct *mm) local_irq_disable(); tsk->mm = NULL; membarrier_update_current_mm(NULL); + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 5226cc26a095..e359c76ea2e2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4229,13 +4229,14 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, so provide them here: * * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), + * provided by mmdrop_lazy_tlb(), * - a sync_core for SYNC_CORE. */ if (mm) { membarrier_mm_sync_core_before_usermode(mm); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -4299,9 +4300,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -4309,7 +4310,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -4325,7 +4326,7 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -8239,7 +8240,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* From patchwork Tue Jun 1 06:23:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1485891 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XDJ2kYvm; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FvMZN07x4z9sSs for ; Tue, 1 Jun 2021 16:24:28 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4FvMZN1kw5z3c0S for ; Tue, 1 Jun 2021 16:24:28 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XDJ2kYvm; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1029; helo=mail-pj1-x1029.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=XDJ2kYvm; dkim-atps=neutral Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4FvMY63Xxhz2ymY for ; Tue, 1 Jun 2021 16:23:22 +1000 (AEST) Received: by mail-pj1-x1029.google.com with SMTP id o17-20020a17090a9f91b029015cef5b3c50so807892pjp.4 for ; Mon, 31 May 2021 23:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=B2ZM54o9Bp1W7ui49t2FYRDuyLvPHwC8Il/uTHhhdMs=; b=XDJ2kYvmv/Z5M24qrG7KDMUE9p7aJ0RlQOBX1p/t2w4hIRyxR6UjTzA19/wWm8wdfC 2bssCYG5blnj9LMkj0KYjULkjvgmSvquyWjePNQYm//MxqhrHejkK0QcMGRIOOJbMpox 7lTAot474lVNko8WQIRuPXUlRf3A83o42kAVvfnakMYQ5V5QvGDcEzfdLv1MpTxUKbNv e+S+jSKkwHUTddDHEL3mGYjuaQDAWQaGCy1sXxnPFkwfKdci8juvdPKPN8bYXqfSQPrI UD3u5SitNYnPXIjG3reVtT1Z66MrR1ddVVxfrlRrOkUzRhQFpbERqoAxcac1BEAeYuRK nDEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=B2ZM54o9Bp1W7ui49t2FYRDuyLvPHwC8Il/uTHhhdMs=; b=CIHbCfc0cSw0JrVEAmmTsaVVgJADEfHMAGIXAuOyqp5iIEg81vPj3bNI1dGgbAQJd/ iiGsOtciracgQluYMsp7kC4mVmlcxoI9F2sZVOp54wmFttMYSnTLevdzN6iOfJy99IjM /sg8TjmfqVPvCO2EcJvWKRhrnbu3KcWjsFPEaxl7jUAPO632HIr0TVuwGdpk7IuAJqcZ bA0qmeabkjDQZFUbsWNj4C/+5P/ZG9xHbwttEK9l5UKVx+XOXTLgMYcq6IQn+brKV02/ vFTZltQ5hZ2YnPrl4Ok1cpq6M5vp/zhMU+oYpHzu3a9xcL3YFJDVyZeHC6TkVlZr3yOm vv+Q== X-Gm-Message-State: AOAM530pON6y4A0yKOg2Pc0VEmXeH4lOh7aDb4R/H7EEBaKV3EksT/EU A/K7yESFYqsxDLAorXKHnwg= X-Google-Smtp-Source: ABdhPJy2DUM8gibAhLieX5iGh16ca9VDqhVgs0t9sh7k8/bnT7er59ZtGvorelTu6yvv6DxaKgwUsw== X-Received: by 2002:a17:90b:3587:: with SMTP id mm7mr21958255pjb.71.1622528599607; Mon, 31 May 2021 23:23:19 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:19 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v3 2/4] lazy tlb: allow lazy tlb mm switching to be configurable Date: Tue, 1 Jun 2021 16:23:01 +1000 Message-Id: <20210601062303.3932513-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Randy Dunlap , linux-kernel@vger.kernel.org, Nicholas Piggin , linux-mm@kvack.org, Andy Lutomirski , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add CONFIG_MMU_LAZY_TLB which can be configured out to disable the lazy tlb mechanism entirely, and switches to init_mm when switching to a kernel thread. NOMMU systems could easily go without this and save a bit of code and the refcount atomics, because their mm switch is a no-op. They have not been switched over by default because the arch code needs to be audited and tested for lazy tlb mm refcounting and converted to _lazy_tlb refcounting if necessary. CONFIG_MMU_LAZY_TLB_REFCOUNT is also added, but it must always be enabled if CONFIG_MMU_LAZY_TLB is enabled until the next patch which provides an alternate scheme. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 26 ++++++++++++++ include/linux/sched/mm.h | 13 +++++-- kernel/sched/core.c | 75 ++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 4 ++- 4 files changed, 96 insertions(+), 22 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index c45b770d3579..276e1c1c0219 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -418,6 +418,32 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Enable "lazy TLB", which means a user->kernel thread context switch does not +# switch the mm to init_mm and the kernel thread takes a reference to the user +# mm to provide its kernel mapping. This is how Linux has traditionally worked +# (see Documentation/vm/active_mm.rst), for performance. Switching to and from +# idle thread is a performance-critical case. +# +# If mm context switches are inexpensive or free (in the case of NOMMU) then +# this could be disabled. +# +# It would make sense to have this depend on MMU, but need to audit and test +# the NOMMU architectures for lazy mm refcounting first. +config MMU_LAZY_TLB + def_bool y + depends on !NO_MMU_LAZY_TLB + +# This allows archs to disable MMU_LAZY_TLB. mmgrab/mmdrop in arch/ code has +# to be audited and switched to _lazy_tlb postfix as necessary. +config NO_MMU_LAZY_TLB + def_bool n + +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. +# For now, this must be enabled if MMU_LAZY_TLB is enabled. +config MMU_LAZY_TLB_REFCOUNT + def_bool y + depends on MMU_LAZY_TLB + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index bfd1baca5266..29e4638ad124 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -52,12 +52,21 @@ static inline void mmdrop(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch which relies on this. + */ + smp_mb(); + } } /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e359c76ea2e2..299c3eb12b2b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4171,7 +4171,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) { struct rq *rq = this_rq(); - struct mm_struct *mm = rq->prev_mm; + struct mm_struct *mm = NULL; long prev_state; /* @@ -4190,7 +4190,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count())) preempt_count_set(FORK_PREEMPT_COUNT); - rq->prev_mm = NULL; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm = NULL; +#endif /* * A task struct has one reference for the use as "current". @@ -4282,22 +4285,10 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev) calculate_sigpending(); } -/* - * context_switch - switch to the new MM and the new thread's register state. - */ -static __always_inline struct rq * -context_switch(struct rq *rq, struct task_struct *prev, - struct task_struct *next, struct rq_flags *rf) +static __always_inline void +context_switch_mm(struct rq *rq, struct task_struct *prev, + struct task_struct *next) { - prepare_task_switch(rq, prev, next); - - /* - * For paravirt, this is coupled with an exit in switch_to to - * combine the page table reload and the switch backend into - * one hypercall. - */ - arch_start_context_switch(prev); - /* * kernel -> kernel lazy + transfer active * user -> kernel lazy + mmgrab_lazy_tlb() active @@ -4326,11 +4317,57 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel - /* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + /* Will mmdrop_lazy_tlb() in finish_task_switch(). */ + rq->prev_lazy_mm = prev->active_mm; prev->active_mm = NULL; +#else + /* + * Without MMU_LAZY_REFCOUNT there is no lazy + * tracking (because no rq->prev_lazy_mm) in + * finish_task_switch, so no mmdrop_lazy_tlb(), + * so no memory barrier for membarrier (see the + * membarrier comment in finish_task_switch()). + * Do it here. + */ + smp_mb(); +#endif } } +} + +static __always_inline void +context_switch_mm_nolazy(struct rq *rq, struct task_struct *prev, + struct task_struct *next) +{ + if (!next->mm) + next->active_mm = &init_mm; + membarrier_switch_mm(rq, prev->active_mm, next->active_mm); + switch_mm_irqs_off(prev->active_mm, next->active_mm, next); + if (!prev->mm) + prev->active_mm = NULL; +} + +/* + * context_switch - switch to the new MM and the new thread's register state. + */ +static __always_inline struct rq * +context_switch(struct rq *rq, struct task_struct *prev, + struct task_struct *next, struct rq_flags *rf) +{ + prepare_task_switch(rq, prev, next); + + /* + * For paravirt, this is coupled with an exit in switch_to to + * combine the page table reload and the switch backend into + * one hypercall. + */ + arch_start_context_switch(prev); + + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB)) + context_switch_mm(rq, prev, next); + else + context_switch_mm_nolazy(rq, prev, next); rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index a189bec13729..0729cf19a987 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -961,7 +961,9 @@ struct rq { struct task_struct *idle; struct task_struct *stop; unsigned long next_balance; - struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags; u64 clock; From patchwork Tue Jun 1 06:23:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1485892 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ePlorr20; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FvMZy5CnMz9sSs for ; Tue, 1 Jun 2021 16:24:58 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4FvMZy744nz3c51 for ; Tue, 1 Jun 2021 16:24:58 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ePlorr20; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ePlorr20; dkim-atps=neutral Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4FvMYB3cm7z305v for ; Tue, 1 Jun 2021 16:23:26 +1000 (AEST) Received: by mail-pj1-x102e.google.com with SMTP id b15-20020a17090a550fb029015dad75163dso1263730pji.0 for ; Mon, 31 May 2021 23:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7SY6QzuWUSlWj33T26/GRZSVwi+Ld4IHiibBoCb52p4=; b=ePlorr20RYdU83BpwKX/s0RfyvSJGLZuiSfUt1Q5Tf+Uj6s0wJkGx8MFmTlzTq06Ly /QdSin1Qqs2l0J37C68YVuBoCY22xVUnKkbWT5c/25xPZUE4hwRl5CveuKHLOW9KwoLz WMUnhDGfB+gMrKJPdIvTLtMDs5EgLCysNJyB17ago3WfOtWSwQRzdwE9GUfx+u24xoXb YvUh+XKO7EACTncuCcfNhm9MUjn8I6F0IEFKp7B02dAPD0YE4xz1HbQBw17+5uziSWyH 6Hbt03PuVPQDsas/U9JfW6GikTKlAqSmRjtpSQ956IQanhxIOkALHMsFIlrry/0xO4L8 Jh1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7SY6QzuWUSlWj33T26/GRZSVwi+Ld4IHiibBoCb52p4=; b=ms6Kk4s3kvkdRpnnJd1OrIQ/ZmkoHx76sO53Wjt7rj9gCwFPQqI5uJ0A+1hsCdLr6T q29JhQCPpAQ5/ypJ3Lx6AJaXA2jMfaHcmJ3fxRxDf2QUymnlEhlW47+pXAmK1fn/pag8 4jVMtnqVLgmpPjENpmnJOJ4ZvGwI/7q+cEhZkegGFaFtwhMtHmzzmBCEZJGopK66aOw4 h2SXYorY+P7TnEmnIF0UKGskERqTTCv4iqseunO8FQEgbXLzO9rncEqvQnaWam4z1k9j GL71dArqSvlukw7nr7nP3jWl+WS4K2p2g8GgaKmHrmKx3QIryWLcTZTiSRAKMK1DTINt f9RA== X-Gm-Message-State: AOAM531CGKlIvsF6emStP4OSzRTP/cWYdkgH2H9VhHLLTbcln4Kmh5Iq Inct5oBCu1Ps9EJnSv/L8jE= X-Google-Smtp-Source: ABdhPJw09TNZqBmKx3WQDrsKTZlhlpstv4mz2E1lJZnWicTzFCQBo5ZLvZlMnKhALwZzNq6+juRuaA== X-Received: by 2002:a17:90a:5106:: with SMTP id t6mr3315886pjh.216.1622528603683; Mon, 31 May 2021 23:23:23 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:23 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v3 3/4] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Tue, 1 Jun 2021 16:23:02 +1000 Message-Id: <20210601062303.3932513-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Randy Dunlap , linux-kernel@vger.kernel.org, Nicholas Piggin , linux-mm@kvack.org, Andy Lutomirski , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 14 +++++++++++++- kernel/fork.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index 276e1c1c0219..91e1882e3284 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -439,11 +439,23 @@ config NO_MMU_LAZY_TLB def_bool n # Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. -# For now, this must be enabled if MMU_LAZY_TLB is enabled. config MMU_LAZY_TLB_REFCOUNT def_bool y depends on MMU_LAZY_TLB +# Instead of refcounting the lazy mm struct for kernel thread references +# (which can cause contention with multi-threaded apps on large multiprocessor +# systems), this option causes __mmdrop to IPI all CPUs in the mm_cpumask and +# switch to init_mm if they were using the to-be-freed mm as the lazy tlb. To +# implement this, architectures must use _lazy_tlb variants of mm refcounting +# when releasing kernel thread mm references, and mm_cpumask must include at +# least all possible CPUs in which the mm might be lazy, at the time of the +# final mmdrop. mmgrab/mmdrop in arch/ code must be switched to _lazy_tlb +# postfix as necessary. +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index dc06afd725cb..d485c24426a0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -674,6 +674,53 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -683,7 +730,12 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); + mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); From patchwork Tue Jun 1 06:23:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1485893 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=TMuuo1b+; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FvMbT4NPzz9sSs for ; Tue, 1 Jun 2021 16:25:25 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4FvMbT60kWz3cB6 for ; Tue, 1 Jun 2021 16:25:25 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=TMuuo1b+; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=TMuuo1b+; dkim-atps=neutral Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4FvMYG4Ccgz3014 for ; Tue, 1 Jun 2021 16:23:30 +1000 (AEST) Received: by mail-pl1-x62b.google.com with SMTP id 11so2817183plk.12 for ; Mon, 31 May 2021 23:23:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=TMuuo1b+x5/KjM8834zLhSTP9eYuo7t0tW46rm7j4SnXrgLEDWE0TOlUaFMqu8a+wI vbs8XSGlHrRcHcTNztnJ6FzCBPFzIdXlEHvPisjfSNJtPoi22YsCPwpfGpRVCfOzBEyA qIpgYtAQFvdrhvVyPuJFVlv69Dqy2++lkPLlvSfYFcFCmgn46h+CL735k7uWPO2iR6ib F/kGzqky6oCb6NGOI9e7WIPMpMGVye8FitjlLICcOtIpc9ld3zB5Wx8ELsL60gT36dHI 97jMnbIhzGPVji0u7wu9KoHMx8MMSH5LNriPf5FvHIo0aVVgBAQXAvYl2+p+Q7UssMwq afpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PZkiKtXqsSUWsFPxoaMD52BWVXcB9txjh0r70N9gfwg=; b=DCqIDqqfmTLEYimHS8v4Jeiy66ABLSzgvZcmhaBmiho+FVpIPvu+O7+A0E1OP6lqLW 7u8bCtyk4mrvke5SWCgPoafgBCBvHxccRc22E/R+oV9GT+BEd4vGZ4F/8ZpkHiGGo59X N4KlCdPshwgXxfXvqG2wz/MhbCs5E7qq2yM721QKIoT1d/SlyQRJiXUtsoi4vfA/JUFS MbkoWEeXd4Qg2TcwfLf4J14sPz09bO1b2TAoKiksGyEBGU+j+uhHruc/JhJJ097OUCmx cI2UFpb4urTxUbXGVlY5uopXQzSwr1ZvXyLXWUWhgtFMbhA3qwSlHbvq6FXlTyQqJA9H BcPQ== X-Gm-Message-State: AOAM530f7X6qHpzZZmyXvdkts7/drDJdx0inGU8Dz3gsVgnc8zm9Z38z 7oii/7OXan5gxtoBDdGRRME= X-Google-Smtp-Source: ABdhPJzO7M0Hh974CEgDAeZuPzkos1it6HgA9FTrTsKtUq98J+3jqEoDd6tOI7VI/NdE9fIIB6URYw== X-Received: by 2002:a17:90b:1094:: with SMTP id gj20mr14135765pjb.45.1622528607822; Mon, 31 May 2021 23:23:27 -0700 (PDT) Received: from bobo.ibm.com (60-241-69-122.static.tpgi.com.au. [60.241.69.122]) by smtp.gmail.com with ESMTPSA id h1sm12519100pfh.72.2021.05.31.23.23.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 23:23:27 -0700 (PDT) From: Nicholas Piggin To: Andrew Morton Subject: [PATCH v3 4/4] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Date: Tue, 1 Jun 2021 16:23:03 +1000 Message-Id: <20210601062303.3932513-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210601062303.3932513-1-npiggin@gmail.com> References: <20210601062303.3932513-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Randy Dunlap , linux-kernel@vger.kernel.org, Nicholas Piggin , linux-mm@kvack.org, Andy Lutomirski , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 088dd2afcfe4..8a092eedc692 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -252,6 +252,7 @@ config PPC select IRQ_FORCED_THREADING select MMU_GATHER_PAGE_SIZE select MMU_GATHER_RCU_TABLE_FREE + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select MODULES_USE_ELF_RELA select NEED_DMA_MAP_STATE if PPC64 || NOT_COHERENT_CACHE select NEED_SG_DMA_LENGTH