From patchwork Sat Nov 28 16:01:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407568 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxD32551z9s0b for ; Sun, 29 Nov 2020 03:05:23 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=fddofWue; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxD31mHlzDsZB for ; Sun, 29 Nov 2020 03:05:23 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=fddofWue; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx854SF1zDrQX for ; Sun, 29 Nov 2020 03:01:57 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id q10so7114026pfn.0 for ; Sat, 28 Nov 2020 08:01:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gttnBtFOibE2tP2Qz24NLSDRYoTGub5YJjfSoizC0iA=; b=fddofWue+O7PL7JyLzbrmfEZ1AmPAs4YpBi425i4fGaiDzUtTP1Cjb/kUW5gGhUTUG zroDx2CGFhUo8MVoFIUF+MP8b1p2psaIqXLAsQr05kPX/p4d/S7xxuUvrdD122e3zNTk +yWVb5RpbczP24WMcIBwgAESCFaKaOazX1xmiySUU7Q4UENN4oLJouX6ftvashMBqtNo k9k7ojLvXh7wWrdH5HhDZtS64VIpEQm0fl12lwLV2acHXX+2Z4z3IDRijQ1jKh4kn6f8 te2+ctqgL6SkhFEZCO0U/smJPgELN01XFV1O3fSOhYhR0PJS+cMx8avXrp2TZTREkvoq hJRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gttnBtFOibE2tP2Qz24NLSDRYoTGub5YJjfSoizC0iA=; b=NBL9la92XVfjy9BXH+VO/3rq1ck+9S1736fIUHqHlROplSjwMSE9R3WqHGsJvjCR6v 1BkjhqHcJI22QSI84OSpIVAC7vK8xuUkqBHec1Q6iMYw32DhwrW3ZLUbY3eJuacn8h+w dEHkaHExw6iFtK7aqS8Fjcv206ST6XgwonCf+IsFpVPx/8kpCjdjMLYZgipgkFqpsUcK ApvFLLSmBERfsjrEJrQW66/FTvepdh3LE5afXDloluV6m4fY10mz9pkkL5UzbANSmvne lhC5D87IvBGAJPBUjknqhwht/rGilLYMYLajXcTNYgdaiPosF6XQooJ8VoPIIyz9OSkY uBEQ== X-Gm-Message-State: AOAM533IzxqEvZTV+ycJGr0i7wjINcoGHowuOENkO2MltRLRcmn7g2BX S5fbcEQUZqn10BBfY0YRywM= X-Google-Smtp-Source: ABdhPJzSlEN2fFhGsSm+AZ067hVpNTL/kp5XtHU/fInnX4+IsK4D3+Ht2oCRMtspesP14x3iR4yVHg== X-Received: by 2002:a17:90b:30cb:: with SMTP id hi11mr16522183pjb.94.1606579315315; Sat, 28 Nov 2020 08:01:55 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.01.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:01:54 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 1/8] lazy tlb: introduce exit_lazy_tlb Date: Sun, 29 Nov 2020 02:01:34 +1000 Message-Id: <20201128160141.1003903-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This is called at points where a lazy mm is switched away or made not lazy (by its owner switching back). Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 1 + arch/powerpc/mm/book3s64/radix_tlb.c | 1 + fs/exec.c | 6 ++++-- include/asm-generic/mmu_context.h | 21 +++++++++++++++++++++ kernel/kthread.c | 1 + kernel/sched/core.c | 2 ++ 6 files changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 827b50f1c73e..43eb1bfba466 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -253,6 +253,7 @@ static int ecard_init_mm(void) current->mm = mm; current->active_mm = mm; activate_mm(active_mm, mm); + exit_lazy_tlb(active_mm, current); mmdrop(active_mm); ecard_init_pgtables(mm); return 0; diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index b487b489d4b6..ac3fec03926a 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -661,6 +661,7 @@ static void do_exit_flush_lazy_tlb(void *arg) mmgrab(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); + exit_lazy_tlb(mm, current); mmdrop(mm); } diff --git a/fs/exec.c b/fs/exec.c index 547a2390baf5..4b4dea1bb7ba 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1017,6 +1017,8 @@ static int exec_mmap(struct mm_struct *mm) if (!IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) local_irq_enable(); activate_mm(active_mm, mm); + if (!old_mm) + exit_lazy_tlb(active_mm, tsk); if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) local_irq_enable(); tsk->mm->vmacache_seqnum = 0; @@ -1028,9 +1030,9 @@ static int exec_mmap(struct mm_struct *mm) setmax_mm_hiwater_rss(&tsk->signal->maxrss, old_mm); mm_update_next_owner(old_mm); mmput(old_mm); - return 0; + } else { + mmdrop(active_mm); } - mmdrop(active_mm); return 0; } diff --git a/include/asm-generic/mmu_context.h b/include/asm-generic/mmu_context.h index 91727065bacb..4626d0020e65 100644 --- a/include/asm-generic/mmu_context.h +++ b/include/asm-generic/mmu_context.h @@ -24,6 +24,27 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, } #endif +/* + * exit_lazy_tlb - Called after switching away from a lazy TLB mode mm. + * + * mm: the lazy mm context that was switched + * tsk: the task that was switched to (with a non-lazy mm) + * + * mm may equal tsk->mm. + * mm and tsk->mm will not be NULL. + * + * Note this is not symmetrical to enter_lazy_tlb, this is not + * called when tasks switch into the lazy mm, it's called after the + * lazy mm becomes non-lazy (either switched to a different mm or the + * owner of the mm returns). + */ +#ifndef exit_lazy_tlb +static inline void exit_lazy_tlb(struct mm_struct *mm, + struct task_struct *tsk) +{ +} +#endif + /** * init_new_context - Initialize context of a new mm_struct. * @tsk: task struct for the mm diff --git a/kernel/kthread.c b/kernel/kthread.c index 933a625621b8..e380302aac13 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1250,6 +1250,7 @@ void kthread_use_mm(struct mm_struct *mm) } tsk->mm = mm; switch_mm_irqs_off(active_mm, mm, tsk); + exit_lazy_tlb(active_mm, tsk); local_irq_enable(); task_unlock(tsk); #ifdef finish_arch_post_lock_switch diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e7e453492cff..dcc46039ade5 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3765,6 +3765,8 @@ context_switch(struct rq *rq, struct task_struct *prev, switch_mm_irqs_off(prev->active_mm, next->mm, next); if (!prev->mm) { // from kernel + exit_lazy_tlb(prev->active_mm, next); + /* will mmdrop() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; From patchwork Sat Nov 28 16:01:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407569 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxFx6Tt8z9s0b for ; Sun, 29 Nov 2020 03:07:01 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=tOVJyMY5; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxFx54kvzDsM4 for ; Sun, 29 Nov 2020 03:07:01 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=tOVJyMY5; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx896RSwzDrQX for ; Sun, 29 Nov 2020 03:02:01 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id e23so1868430pgk.12 for ; Sat, 28 Nov 2020 08:02:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6Twyf0/RNSa3vRRlfp1Oet03l6vxb8aXJKdhGNWclcs=; b=tOVJyMY59cAHL0Zzh3IdRygvgG/jwamJDHuys8IGkS0xRl9q8lB2RMbbQR9JlnInyL Eiuv17njNNKkWNknihj7Gt/Y7sYfrxSdsi8OC7T/3P/067pVTZC+d6GwyK3b+oxynnHv ualnu2jlm8a7xOIh0iPc0la4kJAcrDbOYjgazdmLJTBG3CCm2GZT9ruRdlt+io/y2DTA aei7B3l2rG8ORKFGCWYu5he3af+ipX4rgqppNP+1mnOSxSOHvlzwY6p9/sqY9ECmBnHP u2Uj3PTMr7YGFUnRDq3YRKNrLuM3DfdvKxH7KW1/3XGwL0QQe7tOEpxh9XVVA/35Gknb kQdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6Twyf0/RNSa3vRRlfp1Oet03l6vxb8aXJKdhGNWclcs=; b=XUyZ0nFzK+fx/A7+pUNrKS2fRVpOBx2Vld+kzpdjRdpNwmJl0itGhxTimioPca8BXQ VXjStshqyrTwJjYNGwsL7HDcWM2vkxaVfVSpOae9aQiIRGjefwpt/NjfrtrxYG4Fw3bo bhULmebqNxCxoeLmPcIv42djBtl2PFfgjQBLygoLj1rULF6yU2aS3eWaLMFRlp0gEuii 9nIa5IifoEO5W53S/HFmLnafOSkvtAHRp6l+wxI3xAFRl18PIcQUZ0xcjibhQRozgXvS VY3r9OXZSb7k53c0RNBzDhaD2E1rJo958X5aZKSZIfaXsN2ufHFQD7W9mU8SbePDteeD REpg== X-Gm-Message-State: AOAM532Wlj68DtVTGOkdDUVoxkbbi2D0l6BYGE6hVzOJlY6DinQjqRXQ Bt7xFRRiExqE4qDnHo7JJxU= X-Google-Smtp-Source: ABdhPJwmFPhtuAX9fmgjTV1NdLL+hs+IbGGuWhOPDdSX9td7UVffTtdAIrAze3tOKAkjX2mNhJn8wA== X-Received: by 2002:a17:90a:460b:: with SMTP id w11mr16545731pjg.12.1606579319860; Sat, 28 Nov 2020 08:01:59 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.01.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:01:59 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 2/8] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode Date: Sun, 29 Nov 2020 02:01:35 +1000 Message-Id: <20201128160141.1003903-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" And get rid of the generic sync_core_before_usermode facility. This is functionally a no-op in the core scheduler code, but it also catches This helper is the wrong way around I think. The idea that membarrier state requires a core sync before returning to user is the easy one that does not need hiding behind membarrier calls. The gap in core synchronization due to x86's sysret/sysexit and lazy tlb mode, is the tricky detail that is better put in x86 lazy tlb code. Consider if an arch did not synchronize core in switch_mm either, then membarrier_mm_sync_core_before_usermode would be in the wrong place but arch specific mmu context functions would still be the right place. There is also a exit_lazy_tlb case that is not covered by this call, which could be a bugs (kthread use mm the membarrier process's mm then context switch back to the process without switching mm or lazy mm switch). This makes lazy tlb code a bit more modular. Signed-off-by: Nicholas Piggin --- .../membarrier-sync-core/arch-support.txt | 6 ++++- arch/x86/include/asm/mmu_context.h | 27 +++++++++++++++++++ include/linux/sched/mm.h | 14 ---------- kernel/cpu.c | 4 ++- kernel/sched/core.c | 16 +++++------ 5 files changed, 42 insertions(+), 25 deletions(-) diff --git a/Documentation/features/sched/membarrier-sync-core/arch-support.txt b/Documentation/features/sched/membarrier-sync-core/arch-support.txt index 47e6903f47a5..0763a63a7097 100644 --- a/Documentation/features/sched/membarrier-sync-core/arch-support.txt +++ b/Documentation/features/sched/membarrier-sync-core/arch-support.txt @@ -5,6 +5,10 @@ # # Architecture requirements # +# If your architecture returns to user-space through non-core-serializing +# instructions, you need to ensure these are done in switch_mm and exit_lazy_tlb +# (if lazy tlb switching is implemented). +# # * arm/arm64/powerpc # # Rely on implicit context synchronization as a result of exception return @@ -24,7 +28,7 @@ # instead on write_cr3() performed by switch_mm() to provide core serialization # after changing the current mm, and deal with the special case of kthread -> # uthread (temporarily keeping current mm into active_mm) by issuing a -# sync_core_before_usermode() in that specific case. +# serializing instruction in exit_lazy_mm() in that specific case. # ----------------------- | arch |status| diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 36afcbea6a9f..8094893254f1 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -6,12 +6,14 @@ #include #include #include +#include #include #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -94,6 +96,31 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); +#ifdef CONFIG_MEMBARRIER +/* + * Ensure that a core serializing instruction is issued before returning + * to user-mode, if a SYNC_CORE was requested. x86 implements return to + * user-space through sysexit, sysrel, and sysretq, which are not core + * serializing. + * + * See the membarrier comment in finish_task_switch as to why this is done + * in exit_lazy_tlb. + */ +#define exit_lazy_tlb exit_lazy_tlb +static inline void exit_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) +{ + /* Switching mm is serializing with write_cr3 */ + if (tsk->mm != mm) + return; + + if (likely(!(atomic_read(&mm->membarrier_state) & + MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE))) + return; + + sync_core_before_usermode(); +} +#endif + /* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve(). diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index d5ece7a9a403..2c6bcdf76d99 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -7,7 +7,6 @@ #include #include #include -#include /* * Routines for handling mm_structs @@ -335,16 +334,6 @@ enum { #include #endif -static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm) -{ - if (current->mm != mm) - return; - if (likely(!(atomic_read(&mm->membarrier_state) & - MEMBARRIER_STATE_PRIVATE_EXPEDITED_SYNC_CORE))) - return; - sync_core_before_usermode(); -} - extern void membarrier_exec_mmap(struct mm_struct *mm); #else @@ -358,9 +347,6 @@ static inline void membarrier_arch_switch_mm(struct mm_struct *prev, static inline void membarrier_exec_mmap(struct mm_struct *mm) { } -static inline void membarrier_mm_sync_core_before_usermode(struct mm_struct *mm) -{ -} #endif #endif /* _LINUX_SCHED_MM_H */ diff --git a/kernel/cpu.c b/kernel/cpu.c index 6ff2578ecf17..134688d79589 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -572,7 +572,9 @@ static int finish_cpu(unsigned int cpu) /* * idle_task_exit() will have switched to &init_mm, now - * clean up any remaining active_mm state. + * clean up any remaining active_mm state. exit_lazy_tlb + * is not done, if an arch did any accounting in these + * functions it would have to be added. */ if (mm != &init_mm) idle->active_mm = &init_mm; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index dcc46039ade5..e4e8cebd82e2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3620,22 +3620,19 @@ static struct rq *finish_task_switch(struct task_struct *prev) kcov_finish_switch(current); fire_sched_in_preempt_notifiers(current); + /* * When switching through a kernel thread, the loop in * membarrier_{private,global}_expedited() may have observed that * kernel thread and not issued an IPI. It is therefore possible to * schedule between user->kernel->user threads without passing though - * switch_mm(). Membarrier requires a barrier after storing to - * rq->curr, before returning to userspace, so provide them here: - * - * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly - * provided by mmdrop(), - * - a sync_core for SYNC_CORE. + * switch_mm(). Membarrier requires a full barrier after storing to + * rq->curr, before returning to userspace, for + * {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by mmdrop(). */ - if (mm) { - membarrier_mm_sync_core_before_usermode(mm); + if (mm) mmdrop(mm); - } + if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) prev->sched_class->task_dead(prev); @@ -6689,6 +6686,7 @@ void idle_task_exit(void) BUG_ON(current != this_rq()->idle); if (mm != &init_mm) { + /* enter_lazy_tlb is not done because we're about to go down */ switch_mm(mm, &init_mm, current); finish_arch_post_lock_switch(); } From patchwork Sat Nov 28 16:01:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407570 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxJ121r1z9s0b for ; Sun, 29 Nov 2020 03:08:49 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=hAkQC7bx; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxJ06W6tzF07s for ; Sun, 29 Nov 2020 03:08:48 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=hAkQC7bx; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8G20DMzDrRM for ; Sun, 29 Nov 2020 03:02:06 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id l4so653455pgu.5 for ; Sat, 28 Nov 2020 08:02:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oqskVFo4rCcY4G7AX3q/Z2vOLhCcdMU13oOzjm6oOQI=; b=hAkQC7bxM/rt+UgL+n2DK2GVPrv7Fvv8VOFfmVyChH9mG2M8V1dhWuexs7FMt4k/VH 00X2IJnPJZxukN07v+6IUoaQEfrLRCAqYOOlZ5fq8l1BvW7IrGtlw1huOZSLzz2y0SZx 3ChCeIKf/3CDuf5o8vxiVXRbP+dl0i3USN+GGPFeVf1n0aup+2heHkiGuTtTAlOIHxup yVXQkUUka7ppthU1cQ04gq4WktB3u20vnaqT8BvQDxOruSqzz3f4f2+uoHUVHCc17E5Q QRBdGNwQLOrc1P3jbriPiXC3R/Y7g87L4RcqcZA6HPgDlB6tEXZDAiIKBbkorJcDGkrE lHsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oqskVFo4rCcY4G7AX3q/Z2vOLhCcdMU13oOzjm6oOQI=; b=jR3fMESJWrwkrvVr8bg+12kM9LLNzj2nLNL/qm10RjLouPncn+yTCIOCi2jIgoCZPt sl1UEbt096nd4m8EQGiF5tW6ZtPT/+n4tcXuhHRCxcaI303Cw0N1/QKNM1Z9nvq7IC+g gN84nMJwnFOYrCXMvK1ffTdy+HFP4FVsPfrNJhpuM4oCsrjBfgcatFL4poyTm4PqoX7r y7okG8wsa+gOhhkbSmYDN0pmIhvRQgVETCk9itBzRQoJWqNjPtb5h1s2k0jh4R0JesD/ FOd9BJV2H24v6RxEii1w8YoPGhoSwl7R+rxMkoX8y0UXm+WcX9sqK8cogcG8kRt7Osfx Uupg== X-Gm-Message-State: AOAM530D7JifrYz9NYpEi9xlhqR6zNfarCuwLdTe2N/U43TS9DJaBkSv UFVEEKXWX6aSK0+e6DWawsM= X-Google-Smtp-Source: ABdhPJzGaihamaR9rIxqQPEuAuhYCkZ5jY/oI8W/R+Ypcsv9gPzHWmICwyEssj9w/cCRebDK8ONgCg== X-Received: by 2002:a17:90a:f0d4:: with SMTP id fa20mr13620976pjb.129.1606579324386; Sat, 28 Nov 2020 08:02:04 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:04 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 3/8] x86: remove ARCH_HAS_SYNC_CORE_BEFORE_USERMODE Date: Sun, 29 Nov 2020 02:01:36 +1000 Message-Id: <20201128160141.1003903-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Switch remaining x86-specific users to asm/sync_core.h, remove the linux/sync_core.h header and ARCH_ option. Signed-off-by: Nicholas Piggin --- arch/x86/Kconfig | 1 - arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/cpu/mce/core.c | 2 +- drivers/misc/sgi-gru/grufault.c | 2 +- drivers/misc/sgi-gru/gruhandles.c | 2 +- drivers/misc/sgi-gru/grukservices.c | 2 +- include/linux/sync_core.h | 21 --------------------- init/Kconfig | 3 --- 8 files changed, 5 insertions(+), 30 deletions(-) delete mode 100644 include/linux/sync_core.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f6946b81f74a..160d3ad90507 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -80,7 +80,6 @@ config X86 select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX - select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE select ARCH_HAS_SYSCALL_WRAPPER select ARCH_HAS_UBSAN_SANITIZE_ALL select ARCH_HAS_DEBUG_WX diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 2400ad62f330..9a7ab08f4157 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 4102b866e7c0..282ea9942829 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -41,12 +41,12 @@ #include #include #include -#include #include #include #include #include +#include #include #include #include diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c index 723825524ea0..48fd5b101de1 100644 --- a/drivers/misc/sgi-gru/grufault.c +++ b/drivers/misc/sgi-gru/grufault.c @@ -20,8 +20,8 @@ #include #include #include -#include #include +#include #include "gru.h" #include "grutables.h" #include "grulib.h" diff --git a/drivers/misc/sgi-gru/gruhandles.c b/drivers/misc/sgi-gru/gruhandles.c index 1d75d5e540bc..c8cba1c1b00f 100644 --- a/drivers/misc/sgi-gru/gruhandles.c +++ b/drivers/misc/sgi-gru/gruhandles.c @@ -16,7 +16,7 @@ #define GRU_OPERATION_TIMEOUT (((cycles_t) local_cpu_data->itc_freq)*10) #define CLKS2NSEC(c) ((c) *1000000000 / local_cpu_data->itc_freq) #else -#include +#include #include #define GRU_OPERATION_TIMEOUT ((cycles_t) tsc_khz*10*1000) #define CLKS2NSEC(c) ((c) * 1000000 / tsc_khz) diff --git a/drivers/misc/sgi-gru/grukservices.c b/drivers/misc/sgi-gru/grukservices.c index 0ea923fe6371..860aea9deb45 100644 --- a/drivers/misc/sgi-gru/grukservices.c +++ b/drivers/misc/sgi-gru/grukservices.c @@ -16,11 +16,11 @@ #include #include #include -#include #include #include #include #include +#include #include "gru.h" #include "grulib.h" #include "grutables.h" diff --git a/include/linux/sync_core.h b/include/linux/sync_core.h deleted file mode 100644 index 013da4b8b327..000000000000 --- a/include/linux/sync_core.h +++ /dev/null @@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _LINUX_SYNC_CORE_H -#define _LINUX_SYNC_CORE_H - -#ifdef CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE -#include -#else -/* - * This is a dummy sync_core_before_usermode() implementation that can be used - * on all architectures which return to user-space through core serializing - * instructions. - * If your architecture returns to user-space through non-core-serializing - * instructions, you need to write your own functions. - */ -static inline void sync_core_before_usermode(void) -{ -} -#endif - -#endif /* _LINUX_SYNC_CORE_H */ - diff --git a/init/Kconfig b/init/Kconfig index 02d13ae27abb..82f9b5c937cb 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2334,9 +2334,6 @@ source "kernel/Kconfig.locks" config ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE bool -config ARCH_HAS_SYNC_CORE_BEFORE_USERMODE - bool - # It may be useful for an architecture to override the definitions of the # SYSCALL_DEFINE() and __SYSCALL_DEFINEx() macros in # and the COMPAT_ variants in , in particular to use a From patchwork Sat Nov 28 16:01:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407571 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxKn3wjxz9s0b for ; Sun, 29 Nov 2020 03:10:21 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=skUfCusn; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxKn1NRQzDrRQ for ; Sun, 29 Nov 2020 03:10:21 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::543; helo=mail-pg1-x543.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=skUfCusn; dkim-atps=neutral Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8M1WXQzDrQh for ; Sun, 29 Nov 2020 03:02:10 +1100 (AEDT) Received: by mail-pg1-x543.google.com with SMTP id l17so6744111pgk.1 for ; Sat, 28 Nov 2020 08:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=846xez6akqNpiAUuyJyg7Cgtsg97LYlLXrFa3gyFRtY=; b=skUfCusnapFL4QjTU84UBawv0Zb8dW/d/8QkYdctwdGHUzbPnVqI5dIbMXvJyVUugf fgXzb23Cz1uq763IJaPqMvWoeXOjJml4Bpo9mt9rCdL/6+cuAwsKwfGbh2JTKjN00pjh 5+n/jaWP5ggmGwOSrHnLZMtDcoPOHKbY6UXKvpgUovNRtB5ykoCejDrKXfvgh+sxy9HW MXEkoFva1uTEWLHLOYYXehr+6OI3Un+jP+UFbRtazCKWOMWRQzoluNnvm+jYuY3MdK0a QuV/D0FrPuV58JHb7DlkbeKAMloJwgFzRPJk+JdSGvWtQM5k2cc4yP+2/1I1nSubjBbN tL5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=846xez6akqNpiAUuyJyg7Cgtsg97LYlLXrFa3gyFRtY=; b=PHxYln1CzPCpCPtafPC2lYWgxW6ZBhdH+wMNtbckPtpQFzuFsZG9Beuju2KYtTHjLC p7fmeHj5AnkQrAQqQT+IYebISm6iTduKoOR7LTVP35dtjsypFTn/9wg3m2ETlZ5juRzh dWIqI8ubT/2PU57zPkWd4q1YEfIpXOnQI/UppLWui1ApWmUXuVCpirNWUdOxltKAdlYR xiZjS33xt5MPTBifxkR7AlbQTrF0cMcUe1S3zS4U7WebbxY6Cxqepb+AdOEYnVzZNIhg nwDODq3wEDmHDDJzLHrHJNrp6V3HPWSujRWVBXhV6uOB23v+WvLS3RNutG9zGQRYFtZd cXeA== X-Gm-Message-State: AOAM532FIjyDVhvf96P77UF9BZOdpBAQcryEsB3MmpJfZEk2zgr2g5FM IMbmo9mmM0wtdCD0Fql3KIQ= X-Google-Smtp-Source: ABdhPJzfS/HO/ZAtAEyDNg6acxA2ioQDK2tyBX/rWx/5uj/HtudHJ/+YK14jYXjHb2pJaVwlFAEt1g== X-Received: by 2002:a17:90b:4b11:: with SMTP id lx17mr17067698pjb.154.1606579328904; Sat, 28 Nov 2020 08:02:08 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:08 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 4/8] lazy tlb: introduce lazy mm refcount helper functions Date: Sun, 29 Nov 2020 02:01:37 +1000 Message-Id: <20201128160141.1003903-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add explicit _lazy_tlb annotated functions for lazy mm refcounting. This makes things a bit more explicit, and allows explicit refcounting to be removed if it is not used. Signed-off-by: Nicholas Piggin --- arch/arm/mach-rpc/ecard.c | 2 +- arch/powerpc/mm/book3s64/radix_tlb.c | 4 ++-- fs/exec.c | 2 +- include/linux/sched/mm.h | 11 +++++++++++ kernel/cpu.c | 2 +- kernel/exit.c | 2 +- kernel/kthread.c | 11 +++++++---- kernel/sched/core.c | 15 ++++++++------- 8 files changed, 32 insertions(+), 17 deletions(-) diff --git a/arch/arm/mach-rpc/ecard.c b/arch/arm/mach-rpc/ecard.c index 43eb1bfba466..a75938702c58 100644 --- a/arch/arm/mach-rpc/ecard.c +++ b/arch/arm/mach-rpc/ecard.c @@ -254,7 +254,7 @@ static int ecard_init_mm(void) current->active_mm = mm; activate_mm(active_mm, mm); exit_lazy_tlb(active_mm, current); - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); ecard_init_pgtables(mm); return 0; } diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index ac3fec03926a..e66606ef2a3d 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -658,11 +658,11 @@ static void do_exit_flush_lazy_tlb(void *arg) if (current->active_mm == mm) { WARN_ON_ONCE(current->mm != NULL); /* Is a kernel thread and is using mm as the lazy tlb */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; switch_mm_irqs_off(mm, &init_mm, current); exit_lazy_tlb(mm, current); - mmdrop(mm); + mmdrop_lazy_tlb(mm); } atomic_dec(&mm->context.active_cpus); diff --git a/fs/exec.c b/fs/exec.c index 4b4dea1bb7ba..0a1461bb62e2 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1031,7 +1031,7 @@ static int exec_mmap(struct mm_struct *mm) mm_update_next_owner(old_mm); mmput(old_mm); } else { - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); } return 0; } diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2c6bcdf76d99..7157c0f6fef8 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -48,6 +48,17 @@ static inline void mmdrop(struct mm_struct *mm) __mmdrop(mm); } +/* Helpers for lazy TLB mm refcounting */ +static inline void mmgrab_lazy_tlb(struct mm_struct *mm) +{ + mmgrab(mm); +} + +static inline void mmdrop_lazy_tlb(struct mm_struct *mm) +{ + mmdrop(mm); +} + /** * mmget() - Pin the address space associated with a &struct mm_struct. * @mm: The address space to pin. diff --git a/kernel/cpu.c b/kernel/cpu.c index 134688d79589..ff9fcbc4e76b 100644 --- a/kernel/cpu.c +++ b/kernel/cpu.c @@ -578,7 +578,7 @@ static int finish_cpu(unsigned int cpu) */ if (mm != &init_mm) idle->active_mm = &init_mm; - mmdrop(mm); + mmdrop_lazy_tlb(mm); return 0; } diff --git a/kernel/exit.c b/kernel/exit.c index 1f236ed375f8..3711a74fcf4a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -474,7 +474,7 @@ static void exit_mm(void) __set_current_state(TASK_RUNNING); mmap_read_lock(mm); } - mmgrab(mm); + mmgrab_lazy_tlb(mm); BUG_ON(mm != current->active_mm); /* more a memory barrier than a real lock */ task_lock(current); diff --git a/kernel/kthread.c b/kernel/kthread.c index e380302aac13..f1241e19327e 100644 --- a/kernel/kthread.c +++ b/kernel/kthread.c @@ -1240,14 +1240,14 @@ void kthread_use_mm(struct mm_struct *mm) WARN_ON_ONCE(!(tsk->flags & PF_KTHREAD)); WARN_ON_ONCE(tsk->mm); + mmgrab(mm); + task_lock(tsk); /* Hold off tlb flush IPIs while switching mm's */ local_irq_disable(); active_mm = tsk->active_mm; - if (active_mm != mm) { - mmgrab(mm); + if (active_mm != mm) tsk->active_mm = mm; - } tsk->mm = mm; switch_mm_irqs_off(active_mm, mm, tsk); exit_lazy_tlb(active_mm, tsk); @@ -1258,7 +1258,7 @@ void kthread_use_mm(struct mm_struct *mm) #endif if (active_mm != mm) - mmdrop(active_mm); + mmdrop_lazy_tlb(active_mm); to_kthread(tsk)->oldfs = force_uaccess_begin(); } @@ -1281,10 +1281,13 @@ void kthread_unuse_mm(struct mm_struct *mm) sync_mm_rss(mm); local_irq_disable(); tsk->mm = NULL; + mmgrab_lazy_tlb(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); local_irq_enable(); task_unlock(tsk); + + mmdrop(mm); } EXPORT_SYMBOL_GPL(kthread_unuse_mm); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e4e8cebd82e2..e372b613d514 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3628,10 +3628,11 @@ static struct rq *finish_task_switch(struct task_struct *prev) * schedule between user->kernel->user threads without passing though * switch_mm(). Membarrier requires a full barrier after storing to * rq->curr, before returning to userspace, for - * {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by mmdrop(). + * {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by + * mmdrop_lazy_tlb(). */ if (mm) - mmdrop(mm); + mmdrop_lazy_tlb(mm); if (unlikely(prev_state == TASK_DEAD)) { if (prev->sched_class->task_dead) @@ -3736,9 +3737,9 @@ context_switch(struct rq *rq, struct task_struct *prev, /* * kernel -> kernel lazy + transfer active - * user -> kernel lazy + mmgrab() active + * user -> kernel lazy + mmgrab_lazy_tlb() active * - * kernel -> user switch + mmdrop() active + * kernel -> user switch + mmdrop_lazy_tlb() active * user -> user switch */ if (!next->mm) { // to kernel @@ -3746,7 +3747,7 @@ context_switch(struct rq *rq, struct task_struct *prev, next->active_mm = prev->active_mm; if (prev->mm) // from user - mmgrab(prev->active_mm); + mmgrab_lazy_tlb(prev->active_mm); else prev->active_mm = NULL; } else { // to user @@ -3764,7 +3765,7 @@ context_switch(struct rq *rq, struct task_struct *prev, if (!prev->mm) { // from kernel exit_lazy_tlb(prev->active_mm, next); - /* will mmdrop() in finish_task_switch(). */ + /* will mmdrop_lazy_tlb() in finish_task_switch(). */ rq->prev_mm = prev->active_mm; prev->active_mm = NULL; } @@ -7206,7 +7207,7 @@ void __init sched_init(void) /* * The boot idle thread does lazy MMU switching as well: */ - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); enter_lazy_tlb(&init_mm, current); /* From patchwork Sat Nov 28 16:01:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407572 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxMm6Kbxz9s0b for ; Sun, 29 Nov 2020 03:12:04 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=VkLwTGPm; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxMm49chzF0Qj for ; Sun, 29 Nov 2020 03:12:04 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::544; helo=mail-pg1-x544.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=VkLwTGPm; dkim-atps=neutral Received: from mail-pg1-x544.google.com (mail-pg1-x544.google.com [IPv6:2607:f8b0:4864:20::544]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8R2jk2zDrR2 for ; Sun, 29 Nov 2020 03:02:15 +1100 (AEDT) Received: by mail-pg1-x544.google.com with SMTP id s63so6716224pgc.8 for ; Sat, 28 Nov 2020 08:02:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=74N67GuHsAKlUQCHO+8oJizzPlrfnBq33d6B/Ovug1c=; b=VkLwTGPmwmZvnPJHpLNIUbEQ3XRa/sOpBoQE2Sgr7xSLqciwk0q3eYNRAupvsW9v3M 0Lwkb6wKoJjwhSbv10vfipxeq0nYWJtscGMbipiKOasgKYbjQ1+8k2J3ZSDqVgpZw+5k a0bQ7fDQGWs1BdqUP9p3zbwz3Ygx6Vnnnu13rHLs+Q77qJ10HfJ8afd4voMDkqzvYBSH Whs7pDJNUlUjHy2MHiGrJrMJ+l+1Mpi7jTugmuKoprsToU4KTqKcb0S7wxDlNskhFhX8 kNnwDm1ODJbdMOAOQzpbbfnXq1HicbPfvzXdxI+O6MnJQTmxQ7fKgfP4rLg2TsEqStX4 pbUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=74N67GuHsAKlUQCHO+8oJizzPlrfnBq33d6B/Ovug1c=; b=GwGzs+KY9IDOYtmTcMijG4ll8R0F2naRYje1rQTgGJfcJ34tlI1P+Pynt7eVW1DvBW Uqfo/ruohmJGy2neF75e4VwefMkBKUJsygI5j4raNPGDoeN1rwzTVfScYxgBDrs9HnTK 6bSKTd/7zTpHXLDuMobaOW54HYnLI8khustuPUz+0fhkfzUqjcCZ1bPuVR1tT97pfz9+ WmWYEx3coVbDGRneav2cPXFaUhz3GMlL3NccLykpPe+Nk8XoqK3zVlx1vUdfZsgV+x+W nPoz81rsm1LBxwvibfFO7R1PH7MV6GbtS4M+XJZIJKgCsXWcCbTifFvtG7auEokpRwD7 dh/Q== X-Gm-Message-State: AOAM531ZTYA13Fn/8Likyyn1D2G4jrTun8y0Yg2UbLKcuxvazTANmz8G vpjrx/MccxOJAYZeyKkWr3g= X-Google-Smtp-Source: ABdhPJwxDwD+9zFmjMFp1FY1SUXYVEy5AQmGVT0CSX61N4g2WeQJxNZHGbtbroa2pwOuEZsmwpocJg== X-Received: by 2002:a17:90a:a50b:: with SMTP id a11mr17214934pjq.170.1606579333519; Sat, 28 Nov 2020 08:02:13 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:13 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 5/8] lazy tlb: allow lazy tlb mm switching to be configurable Date: Sun, 29 Nov 2020 02:01:38 +1000 Message-Id: <20201128160141.1003903-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" NOMMU systems could easily go without this and save a bit of code and the refcount atomics, because their mm switch is a no-op. I haven't flipped them over because haven't audited all arch code to convert over to using the _lazy_tlb refcounting. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 11 +++++++ include/linux/sched/mm.h | 13 ++++++-- kernel/sched/core.c | 68 +++++++++++++++++++++++++++++----------- kernel/sched/sched.h | 4 ++- 4 files changed, 75 insertions(+), 21 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 56b6ccc0e32d..596bf589d74b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -430,6 +430,17 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM irqs disabled over activate_mm. Architectures that do IPI based TLB shootdowns should enable this. +# Should make this depend on MMU, because there is little use for lazy mm switching +# with NOMMU. Must audit NOMMU architecture code for lazy mm refcounting first. +config MMU_LAZY_TLB + def_bool y + help + Enable "lazy TLB" mmu context switching for kernel threads. + +config MMU_LAZY_TLB_REFCOUNT + def_bool y + depends on MMU_LAZY_TLB + config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 7157c0f6fef8..bd0f27402d4b 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -51,12 +51,21 @@ static inline void mmdrop(struct mm_struct *mm) /* Helpers for lazy TLB mm refcounting */ static inline void mmgrab_lazy_tlb(struct mm_struct *mm) { - mmgrab(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) + mmgrab(mm); } static inline void mmdrop_lazy_tlb(struct mm_struct *mm) { - mmdrop(mm); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { + mmdrop(mm); + } else { + /* + * mmdrop_lazy_tlb must provide a full memory barrier, see the + * membarrier comment finish_task_switch. + */ + smp_mb(); + } } /** diff --git a/kernel/sched/core.c b/kernel/sched/core.c index e372b613d514..3b79c6cc3a37 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3579,7 +3579,7 @@ static struct rq *finish_task_switch(struct task_struct *prev) __releases(rq->lock) { struct rq *rq = this_rq(); - struct mm_struct *mm = rq->prev_mm; + struct mm_struct *mm = NULL; long prev_state; /* @@ -3598,7 +3598,10 @@ static struct rq *finish_task_switch(struct task_struct *prev) current->comm, current->pid, preempt_count())) preempt_count_set(FORK_PREEMPT_COUNT); - rq->prev_mm = NULL; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + mm = rq->prev_lazy_mm; + rq->prev_lazy_mm = NULL; +#endif /* * A task struct has one reference for the use as "current". @@ -3630,6 +3633,8 @@ static struct rq *finish_task_switch(struct task_struct *prev) * rq->curr, before returning to userspace, for * {PRIVATE,GLOBAL}_EXPEDITED. This is implicitly provided by * mmdrop_lazy_tlb(). + * + * This same issue applies to other places that mmdrop_lazy_tlb(). */ if (mm) mmdrop_lazy_tlb(mm); @@ -3719,22 +3724,10 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev) calculate_sigpending(); } -/* - * context_switch - switch to the new MM and the new thread's register state. - */ -static __always_inline struct rq * -context_switch(struct rq *rq, struct task_struct *prev, - struct task_struct *next, struct rq_flags *rf) +static __always_inline void +context_switch_mm(struct rq *rq, struct task_struct *prev, + struct task_struct *next) { - prepare_task_switch(rq, prev, next); - - /* - * For paravirt, this is coupled with an exit in switch_to to - * combine the page table reload and the switch backend into - * one hypercall. - */ - arch_start_context_switch(prev); - /* * kernel -> kernel lazy + transfer active * user -> kernel lazy + mmgrab_lazy_tlb() active @@ -3765,11 +3758,50 @@ context_switch(struct rq *rq, struct task_struct *prev, if (!prev->mm) { // from kernel exit_lazy_tlb(prev->active_mm, next); +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT /* will mmdrop_lazy_tlb() in finish_task_switch(). */ - rq->prev_mm = prev->active_mm; + rq->prev_lazy_mm = prev->active_mm; prev->active_mm = NULL; +#else + /* See membarrier comment in finish_task_switch(). */ + smp_mb(); +#endif } } +} + +static __always_inline void +context_switch_mm_nolazy(struct rq *rq, struct task_struct *prev, + struct task_struct *next) +{ + if (!next->mm) + next->active_mm = &init_mm; + membarrier_switch_mm(rq, prev->active_mm, next->active_mm); + switch_mm_irqs_off(prev->active_mm, next->active_mm, next); + if (!prev->mm) + prev->active_mm = NULL; +} + +/* + * context_switch - switch to the new MM and the new thread's register state. + */ +static __always_inline struct rq * +context_switch(struct rq *rq, struct task_struct *prev, + struct task_struct *next, struct rq_flags *rf) +{ + prepare_task_switch(rq, prev, next); + + /* + * For paravirt, this is coupled with an exit in switch_to to + * combine the page table reload and the switch backend into + * one hypercall. + */ + arch_start_context_switch(prev); + + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB)) + context_switch_mm(rq, prev, next); + else + context_switch_mm_nolazy(rq, prev, next); rq->clock_update_flags &= ~(RQCF_ACT_SKIP|RQCF_REQ_SKIP); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index df80bfcea92e..3b72aec5a2f2 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -950,7 +950,9 @@ struct rq { struct task_struct *idle; struct task_struct *stop; unsigned long next_balance; - struct mm_struct *prev_mm; +#ifdef CONFIG_MMU_LAZY_TLB_REFCOUNT + struct mm_struct *prev_lazy_mm; +#endif unsigned int clock_update_flags; u64 clock; From patchwork Sat Nov 28 16:01:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407573 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxQH4NScz9s0b for ; Sun, 29 Nov 2020 03:14:15 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=u/jtzknF; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxQH1ds0zF0WS for ; Sun, 29 Nov 2020 03:14:15 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::542; helo=mail-pg1-x542.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=u/jtzknF; dkim-atps=neutral Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8X703VzDrQg for ; Sun, 29 Nov 2020 03:02:20 +1100 (AEDT) Received: by mail-pg1-x542.google.com with SMTP id t3so6708998pgi.11 for ; Sat, 28 Nov 2020 08:02:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=65OkZBDz2HAL7Su3DQky7EvOLeinu5tb1gf0ETLWceY=; b=u/jtzknFAd1tAacYeJoy/46zj5ChdFT3xU8MZDR6BV1HUVh/2VSorfp/+xqvPHsPkn A5kWjznZ0r5vU8waxfe1mqVhtjhfYM+MaOz88K1WoNyQZyYjMWkiGLgwLA3xK3WSa6Gq FrlTNGIlqHWCl/be5jqC8aVx2VyfAfrUaer6LZ0ZQ/nNbXgJa/VHWb+e2mFD1wW/jxDa IfrRS5GIsGLweCb8HbxwZksrk0lJRMKAEcFqCQtSF2EBIi7qeHLU8MfKOOiNLtDuaQj9 FYq964XIIM+e/xk1kgqKRlfVu9uv/c3RzDRXUBD5Oi2Uu4QAgEoknC8D/6A6B7cDDFbc ZOtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=65OkZBDz2HAL7Su3DQky7EvOLeinu5tb1gf0ETLWceY=; b=GcP4a9LFDVp4PbiqdJ2sPd8inVB92O+boL+kbBaFVsGeM5t0WnKaWwijbkKb3FcpFC PJMS+b0NidhimwykiCcxHsYvFO2ZIblpc0PYzep4JH5Kqvpq1hPt3sizb36I1s1BgLtc 5fbYtushNUJeFV4tJH4AxRfcuBUnHvCJkj6YypMpuXU9cUg1gLTDAVCFhhITEGlLwbIa HhQpOvFOwWjqfuCSJtubA13iSWXRU4Lzi12XIdK3aQl2EayvupDaeMOZRJn3PxETtcEY E4ogkyQJQ8UDWyMHfKf2h3Gwd5CTwXHK30BM5VkH3DahRuuHq08UyNAXe3M/93LTGnO9 U01w== X-Gm-Message-State: AOAM530PhGMV+KNSVNXPTMpMGRzECixngLyckgmCobjXsbRe+B0CxRM3 BxFXCBcNjKExlOn6wOBUORs= X-Google-Smtp-Source: ABdhPJwsxob0WXTuSWwVF1+ay5qDAtrGlbPGBtoVsiaD1Lu41q7ZrRVVVOcfY+0p8Z91jyXmyn1W1w== X-Received: by 2002:a17:90b:8d8:: with SMTP id ds24mr6227490pjb.129.1606579338854; Sat, 28 Nov 2020 08:02:18 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:17 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 6/8] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Sun, 29 Nov 2020 02:01:39 +1000 Message-Id: <20201128160141.1003903-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 13 +++++++++++++ kernel/fork.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 596bf589d74b..540e43aeefa4 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -440,6 +440,19 @@ config MMU_LAZY_TLB config MMU_LAZY_TLB_REFCOUNT def_bool y depends on MMU_LAZY_TLB + depends on !MMU_LAZY_TLB_SHOOTDOWN + +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + help + Instead of refcounting the "lazy tlb" mm struct, which can cause + contention with multi-threaded apps on large multiprocessor systems, + this option causes __mmdrop to IPI all CPUs in the mm_cpumask and + switch to init_mm if they were using the to-be-freed mm as the lazy + tlb. To implement this, architectures must use _lazy_tlb variants of + mm refcounting, and mm_cpumask must include at least all possible + CPUs in which mm might be lazy. config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index 6d266388d380..e47312c2b48b 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -669,6 +669,54 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + exit_lazy_tlb(mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -678,7 +726,12 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); + mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); From patchwork Sat Nov 28 16:01:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407574 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxSX4Wxxz9sVC for ; Sun, 29 Nov 2020 03:16:12 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ssXbLd8Y; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxSX1PkKzDrQc for ; Sun, 29 Nov 2020 03:16:12 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::443; helo=mail-pf1-x443.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=ssXbLd8Y; dkim-atps=neutral Received: from mail-pf1-x443.google.com (mail-pf1-x443.google.com [IPv6:2607:f8b0:4864:20::443]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8d5h39zDrQh for ; Sun, 29 Nov 2020 03:02:25 +1100 (AEDT) Received: by mail-pf1-x443.google.com with SMTP id x24so7086208pfn.6 for ; Sat, 28 Nov 2020 08:02:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qp8PGJSLWU/dwswnH2/0kSDHsb+wln5XeqPaVW1B5Z8=; b=ssXbLd8YUR2ZIFy9Jd8hXHpF+Y+ZgoHlIRFX9zhsrsp+ZLU7bKlg7EuG2lSdedrjvR wE/qE7l+MWwYSzP1wi8Op9LZacYNj0TFM33blDINJeQw3k661TnuMoVq1P9kWsUpYm2L GU+Rh70ojG5i/bX6AWDs/ACxCKMtupmq6KLpdZta42wvOGpj65dxqcLBcKS1XebjeT2g HWp9e3wV/L0EXL2Ys40bmy1FkuapWC6KRItPGmNgSBbEo8WylcMXRQEawZThlVjdjrJU +e772jc+1LXeLO7pSn+jmxQ4ISH3MsydjpyvNtpmhbzBrVbHsQbAlOUzcGpBsv2LoBoI u0RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qp8PGJSLWU/dwswnH2/0kSDHsb+wln5XeqPaVW1B5Z8=; b=OZAwcLp27PeALU3M+0jDXRgCnfhK4eEiY3Bpx5ydy9T7SC/CmHxQVIPJr3uYp7+KoT 8tQyM0JSlGSJwOoDQQYYu/6lgJGtQ+eV5wqXP2Xx/3uTTcxXZCihAPzGUQZqGQG6lHiu hZCb8yYs0sT0h3eCKpkz/7hEuyvrZTRV+Mk2qBKOWlYYBUvQ+N1HyZ6+VG7G401ueZh4 MUyUMd0twj9AByyAVWY/liuPcCLAj/6/zl0r3KDVBp4F8qysRjNReaptIV362nL4rZ3K P9elkSoj5ankr7lPUPPNkKxHnOI2PTrSPX2B1f8zHjMEiO0uSCTKbkS7BkCdgrbZCQ7y Y3Hg== X-Gm-Message-State: AOAM532j+7I3ElkJq05sq7C5OOnAoRGEOOo8PEZEcJMXDucTbiq3sROU uhbagqykhhtpujPOuyxSPd3S+0pe48M= X-Google-Smtp-Source: ABdhPJw9UtKcgVJk5VujOdxJVrz6MfFXjvC+/8I9Mi89OeiWtMkzRMjtujCg4EGW63Zc1A5/CzokHQ== X-Received: by 2002:a63:4956:: with SMTP id y22mr11011753pgk.266.1606579343363; Sat, 28 Nov 2020 08:02:23 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:23 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 7/8] powerpc: use lazy mm refcount helper functions Date: Sun, 29 Nov 2020 02:01:40 +1000 Message-Id: <20201128160141.1003903-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Use _lazy_tlb functions for lazy mm refcounting in powerpc, to prepare to move to MMU_LAZY_TLB_SHOOTDOWN. Signed-off-by: Nicholas Piggin --- arch/powerpc/kernel/smp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 8c2857cbd960..93c0eaa6f4bf 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -1395,7 +1395,7 @@ void start_secondary(void *unused) { unsigned int cpu = raw_smp_processor_id(); - mmgrab(&init_mm); + mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; smp_store_cpu_info(cpu); From patchwork Sat Nov 28 16:01:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1407590 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CjxWK2my1z9sSn for ; Sun, 29 Nov 2020 03:18:37 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=O2SH7PLZ; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4CjxWK1vv6zDrj1 for ; Sun, 29 Nov 2020 03:18:37 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=O2SH7PLZ; dkim-atps=neutral Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Cjx8n0G36zDrR8 for ; Sun, 29 Nov 2020 03:02:32 +1100 (AEDT) Received: by mail-pl1-x62b.google.com with SMTP id k5so4119523plt.6 for ; Sat, 28 Nov 2020 08:02:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8ryYzZTfuMdMQoDGnZ6qQm92nrELT0Bznsq8pS5zNkE=; b=O2SH7PLZYn9x9yrSkPvaAZdWZ4Hk1Nz9o1vpICXBIyIIgIdy8d9QFKV+1r92+5zuOs gYKTram4WPDCSdl+hhVkmF9+vrkTQECccC9p+zkSVnyHnydt6RZsEWjgzXbamDwz8Ryf WwRcOCDkLf2yHoNmiO4MHVOErhGamyhGZEu/rYwWW8NOV9V/N1bEFCnSnAJQkVeT5lqs dhR5ybbQNLsLTLCFZze+EY6rm27mttggd2bAwJ7eXS3YKnz/bnqJ0GT0EmjvtORGnwS+ 17SXi5CsTI/P47c8OyIr30m3TcLjy6wRktbukjqqdEaVGzvGIE/erk1qflt4sxiTfF0m Fmsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8ryYzZTfuMdMQoDGnZ6qQm92nrELT0Bznsq8pS5zNkE=; b=bOhWiHPLYay2YDGRJjc6BrkSZYMmAtXfUMHzggeBHNiq/9tF2nbr8G2cuZEML7NkT/ 4j3skHko5sy/9rXpI+TIGJLbfmqGcudDHOgGZVvrOyUn19RSpZWqMbMfC+7dayOa9KrD YB+KQICJ4Y2dfW2+TpdxA5JID5t0g0w11nkRsVl1lKyfaw9UNJ5Vl88WnsKpSoN2HOLa P8XcH2AjAbO4YhyJVQBRc0zgPL5wbYOg38lylFa8ETJP1VhNRIc9veZDSQfMFahm7Klo +Jh+psnR+ZuiMIBHW16G7k3pYypGPhDYdf1qhwVJitBpdkYP72fSV7UdsdR0f8wo81vR T6AA== X-Gm-Message-State: AOAM532vRrSJGr2yaP4XmdE9hnUWNALF03fzIB2f0qCF1MQpJprXV7Z1 teXyEmvfd1q25Mls9FqID1k= X-Google-Smtp-Source: ABdhPJxOmOSo7yBdZGFr5Ee5U/YJuP5XMrKl+Nl3YUOkuajp7hYak8NpHsUyWSygIXkb6jTi/aUZeA== X-Received: by 2002:a17:902:ee53:b029:da:4c68:2795 with SMTP id 19-20020a170902ee53b02900da4c682795mr8244122plo.7.1606579347904; Sat, 28 Nov 2020 08:02:27 -0800 (PST) Received: from bobo.ibm.com (193-116-103-132.tpgi.com.au. [193.116.103.132]) by smtp.gmail.com with ESMTPSA id d4sm9762607pjz.28.2020.11.28.08.02.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 28 Nov 2020 08:02:27 -0800 (PST) From: Nicholas Piggin To: linux-kernel@vger.kernel.org Subject: [PATCH 8/8] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Date: Sun, 29 Nov 2020 02:01:41 +1000 Message-Id: <20201128160141.1003903-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20201128160141.1003903-1-npiggin@gmail.com> References: <20201128160141.1003903-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Peter Zijlstra , x86@kernel.org, Nicholas Piggin , linux-mm@kvack.org, Mathieu Desnoyers , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On a 16-socket 192-core POWER8 system, a context switching benchmark with as many software threads as CPUs (so each switch will go in and out of idle), upstream can achieve a rate of about 1 million context switches per second. After this patch it goes up to 118 million. Signed-off-by: Nicholas Piggin --- arch/powerpc/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e9f13fe08492..d4793c0229d2 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -231,6 +231,7 @@ config PPC select HAVE_PERF_USER_STACK_DUMP select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_PAGE_SIZE + select MMU_LAZY_TLB_SHOOTDOWN if PPC_BOOK3S_64 select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN select HAVE_SYSCALL_TRACEPOINTS