From patchwork Wed Sep 4 08:40:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980503 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NP20WRrC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-955-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGD84JQCz1yfv for ; Wed, 4 Sep 2024 18:40:56 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGD45Gf1z2yQ9; Wed, 4 Sep 2024 18:40:52 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::634" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439252; cv=none; b=GbesPAb8uq3a0w6kX6u+IpF/aA/Mg82ttotLyjBcO7M7bHN22hx1AdzScaca6B4GVkZx5GI8M2otfHgGHu+tjG/Bgd/nN4SZordm01Rk9B1qNdxaM/Apk87gDzuVD9QrbXIQyBkP6eiNJ0ttcIUW+mMCabpiucFgQ6ytB4sWoElw9zasED8Z07HMUXFKnub8LIU/FtvtnaPpQJrnTHsaOfzLjVZNkN7URAmXQPDlLpyjJRH4m2vX4h3kTHjeDZ1QCpxu5ObnTuLf6WnOGVzaaecAL7gwCwzGNLaTtEUHGrHyWoSnSNfZ3a3FR/zvBjap/aUxLgQQan2B9w4OQ6fZkA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439252; c=relaxed/relaxed; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=DEGR8vj5mZX/0/lwvVFxlvPYQvXSAenuTmeyYb0EjlW5sqYrrid/6aJtTYJ+kRsDEG15hi0ZsYbrhcPADo855uKQ8qioy/vUnSQ7WvukY0+NPWplFC2j6T7zQ9WX3HB77jx5vcjmnIQk3W2CH6n1ZLsweOipdacd9JghtFXgZCEzMqQdLVxSgIvOo6kzn8uasRjvADqKRritmR4c2Yznwt2l0j+swVsudbsCcw7Hdp+Cb1h3BzR5tCQZizjEiF12pY3I70UVhm9Q28g1PqN+4DQjGNJinrZgrV+PJaw9/NPAWq1CLiI6l815MPCvrk+fQ/gaGEM43+E1m6FIxxYFQQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NP20WRrC; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NP20WRrC; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGD42xZfz2yDx for ; Wed, 4 Sep 2024 18:40:52 +1000 (AEST) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-2053f6b8201so31851665ad.2 for ; Wed, 04 Sep 2024 01:40:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439250; x=1726044050; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=NP20WRrC3NKs1IcqQPIpGYPCqz2FsVzWN/tHag9oXmdqMssOtTm4pI6xqlZ5Dumqsf mvVFHNlk05QIm7KA8BqMX3q6IO520iXdXhRVSGwHcMfGkfiAmFz1YFysnWBrSHmiFDsd RUe1Vs4eGSqAemTpJg20rIFUuEG/RSZS/PH4I1GFskDQl/TcnuuTenImEtJ9I7ZqmZT1 XApUCQmTXYbOXURn6KGUc5eK8DQ5XOSbnl4JLUzXy99D2MJkwef3B0TV2LFym/PA5uCD hmQLfnOTy5GGk5mHxPIS2hrRbDJ5T/S3lUvxyJb3ZSmjSfPbe+mo0Fy5RIReZ6abyLBT zGJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439250; x=1726044050; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=LVnDx1xD0XILTmmUTC9m//JU49Udn0yCvvTJ//xkTebmjn7NE17XcAtCFP+VZmDyA6 Biew/gRJUDJoMTzTO4mEELMHbts3t2KBiuOD7XLgBVA6tHxJ1NJU99IMb2VUD+y/4nr8 UmUDeoZ68anbKrZsYuiFqzjHODzd/nSh+4zn+JuKaEWuJAFoYvuxjwpegJFFS3oHuUmt qSNqOPD4jiJKYV5XHwFlzL80y3V5NKuHg7LoU8UtyjtR3FUjyksEqMG0t6AHIM2UZn2Y MWN5BfId4qrSwhK+Fn9jamaCpqjXkaOISR0ZgO83UM4t9W9VmGtEjOJarqHq0281FZ8e ukcg== X-Forwarded-Encrypted: i=1; AJvYcCUIV2CnXeVYyR1RJ9nUJ8zescUXgvw4EkRCp4cEKa8fwOny0bgA9XP8nuY9o0xghrzgMB/Y9QDrkKae4Ig=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxD3k2BeGRhLb0feEftv7i4ghNUxba57+NzdSGNwqmAEY+KliRq dUExNhSMEx8NFGiGH0gmsLJXJ/Upg0MgOkrDOrdAd4AbjOQQz2bYdUJTyenPGtk= X-Google-Smtp-Source: AGHT+IHLu4af/0RdmwIJrtep3I3PIUPy/4FB7EG8zsCDexMsa/uyvNrQYTXwy/3KiLtDhw4LWBklXg== X-Received: by 2002:a17:902:ce0b:b0:202:3dcf:8c38 with SMTP id d9443c01a7336-20546b35d3fmr190359795ad.44.1725439249775; Wed, 04 Sep 2024 01:40:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Wed, 4 Sep 2024 16:40:09 +0800 Message-Id: <20240904084022.32728-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 +++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 50 ++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index a7c74a840249a..1fde9242231c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3006,6 +3006,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..262b7065a5a2e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp - as + * pte_offset_map_lock() does, but in this case without locking it. This helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct spinlock + * pointer for the page table that it returns. Even after grabbing the spinlock, + * we might be looking either at a page table that is still mapped or one that + * was unmapped and is about to get freed. But for R/O access this is sufficient. + * So it is only applicable for read-only cases where any modification operations + * to the page table are not allowed even if the corresponding spinlock is held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pdmval. + * It is applicable for may-write cases where any modification operations to the + * page table may happen after the corresponding spinlock is held afterwards. + * But the users should make sure the page table is stable like checking pte_same() + * or checking pmd_same() by using the output pmdval before performing the write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Wed Sep 4 08:40:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980504 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HoKg7pjX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-956-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDH0hDtz1yfv for ; Wed, 4 Sep 2024 18:41:03 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDC3yKlz2yRF; Wed, 4 Sep 2024 18:40:59 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62b" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439259; cv=none; b=XqIDZr4DAyNMTf8On9SkH/39HooIoeq3ZQsTcdGy6BDP0X+6zsGkctIw4o31s+stFiFuueSsWzYagWPEcC2uHVsVs/hQ0Mf1IUDaOeNH/WfBa6Fn72L/5EZeXOxzf7kNV7kchEMSCITgtYrbCo53Utf04s42xox9w+R/PJawW1xvBIfK4zU12RycGJ7fgdDBseZuSDarihjAJjtxGUrMWJ1pCEnYPFfJwFkhSwc5vDBciv7hSIilgO7ouLHH0za9HbQxYploQot+XzAbwkqhvAaceNv6n+hssKUAJCR5vmr8JAd4NFLZwsnAjGtTS5KaGaa/dvtQufGtU/sP0zSZUQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439259; c=relaxed/relaxed; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=fW0G4QuKSd5ygxzAocfK4i8exImKD2q3uLFOMIyVd+JFfWShoVJcGBrjvGcqGpSkmBpqXCWR4B7h9gJBO+r3Y/nMUh3mmyzY/iCBzba7VjTxBvuXcK5q/3F1Of2Ff16Eq1lKRnGHB/7yCZ2ER//7FOgnAMrzNkHkz/XhZj0BRsygaclpU6Q/blJ4lR3hHcWtItE77XKw9A8mW/94tLC3/mUSnrxpKz5X9zD4RVbCagL6KgliwB/NNc1tfBaghnhAjEkKXE1MQuvjKUND1ngKztMUSJT13Ah3vArt+GmmF6GryrtNlR6JptnDCg+H9aibkPgW9GqtMzsa5KNE7evZZQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HoKg7pjX; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HoKg7pjX; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDC1kK8z2yDx for ; Wed, 4 Sep 2024 18:40:59 +1000 (AEST) Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-201d5af11a4so54906595ad.3 for ; Wed, 04 Sep 2024 01:40:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439256; x=1726044056; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=HoKg7pjXlMHah6W/AxEsG6Yw8US832iwUYnUJOjSRSilvP14AUS6ECzeX3L/jMJ4km o9FqK643B8Pq5u0d7fe3hedM4cbhBsvfKszrjhCSr0nyyg9GIrPt7HXeLYd/+pCoAk50 AN+DtbafrKuBVhTXQ6FifiKBLVkyabc2pjdbK1qLEs9wnztgbthQeWzX3AeqAZtJ2v4b eSWmclUtqRN/98qIp8FYmhfEa9w+ofsJpkJeqI2iUELR3oxFcqKO9e46mdTpYEi+jBN4 Fk2e2vChBJaExbS+90wfmB2nj+SVfUOdYa/eX64N0qxrnjqHkxUa0dJSPjrG3Y0hh2sq nnyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439256; x=1726044056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=C5azjvPI0gNQ+4AB0gcLFixWm7Sq13HuKVm4HrEixGNJO7VelCK+QqapDQf8Vdm0iv kypt2ZeCI0P32S+XaPdlW+rHMkf6lKxxgyy80tb83/7ebo3qPOQ28e3W+Kwn44jXyx7p 6y5RcQ/u+q5FaiqdTjG6NJeOHFFgoIgY2AayE+uWsW5UXniUukneC0FForwfjHBszEnY lmqwQehFH6GcMwGDefkEOfQ/LlejTsiViQ9/Z3UJhQHEyvBMsg3xhZgeMj4vXnckOFdA vLhKZN5GUZmJiLLj/kN23mTm7kTOPzhEt2MFDsYLCLXnuGZ3t0BCnn2U2SG/p8xZccAm jefQ== X-Forwarded-Encrypted: i=1; AJvYcCUcdDo55J3y5xiLLuJWLrTuSqUU4OHlQM6yy7Mq0FmwG7ZF85awmokN+MAMl8i8mby5mEymLRsPL62nMWM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwlLjbkQoQ/++xBf/QKUHSCblREywLy0PpaPROfEixsuI1hlzAc uScE0AKRVvJPJb7enLP95t6zoqnN8qyF0u7KeBzjenCbsO1MSTqcYq7iagJbWB8= X-Google-Smtp-Source: AGHT+IEEdhBdCGQBBk6sF5V8bi+PhpxKTNotIZgwSj7kMgQxLZV/04loActQGvyTL5xDqZZl1Kbj6A== X-Received: by 2002:a17:902:f547:b0:206:b618:1d7f with SMTP id d9443c01a7336-206b61824c9mr14802495ad.37.1725439256381; Wed, 04 Sep 2024 01:40:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:10 +0800 Message-Id: <20240904084022.32728-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Wed Sep 4 08:40:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980506 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WqmwLYsU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-957-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDN6psXz1ygx for ; Wed, 4 Sep 2024 18:41:08 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDK6cHxz2yPj; Wed, 4 Sep 2024 18:41:05 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::536" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439265; cv=none; b=Xfvxte4t3fV9AJBWUy93CGQW5KSZdtUv5xY6SKokH7U3Pm9Z7fy9xWo8mcAZwq3YnZzGVy1Nm1wJq3VbUzCrtdWWT27OWbTOkucp48qbKm7U5ekwUY0Y19yslSTphh84DFr//e0mMRgrsFBgYIUnR7QPmbdEH+vsQb5IiJfL8s09DuHpIYg73/7oX3EpXmykvdUIhaaS+1nr6c50pyZYi6+gjJzuJH6QNx0ZUdVjUUZZRy2iNKvH2hSBCS8zVccKx9QceJYP8mKiK8UC9W8vLcmwWfLFrvAa0sEaEiU8r03Q6v5FY2p39m8vMzjkNo1jAXgW7MPA1yO1z8SQXf4VGg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439265; c=relaxed/relaxed; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=EZc6RAWojW/StqAJf95qRR+hGDqLnXN7qJfLp6Q0v4izXAavXydcmSvjdA0+tQHdtwsGAx9E99YUOgXPL8Xj61Ofi2yLLUQNtCd1Juof9Ad2fCtwPNZH6ACbYqeyIeAgs1kWpjNlZbHKfO2eMcAbGnwKAowzWpBvhXJutHL7Vt3UGeVjkOiYAb+vGtVk7Ho/X3EWjAv7iWKxnqDGwWHdSK6Idut9ggY9mK9yPFIt6e9uZw6QbPss72bSPPWBGK0M2WLrYusUCfflVALunPsNOcEQ95HJfStuY6RT+urvAMIWyimW/yAfvPsLOcQzIu988oWHU3wHRtTMN+YyPDh1BQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WqmwLYsU; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::536; helo=mail-pg1-x536.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WqmwLYsU; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::536; helo=mail-pg1-x536.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDK4RyRz2yDx for ; Wed, 4 Sep 2024 18:41:05 +1000 (AEST) Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-7cd830e0711so368553a12.0 for ; Wed, 04 Sep 2024 01:41:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439263; x=1726044063; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=WqmwLYsUh68MkhY0gS/zmLQ+qjY3gJ7MhzWMXq5fhlJAhYVm1H5enDjnex6mZlAHSi 1vn5DLTtGeKQi8/deStbaOf256Qm+9EeJzbZrHkbUUq78S3YIcv6jJa8Y+cQ8kjIhboN 0Op8yHj5npg/Ty5XGZ3KbAzrkBZd9bd3SevtRo+qzWvl9NA76yA7z9IyfU4MrVG89s0G u2h16/tZzUztfqGxvTBR4MCo+oNMh/raUNJeDrB3FebNH76KZdL2cmqNyQdCLLWgRwv3 pPeJEBRIOrqWR2X0/5wQHQrygr0SRV+anijLIvmWmQlJPgqfZY9ZDUQ4NcL1njLMdjTF +PdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439263; x=1726044063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=e72/tAC3ro+mdwW/ZwcScayFtGSbE6cLuj+iNFm22bUF0jfpZyB8NJh8R5mB0KGvso eOJvmjkloqYwMjL9g2Fft+q689A3xUHxxk4qeO6Xq7Pt/OslqV8URJkHtZBDEGs2snai eTODS7sjVeFeL2I0ekqMeALWp5M7V+miFEF7x+iJr0tN8Jai4FfVfU3UNw1NV2knJONU YXx2w+tMGu49SS2Ms/OUMHhUrxnghsOvVahQ29KBgsc007SuGIi81DFPHSBq+W2vF4KA se5HTmdbOr4YK8TRPnx5JMV+bxDSVk1V/3f8m+VRDLkueSbiMVeEGrmNOaT1n/PN4S9G fxXQ== X-Forwarded-Encrypted: i=1; AJvYcCUFmaKczYo174jAuH9awj5FxsE7Gxgc4nDweFKGPNNJNs0Eb3Llo7WJftueWQ+4p+Yap0zdsDarCA9VtMI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxF2CJb0Eil6h/S5PIXANMTd2dJXzBWqhyTX+6m7g+Lyk5GBNJj GdKm0BHEiZpy7sJ8tlsnnxAOfCsCYVmxxKqidA4W0fVK/1nNH47emB/j5uNt1eY= X-Google-Smtp-Source: AGHT+IFosxGKKraD23O7QyExDWz973MicB0gfFUEF3j/RrDnY2sLd35727GFsppJnEl62/xkyI7W0A== X-Received: by 2002:a17:902:c412:b0:204:e471:8f06 with SMTP id d9443c01a7336-206b83566dcmr20294065ad.17.1725439263160; Wed, 04 Sep 2024 01:41:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:11 +0800 Message-Id: <20240904084022.32728-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Wed Sep 4 08:40:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980507 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dpqv6db6; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-958-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDW1Y0vz1yfv for ; Wed, 4 Sep 2024 18:41:15 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDS1PjZz2yRD; Wed, 4 Sep 2024 18:41:12 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439272; cv=none; b=c8e6zL1NzoiRAsWgh2t3bc74xL4XGGdYu+GD1RoEjCndIFVA+WFxYlzFuI3RBRaIPVaix47ROsmau5lhGsdpAKZk6IHhDfHUVK1eO0rYZcLzgr2PNy+9de8yY2F2iVYgHoMGzROr/o8LwUdiRZ28lkqzbqHAEbaw/ptTYCKntZuCHjocNazNVdPmmDvdMgZDBknzIF7tNRa/Gv05mQbe7GkF5bL/NJ3pq4qvtJkEEhY7gMiRmkFEJpH+eISu69gPC0d/70z/YtCNrN7gRb/yFM0oiike7mNble8tTnifEZAiNOtpHHX3S64X+KZdFS7m91t6TtCjl8K/emuSG0a+1w== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439272; c=relaxed/relaxed; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=cBpH8A+E4C1UYyDGTFA6oaxhEt/vwcEtzcqKNonVqzumxbO32ZQsMzBN0zOUyxcVglnVudiQF5fKKr8RC19ThiUwl95WSaBmuJWUqoeomZAz+9r38zD0dMgnQf37JzItKijDqHGRYhWCS7tf7F3iTgwxiOR4wbTUh69ZPRH64xN1Bd2rO5zVao851dH6V8pqkxqTh7n/dgmA1oxiOxbK+rOm0qymJzVkdzzy75ah+G3yARyXl8nA95yhBuLDQLilPjfnV7aL6TXuUedtfER6Tj+nBJyYLKAlwDeHiI12oQp2EentWcrDxJOGum+MYjhpuAyoKqys+yBHdlG73x0ppg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dpqv6db6; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dpqv6db6; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDR73NQz2yDx for ; Wed, 4 Sep 2024 18:41:11 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-206b9455460so3336465ad.0 for ; Wed, 04 Sep 2024 01:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439269; x=1726044069; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=dpqv6db6gtl8aYHNYRRYNml2d9iqTUZP/eJMcY4FBDLBN+qGrTNStfYPnbdxRUdcTb 7qTTS5N5XXSnLhm9XzXyBQBBxoRSBVttwRhzq8g9Z1ruFCk5IwpYl5JQMcFBa2PD0Tcv jHI6AVXM72krHX98tLvMWOodwMWQDcPGgzF8T/ZGqCEopnHfydSPOgoDP6evyr2n6ioU D+P0k89+NA4iI36dH37a+VqP+uj14PLPYm2/d0Dp3/SCAE/HZluKXVnOF7KKvzSt8cEm ouM6Wg3oo5H+09zR0i3vet5t8LrdDQaolMQjnCyIfg9Hgaat6hXVpkYvknMCr0dHVUcV yqWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439269; x=1726044069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=lgKsWAENNfnDlIijRthnATA2S3Xz0ge14DP7E9/SLMIkwJfSHpw8n7o/nhEcq8bp6a fasHNCUA8EovPuFFj+6R1RTjcGA0d8/v3E1YoPIyPnbibP6f5RCEwMeNMBK0qsCOfmTl FAq1W8cLyAVWeNef7S3hYEww5Vo/u+pApgu/4THZV0hea/YmTKTUGVbmwIxXHsizJZQa 1DpGl3p6xFkxlSS7XhUti53hZPX86W5as057ESMrXJQDF0YXFz/yXzoWNbI4erR7bvkP I0VWJQthW++iXkqiqmSyjEUdqFMA3g2qvaoi7hRgWS2rnQQiSTX/OANyMWqTBei6MTxl uJCQ== X-Forwarded-Encrypted: i=1; AJvYcCVSNPndotYimf1p0RArgn/s1Tvgk8g9XcLNvYq5nbm6W+ORKew0pp5QF6/YX0rKUJIJb8nMBG37WCjEOF4=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyScVVrXAkEkI4fjrkHTN3SJV/TLmYVbI9bIQ4wSH1ITCJRkCp6 keKAbBV0AZ3xvhVPODzfcespeM/UmK6gBxW/ORxx5iUnz0Bmpt2z8/PRChCPIHI= X-Google-Smtp-Source: AGHT+IHVi0h+YpQWstt2F+C2N++FuREpmzHqlVlqHX+adh/zaJT2MbahvdDpXAnk4gZu2G3FJSfhgA== X-Received: by 2002:a17:902:f60b:b0:206:9dfb:3e9e with SMTP id d9443c01a7336-206b832146bmr25150715ad.10.1725439269514; Wed, 04 Sep 2024 01:41:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:12 +0800 Message-Id: <20240904084022.32728-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 20346df53df3b..216405ba497ea 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3251,8 +3251,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Wed Sep 4 08:40:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980508 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fkNQ8Bi1; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-959-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDd1H1yz1yfv for ; Wed, 4 Sep 2024 18:41:21 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDZ74K1z2yFP; Wed, 4 Sep 2024 18:41:18 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::530" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439278; cv=none; b=WBrzPhk6J6VBzuxdlukMFJwJ6LIYbqsVtMvu0Dsba5stdoVaxDk4Jx2YZMyx8flS1ej2zWlUwwROs/f1iUpfVFs3vQKUOl7Q+7Y7wlokUMe5YHr/ITWXkUV1datujjb4DVgllVg2W2lfuGAjHTEplaxUfIo650urUe62uzMIxhYqHLM3rGxlo24gU1DYqICYMjxVUkZl68fpfwX8ImeSs/TcJPl62xSRF3PWe5ojrleG9kXS6KmI/QFCzsc2/xEcthl5bzLr3bf7DkwNXDHM9W/wd4iG9j47w7fOY0p/QHoHfn+vF83N3D4MkhX9iXaCb/89tR3RL6GKk4hyZl1fWQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439278; c=relaxed/relaxed; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=kDQwTAb/crI2xm+MV8S4suKOTSfJ/vsRoKn1614l2oO3dkuxZG7yF4iyPbBF6HjIPFJ3uwdAmULTQP5cJSGoWXCMw2JMfHxWGa5mMTMTlegnmPdKApbGLJrLf1gbga5znzbv3M42KhmfTTFDywnZhS+zm+IR1AUtbqSn0T1tVGrYNOVP2qEAGURXT9DZI3mEAJvsvIVkP9zud0j/lQWCeYdoQ6mHVlMID5/vMgAlISAIeGwdYe83HYE/6MRSaPCuDSvD9IpTWSaLXMN83EHMwCta/Kk5TRErEqLmB26OUzmXOflI8qKXZ9pg74BlFJA2R3eIead3hKd7qZUXxyVohg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fkNQ8Bi1; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::530; helo=mail-pg1-x530.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fkNQ8Bi1; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::530; helo=mail-pg1-x530.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDZ5Wp8z2yDx for ; Wed, 4 Sep 2024 18:41:18 +1000 (AEST) Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-7cd8afc9ff3so369767a12.0 for ; Wed, 04 Sep 2024 01:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439276; x=1726044076; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=fkNQ8Bi1aLxMnQ4qN9uDDmKOaXe9CHd4j6CvpA2XCg4AR3hAEKjNOzZLbngyQwd5aX hir6IOduAcCxzUrPUDqJcws9IkKO4rw1GnPoWlKjvmE5uej+Y7hVh9oMEueYvqbflT2r qCgX00MG8BJb4EDTT/Ljp3a0gcXtN1IXEHC+xAkWrkoepWgAPJvphoApi8GgZu852rug sLLt1xg+Wn116M7HwBNNGrpOKvCP+7MvLCYzVcO05Ve3rmhSU0eGdZik06n/H0kKepvp zsneocIYnC/FrZg0WNWN+JETeZdJ9C4AtgqFyBN4mMwEJBtv52lidnbxnYmKp6TdaCgN AGsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439276; x=1726044076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=d9B3m8RYdeOmr5xFiOThPlCmOQteqlZIsL+f5593kS5CkJBqxDCzVF+m4m8jvBuOiE aCCf4t0g7clxMVs4uID6Qtju8ds3sFprfDhIeOPOGtP9zPUoKbbcOcA67kkWDyQVw7Dh 3wgJyNZsJVO7OUbneIkkeHZWylC/B5cdrForo/KZzayY4X41cv+XwSNO+WDK/YiLOGyj vdecm0mY490R67gdj/co+oHIbUpURYBAoeMbbLUCH/yprHIyEiXDiaSzvn05BzGQmODh 2WZ1PPY3vcHyOHRALkzqqroBTNbIFaCSBjWYpGvRMmaHhiKO8LQ8e+INmnAMSSwDMXWw C8RA== X-Forwarded-Encrypted: i=1; AJvYcCXiq9os3Sik0UGpQIGWQZ6l+S5ocBDVDCDH2xiTdEdr2fjreIsJ0HjacYgh/fJjkZ2Aexfx6OB5321K5Cs=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxJiRCtLT/cZ2Uyffa5KILtojstV5mYqllv6n4W8JswQC2saeP3 LXsehco8qsl8DNXYIHlywLMdRivHYbswKitRt8MDtYjZ94kzGE9SCpKMz6yFyd8= X-Google-Smtp-Source: AGHT+IEy+qEnEqpj3sReiKH3tPFUkwjqR7spuVkWXeQJodyHWsfCM0cenE/z+pCdCppofQTZA+ekKQ== X-Received: by 2002:a17:903:283:b0:205:76c1:3742 with SMTP id d9443c01a7336-206b7d00d3amr22901025ad.3.1725439276249; Wed, 04 Sep 2024 01:41:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:13 +0800 Message-Id: <20240904084022.32728-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Wed Sep 4 08:40:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980509 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=eeEr2294; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-960-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDl3bsvz1yfv for ; Wed, 4 Sep 2024 18:41:27 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDj0rYRz2yVP; Wed, 4 Sep 2024 18:41:25 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52a" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439285; cv=none; b=jCEj4p45MaFdTE5nOO3LPFAEVdU++CY6l1F1F62GMFHwvHcyklaoPDFL30G3Y9lS1pTn+Rw5s848TJHWUx99RCkfcUOkcalUJwKvjkDD6YBfLnnxkZ8DZ4c02czMSGEYJtxJ/RO89oZ63vOvnE8tG9PytsAjixSpuc+kn2IPIZkbxCmZpM1V8JC2dVld2tcUqple8SeMC43KWJ0h2MWW7Q8n0Low1kvQ3H8EYUpDLMNZpfPJpA5f/6DMTmxoWmULic0lEujaxSyNl5sqeptZLdU0cacEbEEf+4tuYJrOmWRNhDpfhSCmyWGbbgKNJYoTfVpHWXDoBmlhLAQfRhkIIw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439285; c=relaxed/relaxed; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=N9bdcZQ+A//Ue0OIn2nsynTk3lOFDM9nUddkV45Wa2N4Q9rm6IoErspu7umj8iM7++he/P3V7BppHont6lFHqAyoY4LI5jOfexOl8tIlZmSjAoPmybCE9SD9b3ZjLJgfYyNPtqudmxP5Vucetk06O+C3GCzAj66FVoPvcHGjr4E4rAooe3xuNbiXWPqnLvXt3Yj3kyYSSY1idsGwZ8pL2uoByjMh1+GVu5GJm6O6boCkcahlze80ga2U+rNU7TkODZ3Zhuc3VFVFJfdH2+c6Sb+uu0RvoQj1S0lDnnB3MkjZ1zz6ZzzAiYwJYu/Hrg5Pv/R1nXhGlRTIHhe+EswPUA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=eeEr2294; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::52a; helo=mail-pg1-x52a.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=eeEr2294; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52a; helo=mail-pg1-x52a.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDh5WV0z2yDx for ; Wed, 4 Sep 2024 18:41:24 +1000 (AEST) Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-7cd9cfe4748so4059125a12.2 for ; Wed, 04 Sep 2024 01:41:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439282; x=1726044082; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=eeEr2294HR2BIn1ULieA40gCnE0AOYZiOkyftN8W+H40IFLkFvVhoWQl/yQHXAjNUB BF4juIcr8D7njcODKxz/no65qb9EpQB55IsRb/D/HKFtBon/D20E/YqALWsPsza5P1L+ zTZrEWuZvYxAZxexXyhZ8r/8d8BusdXGqGGLSv+yrOPueGRNGkVbb3RnIQoDvlOrZwj3 UM88nUEMhUQqeK1Cl8p3TeDe4DtdzT/JxXnkNDDggmVAbwvqPhtXfqqvqkmdXtIMP08A 4eA07SV7a6aXe22w98viSulUTR8gGjH0Fff1YZAM6LqVqV/OGOoBErxKZmwfwWyC4GeV FnYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439282; x=1726044082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=TB0OcpbowZAfRtua1gLxWvlT5xR6hl4hq4ZwqUxk+IKmTNDXzNt7TaGUsvsppydCxt /y4O9V9S3Jo43G+cfUQIznwtXSzuU2fVdJF4k9hJbLenq15Wwz9s3hOEHqu81iZiM73f 7KNB3o2F+z/Cj3uskrH8f8Eg8kX2tiagUfcu0fbM9CHD8yzLeQG6RktgEn27W46ZW4M2 5hRmFeyWEKX6pDaw69QgGpAe/mWjbjTMeobZSg3ojFHtgoQGwJBpXEyoxWvrBMSEaWex A+/7Ejynij5a228FezKMNfrdQpbbgqu7VdOlu1FDY9annPWzVN18Zd8c1gjHKgwIq44v kjQQ== X-Forwarded-Encrypted: i=1; AJvYcCUUdfSO1N3Nk0LfVF2MNB0hIlB83neuGt9LhfYZTbRVNy1FabVEqSFA+5aZgWhYsJo0T5ocbQCbdL3XKHk=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzIesI1VYJOKPVO3bkAALLNNzR6mDB41K4qe9f9pAhSpt/GMf+l ML69F4nPbm0EEdfJQ87V7doWiycfL7h3o6uKvmj1HFNARIoPcaUY5FeQhYVKiHI= X-Google-Smtp-Source: AGHT+IGQLG6wPNgDOgfzFUGIly89eF+GLm9izjxQGbRYP1g8ZSKlFJK5pYdbKQOnPMpn9+eqCP/7EQ== X-Received: by 2002:a05:6a21:2d86:b0:1c8:b65b:3db8 with SMTP id adf61e73a8af0-1ced607378emr11775108637.9.1725439282360; Wed, 04 Sep 2024 01:41:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:14 +0800 Message-Id: <20240904084022.32728-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 42674c0748cba..06674f94b7a4e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5498,14 +5498,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Wed Sep 4 08:40:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980511 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WfQyXzy4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-961-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGDs4KS3z1yfv for ; Wed, 4 Sep 2024 18:41:33 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDq6ZCbz2yDx; Wed, 4 Sep 2024 18:41:31 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::431" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439291; cv=none; b=GHk5pf9FEVjnr2aL+Fp2iWjvje+fQs3+5CV48I3xguRg1uaN0Xd69xEwPzo+Md39ScCj31S5+6R5xw+3CxTJmSh53zgsNHUBbpvGu8dUt+9dFQn79VtcXhaVhx+xlyRAj5nKpOHa2gV9JQ0dNM8rFnQByLq2/J/S7vRpHJ3e7URIXeaw12dq0y3vxblcHBhhNiOguydBw2XS1orId4neccCssV6XuQRj5FhbdUi5HpFyJgboZcHp1aaZaJ9WneJC7qu1yIOYiySGOeKu2ib3pBVCf1vxA4qxCIKh7pUa9/JH8N9v2uwj7Y9QEpR9QUpBZpuFyBsqwcbMbWj4hSUhJw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439291; c=relaxed/relaxed; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=O6fEAgcM+34ghuK/+ZhsGTuyc7VNPMM4OcrjIAk8FldY1Gn/KRaNwP8/W3pGYnTw1a2ZVsFYfSQAK3/0ILfYPhL5gioTJQDmhQKNgV3JZgUGOcaKZouH+uGKfwHmeN0uDqhoKLmJBOZrxZTN0/3Etki8gq1CrMzqrcYWLrBLzK+uXeOHzwj9CCYidrxrka/NKKBs/z93hf1xGvNHHQNfW0Ipvm9bTuyu5bruVxfibHKtg1bu9aD+hyWlZ5VqzOXo5GsbAOmCkZ39VMZV2UQxDTyn/SwzdCbg9pCOmuylchUNAZwjLki4HiHPnQourqItzNbJtg7PNeYJ3i0U1wTOww== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WfQyXzy4; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=WfQyXzy4; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDq4KNRz2yTy for ; Wed, 4 Sep 2024 18:41:31 +1000 (AEST) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-71781f42f75so75787b3a.1 for ; Wed, 04 Sep 2024 01:41:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439289; x=1726044089; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=WfQyXzy4MYWVqRsDoBGvYNZVoisPcdUQrxi2o3n6nSNCyZE+BJRajNXHxuBXj6yH1h ThO0kR7TK6ISw1Fknwdbo/lTjSc+BXS6rtrBFfGCRRH735E+KaL7cXszrZLLhA5oP3mY WfZmGNfoG1cug2k5M+Cv1QOoTo37HD0ySAnSHcmYooKlDdiKLcmoykUTK1X9/xI8DlWu SukxfRlbzJLaGHgoPwvm/DNzWMo1kgFMGM2STKPYwLb7jGxCfIndje+Gyfivg9uzV4X2 2mIn5WxKdAZnAFwMrlE7yBohYdsIGHRUp48I37HHtj3J7jRxRsicWtWRyV/zVFTMXk0o ovnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439289; x=1726044089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=a2D/1JgKubAgsREohi4lJJ/Q0QEgrTXsOXN70vE4Eyrfxj4nRMewfaSwyZ48JdwiLF 5/5RhZRPoIfHN4Kqrd0T88Ex+OFH0hx6sS/xcNxgTJbeZzeYZrET5OHWC+sqTPDVXYrB cmTDWfqJfrnFYaXVewHEervixrJS0ZyZ6nifxLnagA8+lEYk76Uaf1dQsqjpuBK1BrWD UeXjMig4JTCUMt3cPET7/ggE8uY0QKcQy5h+Wm7PAJsPqf64nhsNjwPdqdwf8e4hS8X7 oRISfcccjI0mZLe/o6aNpsnw8v/7KRJjiUcs1D+XwZCjud79hQrxgh8xS0NbQmd20k1H BSOg== X-Forwarded-Encrypted: i=1; AJvYcCWLSFiXWRZDrXNa6i5pSSBpnWxn2cdSHDMf0xgjFXlPXpAfLn9Ax2dcJBW7yLCJNc4mheu3OvQhQy3DTzo=@lists.ozlabs.org X-Gm-Message-State: AOJu0Ywm2lHsC+9BB+ysQeC7kNWGHxkz6TvGgkw1r+ESmMgAWXgzev7J q0uyOfnSDEUjzX4PpO9PpCsxXyj+zo4+Ot6fV0Kt+dj/+IjzR/sMhyzfJq9vAxM= X-Google-Smtp-Source: AGHT+IHVhSodGEWLJbqfntGARXqFTBke8IrjtWuvc6P4tKsUnW/m6YB1esVTiG0+MVp8pwa1UVDERA== X-Received: by 2002:a05:6a21:330b:b0:1c6:edfb:431f with SMTP id adf61e73a8af0-1cecf788518mr14875554637.44.1725439288977; Wed, 04 Sep 2024 01:41:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:15 +0800 Message-Id: <20240904084022.32728-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..a117d35f33aee 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1658,6 +1661,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1689,6 +1702,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Wed Sep 4 08:40:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980512 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MK4jLMC+; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-962-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGF02g4Cz1yfv for ; Wed, 4 Sep 2024 18:41:40 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGDy6rtNz2yTy; Wed, 4 Sep 2024 18:41:38 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::535" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439298; cv=none; b=cSEZuavwM5A0Qh0TBjAHOslK+DkL0rzEY9MryGKfNTmoVTMYScK7Xsz7tVtC4J9LNfTK/yfd03HOADNNDAFq7FQjex/UzD0YLwV4WoTIdteFTiE7dZadglEvTnZKBecqetUbl6K9gNC7FJpOyd7D+tRO8oq2zrKWDjgcBz5WxXl+/q5VciFuW8Way4GYxjnUgIXV9nXguVhV+kjc8RsSw7RQK+hiP5/kN8z/J1nbXYt0dxtKc95D+sNY4Htx6WNhkshNL2JpLzRny6jekH3Fbx7jxt42qmbtlDYqiDIHiyRoPZsUsEdBZP7PRh7zm+Ji6k0L8fZuRwm1bH11bkbDdw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439298; c=relaxed/relaxed; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=g4tJlS/df0Go1FQ/IhWMgrGsUx97HK4wiM6biSgXHdXa2TMYjS8k2QA907nNa4BX6DDzR4TRezr80iBrT3FvCMqVmgAk8AWgAWTbuMGkuQbc7Yor4CMT8wZUpNoguaUQHqDTS2UF+mNhtmMN5bet3ABRGbNTucgWrAvPEKpC2VEiYSNAWeJXkUinYja80n35JVaRbUYI76yR//E+lsx20UTWGRLSgtrY80SFWiFtuWG8UuRfsIbDTso7PBinr+et5QYRbnvlBIOPPH/NfraWJbtUh+SJUPIYQZ1kGVB9TdXHNykMHY6OVF9qElwODkJuRV/7iDgVRtR1d+PpjV/cuQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MK4jLMC+; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MK4jLMC+; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGDy4S4Vz2yNR for ; Wed, 4 Sep 2024 18:41:38 +1000 (AEST) Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-656d8b346d2so3803160a12.2 for ; Wed, 04 Sep 2024 01:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439296; x=1726044096; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=MK4jLMC+W26E041CI75ev3GCmAxkCs/ttAJJABpnk72P7VmBVqF4xfrTdcNLSjPiwI XTBeGMiCg1pKi7HM7ks1uI/fJrxolNuf5tZmBL9bKelHXS0oe30rWvaEi4Sfko/IXPKy 9ARWAK9LP9JsLAgl0DbspY5ubvNUOhN5lWo1cnmYj+MgW1DmnrXUzC5Ak67j2HLPEvsz iyME6RDrXNsKp7nYPi9oADet2zsNLRx2DSJzTNSTZeYlPawgzso7n4vSqEpbpcEsOCT9 ZfW6jIDl9axl9HZzIMkbuYhKtBDAyUoNkSe/6q/YojrIUxjdgW9U+Ese97m6SOnv+Yum NvsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439296; x=1726044096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=gAsj5Dt7ztPWb01q1Stpm00C8eHZ5Apm/9K5hMdmmROTq6aTkTq9awOVsjye+Ah4+1 r9x4clAVOCuSbbLPlEdyTgH9jJ12n3gvk+YWAgKPHHCd07idTCx+A9Wk5Bp4OivaBxMf LDZtT6CpVG76+7Ntm5aPrKUwDJjwpEnGZK2tFrgwz2P06iQXiQqTWOs9BJkXrb2bd6J0 d5qUV441bFPT07mCcFwBCoKaw7lTijhXBwJ8mBX7NRO1RXZbe9jOaM76772g9zmV+yRC ZemZehbRK/DWayrMKAD0deIi5CKyTpl7Y715rkAbyypeb5PqlBn0hpvwoIiEVj4MyT/N xIqA== X-Forwarded-Encrypted: i=1; AJvYcCVxBM2W75cj8SIy4y93PAnI45jRdkvnHN2+CZT9WEyF2ony549Xrn6l5Cx8s2rRjeJPmsyrKTCkvfySMh4=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwI6BLH2d8k39vnpdCut0liXFRyoxXq2M0uNTuAzmIUD9zJ/4x+ 4QnCqbIpJgdyZ1/PeUY8V8LrPM5Wd8CoT3RI3r2/V0xJMQ6aq8tRxoyv2E+V1w0= X-Google-Smtp-Source: AGHT+IHnX4UPdOrG4UbMFan2fo99NWoUyUAqXexLL4zg80mEgnx1zTpF+DDeMcB5gD3OIXAOIo10hg== X-Received: by 2002:a05:6a20:b40a:b0:1ce:f77a:67d4 with SMTP id adf61e73a8af0-1cef77a69e6mr6594806637.6.1725439295802; Wed, 04 Sep 2024 01:41:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:35 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:16 +0800 Message-Id: <20240904084022.32728-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to get pmdval and do pmd_same() check after the ptl is held. Signed-off-by: Qi Zheng --- Hi Muchun, since the code has changed, I dropped your Reviewed-by tag here. mm/memory.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 06674f94b7a4e..47974cc4bd7f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1082,6 +1082,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1107,13 +1108,28 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + src_pte = pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ goto out; } spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(src_pmd)))) { + pte_unmap_unlock(src_pte, src_ptl); + pte_unmap_unlock(dst_pte, dst_ptl); + /* ret == 0 */ + goto out; + } + orig_src_pte = src_pte; orig_dst_pte = dst_pte; arch_enter_lazy_mmu_mode(); From patchwork Wed Sep 4 08:40:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980513 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LRXLZVbt; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-963-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGF64M51z1yfv for ; Wed, 4 Sep 2024 18:41:46 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGF52fT7z2yNR; Wed, 4 Sep 2024 18:41:45 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::630" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439305; cv=none; b=GpGlREDrcH1owIuzogIFBSA0Y5PJqU9eZrKacoTas6o5e0pp2fwJaTqzDUbYMFYjFwRzsOFo5WuudOrSRU3UXjWoCuDNZQcfVwSc22oK78MKvzrkbYPRMKgftAUbe/e3an5HLOb9RaLawP6xvavr6Wi+mJZTMMskVNah64fLxZe5DiF+isz0wWs96HPT9ft1+1UmnZbufTCiLUW8PUXvdLmKaNamfipZZbtZOvhSB/WNAG3DFiMmrHTs8UDurtddBhgid9/ZdK+AE/UCUn2Ejx6GuzUntyLDod9vBSUL4wHFq1mHDl51AGWrd2+9FqEiWGflGztVaX06uAlXe263Xw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439305; c=relaxed/relaxed; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=D2H3yZldd5vYKwP9VLOJ7AdfJnF5Ia0wnQB/ZzjYL81KGPEGgQny95T2aIeBxwkzvIUOp5gWu8kG3l7EDDWPW8FoSRgugNCKFvNOxgdPcszxD59EaHkC6aKQ9Lik5ix6F2KX+jymuT1tANRrBByrKvNgh9qSJjIhx+BLS3eNt4RLBgHyyCjH0zWV9kZhPg6VpkSB434J0l/isynviUZw+otSOLvFLKEUVlQnzdiDEVkfPwgFMNZb2YOkklwah0S5V+UFAY8wXXpKr62g4wEvFYvWXTtdL4zCsWjx/UTcgkFDVFfI00s7BxicFm5o/21m/qu6s+OhKZHb7XI4WTAX2Q== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LRXLZVbt; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LRXLZVbt; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGF514pNz2xbF for ; Wed, 4 Sep 2024 18:41:45 +1000 (AEST) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-206aee40676so6577715ad.0 for ; Wed, 04 Sep 2024 01:41:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439302; x=1726044102; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=LRXLZVbtsUo4cz9sdzFOnF+XuUY1svO9GTCX+oNxO3erWWyoK39O73p5IAjMfvMM6y HjifTKTpU4fiW5S+Y6YlduuVi4kvv1hi5OYCQPGxwlminSnTty/8BsdFRAdiZnFkbWuW yex769A48SF6XJiMfbm8XxIQqeczkSLs3Pkx6tgibR6rNZ82vPfYXb+vS0PZmoi+/rBF N0YwxGNqot4uZF6qbQVoiscm/fJOgLZnI/TzvQCv3vucTAJO8YS47HCtIHH2PvwVnonX pZrSXYOakuXE+dRLoN78R3qzVj/eBKEVJNduSZChYMtWvokqH/TsLB1TB2kr58Q7WAkH OyTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439302; x=1726044102; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=cbLwkK9Ipru59eV0xhPct8ZoPPQoefD+7sI9afeFLGfa/t3fXN8FOPoUqpTx1Sa6RW J6PIUecVnqQv8k4244CGfrifAvdRiZDAS7Td39l/8jrByTmuWv6BWZQNp9lnxHX3eImh l84CFvPg9Cm35U5ZMxEzjkVrC0/fUwyIq0tVscd+e/vTItF6wAeMeMoEkZxcNB9vHWN5 eTp3MnHyBDsKg/FRPTSE/Yrd8M5mGEzHgXv9XYRcvdNUg+BPOsmkRl6OJuaHkF4UKAXV fKPI7mbrhLbRi9TPxmWT4znbycSV8rSjeQnpMYRkv4P18L6Z+1rSnUi/Nf6wmwDkym1J Oz7A== X-Forwarded-Encrypted: i=1; AJvYcCWNj/9aEvYyOveR7Dq1x56ugRciejLxUMNl8uzuOZDi2b+ONBDbcpXmLhc/wJ8wVWS0tCRxyTrVcUJQV5o=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyZEXTlqV1jzn84U0ZlnxtcJp5nF3o5BPdrM1uhtb3e8HtzB11R 4S0LmrrsP2kUnITcH1UOk71/mtHZSHU86WLUYsDlrf3gyJBoXXLG4VICAHSFKiU= X-Google-Smtp-Source: AGHT+IGlDE+bZkJMe3LtnR3dMefm4RFnOsn/qpiRzdld3O/wCPCkO6r/LjjSl6iGWtDIKmNPthI/eg== X-Received: by 2002:a17:902:b68b:b0:202:4a24:ee with SMTP id d9443c01a7336-2050c45e710mr147539585ad.55.1725439302519; Wed, 04 Sep 2024 01:41:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:42 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:17 +0800 Message-Id: <20240904084022.32728-10-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to do a pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- mm/mremap.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..16e54151395ad 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + pmd_t pmdval; int err = 0; /* @@ -175,14 +176,29 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + new_pte = pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; goto out; } - if (new_ptl != old_ptl) + if (new_ptl != old_ptl) { spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(new_pmd)))) { + pte_unmap_unlock(new_pte, new_ptl); + pte_unmap_unlock(old_pte, old_ptl); + err = -EAGAIN; + goto out; + } + } + flush_tlb_batched_pending(vma->vm_mm); arch_enter_lazy_mmu_mode(); From patchwork Wed Sep 4 08:40:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980514 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MtAzR9x+; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-964-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGFD5LVQz1yfv for ; Wed, 4 Sep 2024 18:41:52 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGFD33M2z2yY9; Wed, 4 Sep 2024 18:41:52 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439312; cv=none; b=iE+ikXJtyd1pOIE3Dy+1+qjRE/6TZw10avY/79+/1ZZcN9gZmfDCM+vp4Y7J94LtBMyPzLkHcKKVSCEhejxWKXV24RpxagT7xSFyCd7j4jBicJVZL/LgufcwokVEF5lHEl0igIk7XArkdREejUBF9Lfyo2Korb2vsSwJMs/qaG23saIRLWmCmFk/MG7a7iMIZ+l4zcQHOsSp2ImtLEsM+oHBiYELswzUqkmo70Upff7GIJ6tcPPy7rUc0sTI4fTh6BrBJZ76p6uxDtJgMnyypRs8mpOTINz/WSblXIrY36xij5Q5+RaSikK+IDTYVp/xvNpluGoBtaTnd84dv3ZpGQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439312; c=relaxed/relaxed; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=XZ292SDKiqYnO7w9yNfJ+udfqYcSdzeeiog4mMqihsWQi4yREnij3+TF2sVhypS0Iy5oA/Hk6hkQDBkFZuYWLyCQwf+VNeJZXzQKuZDh+e4D6pZk51Rb+ndCZeJj3MlquyhETATN0Fao311THrW544/UiEX6H9FGThZF2EVx0ozXstP4MmpBEmWgfcH64a0qImxam2X3ChhiaW9ifgtPQrFCrUgTYAAgBfJSn2lqNi1sgDsIqdJJECuVNdkjFj3GUqniA879wxVkQYJXnqryJsWHzVjaZX0zrf1f/rANKemFtCpZyST4kHnWYC8kLal5rOyLqymky0vIhQG4qcx2Cg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MtAzR9x+; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MtAzR9x+; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGFD1SvTz2yY0 for ; Wed, 4 Sep 2024 18:41:52 +1000 (AEST) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-2055a3f80a4so23926005ad.2 for ; Wed, 04 Sep 2024 01:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439309; x=1726044109; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=MtAzR9x+HwKVqg/Ko6jFJP8+74xjvVzuZnBtiDD3z35EjqVSynRX1MxMlWn5LVa38p CbZRbmt9kldBFPWDtgiQEBQNpfd5i9xN1W2FJe5QExLzHaHi0/ONPwStA7NkYBubQ+w0 KGC3txJ2b6G/Unbyj8sEpivfp6GtBbSF6VesBUKccDtOohS2KayRkwHWEzKlotTxXVrS 1lV4WejguS7zzJvyfHHzjA2v6nFbLgbeCpKeaPpulxlhgVVy9h/FgzA+ffagoBg2R0fp 8ilD8QYNepavAZExVMnnshUZ9bL0SUTNtXUJOS2HrbalkG8lJpet0Jdm6qy8LzibaGYX Trew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439309; x=1726044109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=pjzWIPY9FYYwPqyMXnZy2Aa+pC1CetVxDeArZNOPqk42P0mhEyO3PoO2Ck2XOChBMh ljgwdk6DabTCwwCFXgJTsaTGq5opqrh/ksxteeWl02StEh7lin4wyz2FIMcNSSoyE/mu olFpScDU+oqH2GV7MqIOSOGZkNiFgE8eTA6GOejez7xax9f1dFv3JVdfaUWSiHYsHE2n oDn9SUlRkdyeRyM/zrX98+hiWJt8+JMBjoYy/5dWgGWKHNKhpsy/KxYYOX8M+5k/fUKw 53kpyo/5ShPhtrU8QdqIS/a5RgllVfjoGQSIQz/QvKzkR8F1fg9h+4uOrjChovzsAZnF tjkQ== X-Forwarded-Encrypted: i=1; AJvYcCXA6nnQ4t2eoRdt6MJ1sV+Hc31X2yWQZMx2X9Rcfb6Oilduwdik5GwDM44YYQt57A1osTxeD2xoBjknH9I=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwBXp8Ebqw6ZTXJweaJbAgRP+lX7nx/DNjFZ6Ta4WuUktukFaDA NDrM+tPsV0cljdLo+pt+cOPw54FPfME0pDOhOixa1GB6xtC5iM0CMI1yofh4zq4= X-Google-Smtp-Source: AGHT+IErn+gfx0jNqN8oYReDzi5ALFGdG/LuObb3FXvUpU+trUkKEzs/YdM2MbdzLHq+FDgFhXw6+A== X-Received: by 2002:a17:902:d50a:b0:205:6c25:fe12 with SMTP id d9443c01a7336-2056c25ffb5mr96453155ad.34.1725439309345; Wed, 04 Sep 2024 01:41:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:18 +0800 Message-Id: <20240904084022.32728-11-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Wed Sep 4 08:40:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980515 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ZDlgTysk; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-965-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGFM1qRfz1yfv for ; Wed, 4 Sep 2024 18:41:59 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGFL1X6Tz2yYf; Wed, 4 Sep 2024 18:41:58 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62b" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439318; cv=none; b=GMoKnfDH2gh/eDt5qaKpr0ZIZVhF706qrNuAQM1YjjMuW9EAfkROiF9ZL2/+alaKIAOptL0yvQqHI4omPGY1hZDkAsLMzr/6+3AaG0Kp/6BactEOIezVgziIKb60sx6lSMDobWMdJuEHR8pZZAEmULLeUO2pEoW3uNVBsXKBOR+m03+HsVJrdEnoqhQsDurSaK5uwpAqbvg4VWlks2duPvYhaBH6dbn2bcPvhSs+iFmk03q6wNlYxJC5D5E4aJCLUs03sfFWzWqyllfZdJeQaMM43NeksW5FxEC+/Dx9Drf7wMud7//ddXPr2C/qRyFe7PdEbwu2GijLroGmTVDREg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439318; c=relaxed/relaxed; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=JxG1lNybjQ3wmV2RctD4UyRMg6Eu+Q01cQfzh6L+gGlQtP/B/qwOIldPk9trB2cGz/Mg2+a9QeJ2KaoUq+0KQbIP5sNB/0ymhEEioS//CWo5SC0XvTTiMq0ZhOFmI+cUaSFGaKyyZJemiWkxAE5pwY5re5rPNnuD7Mr/H+U1bbrsxah83yNrvfAZz2Sik/VyR4oZzD9LKHKQ+lpiKFNk+4hEFtWI6WRTV6gmAXt3WmZoj2w30q66+mdWzKa0Yfzyyq5EKaqgQqzzDqsIiv0I/iOuTbeargiOQzRxvbAxMw2E0jn3d0EJBemeLYkjc5g27M97psbfyX32Sz+E9h/w2g== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ZDlgTysk; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ZDlgTysk; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGFK6Ntmz2yY0 for ; Wed, 4 Sep 2024 18:41:57 +1000 (AEST) Received: by mail-pl1-x62b.google.com with SMTP id d9443c01a7336-206aee4073cso6775595ad.1 for ; Wed, 04 Sep 2024 01:41:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439316; x=1726044116; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=ZDlgTyskaqazQepugMUvKIoHyhQ4V+cTPQSR8LuRqRMGvQPBbdYRl0sXgcgPdjkJAx oFkVeHYiVfoWPvKrMEZ4G1ZlRusEJn/qpXb5zUZx4dJoOAkKEB1fQQp3Eq/RSAOVMzMC RKrcw64TkkiYrnYaxT18Bu+D+kZfy6ltDoTKjWQnswMtDvlXwtNmM5r9XhzHpDIInElo HGlOTRuAejLwM7MZqk9o8LS/SufvEVa1Y6FXDK+vuzEY9hwtg+f5JbHNTUKXGQBNJRiB pgzMtxQUMUdX3BPkUo+HITe/cfBqVn5Azy3zjytwAAzfmG3iNcAQZt4oJH1C0iNinxDL zsMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439316; x=1726044116; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=R1oA5MrycWPZo91SwLhtoXddaWGeHFyksdcZG+wzIvNABzu9YIv8Ger5p4HPlJOB1B 2WcIWwFXyvobqS3nyl7u/ntwVV8SpQo9Il23I2R7GNSLaG2gn+Kh0hFWqcEXuDrQTO+E Bczbkw2Ljg9XDQ4ICfEPcQPJt82qNtrYeB3bn8zocSuRCGesp9y3UHwibALS48gBK/Y7 JarLZURgkfSTt2a/UfvvFl6gCFGxlIvPYrKbJZe3cgrWHv6uz7p+PTlXvWUh7Ex1Nx5H M6gFRUE3ly1K4ZJnXvrpVeJFOms+zdT1mGk7Slikln00xDRlztcrqS2fI9RF5z3Yiwhi omag== X-Forwarded-Encrypted: i=1; AJvYcCXso+I+pweMK82rwbo8zbccyj5xzizF4KYAZdNNM/dylCvgU81efpXv1Y8j6e4L5e/1YpPUgfHrNamVjl4=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz5ECTla6DX6AO90buIlIXIf4QF/dNuznI/1UaHsNHZTQEBd+BK +G2QLQuImM1u3x2/G9Zrlfy24/h3BCu7xA2QzOpzcYgBJl1fFOphQEMmtzpn8aY= X-Google-Smtp-Source: AGHT+IHUb0AH81Bf+t9sl3x8o3nWMR3fxDcIYnthyKOt2Ee/pQCOOrlcmsn4X1RsHo/HQ+Mvmo6mhg== X-Received: by 2002:a17:902:f94c:b0:205:4273:7d65 with SMTP id d9443c01a7336-20546131866mr119665875ad.21.1725439316150; Wed, 04 Sep 2024 01:41:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:55 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:19 +0800 Message-Id: <20240904084022.32728-12-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte = NULL; pte_t *dst_pte = NULL; - + pmd_t dummy_pmdval; struct folio *src_folio = NULL; struct anon_vma *src_anon_vma = NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte = pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Wed Sep 4 08:40:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980516 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=l2/DfPUc; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-966-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGFV157Kz1yfv for ; Wed, 4 Sep 2024 18:42:06 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGFT72qsz2yY0; Wed, 4 Sep 2024 18:42:05 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439325; cv=none; b=NCMIpzC63Z1vOK6xbLKMDgF72XsnNZyRotsILQR+QW7XpI5FiOj/A4rKRj7dv4GBOBDyye55sjC5r/AweZOhOyjC75+yPKn+TAJpUGwXpZWfSf70+VGO/K5yQaoh0aCq1IT0sE7CMCBUSneND9rDErMdpgxqkDxgEbWcDcg3uinrG5iAmkO72auGOaMQW8ejOO9BALk70pO9V02E8dLh+iboP60ueKH639IVCTKddES0+aYc0VqgXxQMW1NADRRSxAerQ9eBErEQAaVNYyeJatttXZhdPNYvpPDVvn0+jhX73johyMMNl/h5loV/Bv1yDB/4oM8HT/hxFBJVhMMj4w== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439325; c=relaxed/relaxed; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=M00LLIGdVNjxmR7zaApzUff1xw+w5srzBLhE2w/e4R1cXun3xRfLiGrZLYny93gsrGISt/w8/HLmzSbj0KwJrgCZb/3ZOJsn5CCi4KnaOOCe5M+fBqZ2Ql2QnDaJpfTMErX7Ysv2Kd2fyl+Ccj3JRHDEe2095jHLrp2hc8FAPUmNQOG7odY+fxTS9V0yTIzYaSB9ls35ly4BvuyAC9jUmgO4t+ImjOL+YFFNjmVoXxAgkjcQ1f7wYBVbmoOZRKWQgrh9PKcc088NqvLjVcJHGr9SeY7hCkTaxFJugQqlgDbzyvG8gmlwmHIIh0h+s2ZgrnEVRdsSBo5Jt5pnIYLn0w== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=l2/DfPUc; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::52c; helo=mail-pg1-x52c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=l2/DfPUc; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52c; helo=mail-pg1-x52c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGFT5YYdz2yQn for ; Wed, 4 Sep 2024 18:42:05 +1000 (AEST) Received: by mail-pg1-x52c.google.com with SMTP id 41be03b00d2f7-7cd830e0711so369082a12.0 for ; Wed, 04 Sep 2024 01:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439323; x=1726044123; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=l2/DfPUcXLT9RiWrxIbIdgc3+jydhrP5UIeriDpnBdHSJ7dYAMs7DcN7WL4gIMNloI zcjCBQ6yGyl4ipQSqAXif5dkGLjdmef+HsMiytlDmlKlJwfSnGiRtXd1Vd5UkAeKHuOU /1I+QuXAEWY0jwsnIteT8f2pLVUIY+kPAoUPN7lE15vBKr91tNoi8vLttlPqWF+VM9eI UPQn0HJxeuO5KXmLAdwfOTjrGXooPDe7WWB8CAsGuiDOxqDxYkH/YEom2liqBjwCyQMw Kw6NedL5bYqn9ZvDK6RbGX4Pr1NGcwwyZLT19Fcd63sfxUEIlWurKLQ/Ywyfn2QRwAOZ 3l0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439323; x=1726044123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=qWsRecfYJMF9S7lzCLofr0VDSY5fTE69qFvcYr0jfdngdh4jRh73D0FJ0hD33R08F2 3iWLzmBN4sU3eZTY3fKguXYsYDVwjRhl3217ZkVEAAECr0XKMHHJrq0VUW97XA+rNIwn yfr8F0hNiNsWFDcvCgJ7maiDqJYzEzFlxxMWbd2od2+/4n34T7wrLBTUkLwEz6Tv8J+R dLIZ3jUL/bmxvibETWSXKT4oBW2VUQ03PjsiEA+r+x2RUgpa6vCcyEXE8NEScnuGJa93 qoybCs1WeURGsLMPgOoP87H11y2W6n0FeqV6P+gP8mUEIXL3/DjE92fSTby1RKaD2RGr 3pnA== X-Forwarded-Encrypted: i=1; AJvYcCWIIU/4gsW7Y844WOuS45T3vfdpoSyLkfngDw6nssUPfIv93i7QCvejWQnVaDiody1AgzwjspG55Qzy1H4=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yyf60rotOGYN1o8DBNg3iAu5nlbJf353pmXhIy6gt4kRbzaYtoR r5rWs2y5MjVODKHRJivxkQad53kzdXCirbYobPvBrzrbyPJIP19JpTHTITY7974= X-Google-Smtp-Source: AGHT+IGXU/qy5PPe2L4ehUmGkYFpO+LVdbh/09mdfTo5RpVJKjaH3T6TEVRYBI0feUcEJkjAodGgeA== X-Received: by 2002:a17:902:e545:b0:1fd:6ca4:f987 with SMTP id d9443c01a7336-206b8341fdfmr24676655ad.15.1725439323059; Wed, 04 Sep 2024 01:42:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:20 +0800 Message-Id: <20240904084022.32728-13-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9b6a8196f958..36b84e46cd7b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Wed Sep 4 08:40:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980517 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ScginwWE; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-967-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGFc5M5vz1yfv for ; Wed, 4 Sep 2024 18:42:12 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGFc4HJ9z2yZS; Wed, 4 Sep 2024 18:42:12 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439332; cv=none; b=Y1iXrO8uy7cmGv2IlREW12vR3jWoE0RS8NWoPpFSItal47sgxi/xWyWwgtMfi4Ji051UvOn1Eh/AscxUOx60Y//g7m2eJC9NNW6oH//zU+yPzY5gf33R4KypySkS57QlJO/8xLfk3qVhd7d/tmZwSLYI9NLxBmKjCQvf8JuxgbJMrRrufA4dQC/YYqSwX+7rWIlM/QzzwuFFOWt+kuqgwLywL9Cy91Gwg4KcntyFbtWHKJltMn9exB9CjkQaaLjtkFxEK3ZKwyFAdXZq5ifVgR1F5KCCynyN6tl45zWugc7Yni4ZIZELubsgCVL2FD0es5/OHX5UEcaR/zLTvlhxCg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439332; c=relaxed/relaxed; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=HQs47CjHkBKeQFsW2UBuvYp8CiGVcXgUUNdLJDfsKQaRTXz3Lmk3qgrgpMTokMUdVKmZLEN4mpB9wvhRtcM5KU+praM+1LVQFNBb+i2hrtZYTGYlcrLnyPcAjcrEaeKXtGCKD5xcMYVkmYIMimxvT4ZB0CPNzySjpdA927XCvVJGuVGnPkJsQ/QA3dhSBjMj93tenX7OukYZWACVFnjVM8IIZJgTykplkLe7eNN3gUFdfhQiw+u7k3SmP+8+T2TnZJfRghtLzAos69+mxSvpHTuTWA2LCOJtNRCHuwZxyy+Zh5N82n8RJqlVA/DGI4NQhGfR/fzcLUjUva4PS+xQsQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ScginwWE; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ScginwWE; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGFc1z7fz2yR9 for ; Wed, 4 Sep 2024 18:42:12 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-20573eb852aso3457795ad.1 for ; Wed, 04 Sep 2024 01:42:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439330; x=1726044130; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=ScginwWEDXB2qe6tqKlRVJPFZGF4sMGAd3xBDsmZAslhW2gmMYfGMMRmMvU1IOS9IH dbSbmcw2aKZep4jXgPVyTA7SKzKy9GVlgNJpEG7YmUd+7gKovI+QKpg+H0q6qg0Do5hB jn1Ni0Gxhjk9CaKji/L3ctSSixmNQi4kJOxumhyor88W8R+nHojJLXNwMojNIUeRrAvV pciTKeOY6358Kb53Upp7SuixZZGxbQAHWJwiyV2yc9OgepY7ZnhbtsHg67fuqob4Qiif NZPcIld7NK6DvzZPJbFFFBt7VP3j+1PuBM7IwtVNADGHHMt0QaBSeenfLdFMfZsu/dVK qk1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439330; x=1726044130; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=iQINWn3iOmGxJ/1YypgRve0KoIwrjZrVlCbky/3lmEtMV5KRTkhh5muNs08JyT6b09 8rEkCs80dHZWNZNBsUdIalJY8H0eInP3xYBZKauwztgUG/Xy6h2g4eUrwHLOtPnCgOLm jrtEA07BxowQnZBFmpRMTULToNqRRo2JFEoG//qfhViVR77mpbZaHlBv6dOG+EulE0Y3 sT49rRCifoxt+Bnwqy0M/23X4fGJqVdxpfXbU9M/9KW77WPn5+nRfo1KvTy2Bea0AatR ulb/XMM7tkTp8oJUql8gjSzoX9gAiG8f97bLV6W44U6vAPfFyFSU2BZSTYHZeRg85seC mZYw== X-Forwarded-Encrypted: i=1; AJvYcCWbSp5mDoBk5w6r82zehrHaTN1ilnN4OwLqeIrk+pch+cne9dwXuRJ9S1iYRyaJMLq6WWlsrSBbjzcHV2E=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yw6rYDMIu6X3NVl5Yfw7l1CG441AtniqkMuxqvHu5RIEMg4a/Cu SCNAkYfzsi8+YKTRbPu2nGI3sao+AYV4e/mxG7Kt7iGYlkoBApEyisI8Cz393Ag= X-Google-Smtp-Source: AGHT+IHV13v46mfi8eJv03VSK3v4h4kaXbg2iDfLcHBwpuweJiwY6TIZiDRC9qw17xCxMuqJ20qctg== X-Received: by 2002:a17:902:ecc9:b0:1fb:7654:4a40 with SMTP id d9443c01a7336-206b833e006mr24427095ad.14.1725439329649; Wed, 04 Sep 2024 01:42:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 4 Sep 2024 16:40:21 +0800 Message-Id: <20240904084022.32728-14-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fde9242231c9..5b5a774902bd6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3004,8 +3004,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 262b7065a5a2e..c68aa655b7872 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp - as * pte_offset_map_lock() does, but in this case without locking it. This helps From patchwork Wed Sep 4 08:40:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1980518 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=F5P1cFfj; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-968-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WzGFl308Bz1yfv for ; Wed, 4 Sep 2024 18:42:19 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WzGFl0VDcz2yVt; Wed, 4 Sep 2024 18:42:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439339; cv=none; b=Pll/7iQkr9wVOK3QEtz/Pz9cm8a3aNvHnfaDMn/f1pIBIOLEaKoUtMhTH+X33j/HYirWP1F/zPBQNhJyaui1PycrJB8vSJtk6KlxN4WAD+DEtopAxxvgjU/HsCYSEnxf8tsHoie+cnwlhsrBOIuXxOOIjnV+M/DNGKSoeNMFWI52BxxfeWHOs5SPQM7597WZDhD4RwDsxlTP3ED2PGFjWrha3GKAszcgPe+HilOcya8qFweb4kwiTKI2Dn3vtZadv+gx1YkBxE14RkHqM77bKWwpOkqG6eLJ/8BEIKPAyoCD4KGn/QIC8vOcAKAICGtMqqFqSK/JuBYUA7uyvAITPg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1725439339; c=relaxed/relaxed; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; h=DKIM-Signature:From:To:Cc:Subject:Date:Message-Id:In-Reply-To: References:MIME-Version; b=Mfx5HGohaT+k7wu4w0iZTc430fIRhQOgCQVidMYfgEqpr+cYPOIRCs4v5rIdUhEOyn4BaOm+fV5tZcHusrgowM5i5g5dHXzU3BjqB/AqReMK6qFZgfYTJBwTU+lghumHThoJD6x2uUW/A2ZS+f8b7dO73Ierv5/WWpAdw3vgABvDUNKbKlCAHb93bDQmJft8a4t3v5iMv7/2+Ch+HED/XKoKgmfh6TXXYdT7p0ZCecdooeVLwt3Vi61mFUtBAJgi0DB2Khlg0vnJTWcbeGg8A7tyDeUAPtisRhsqn9NloZrIdGNW47cLJmEKFjkC/f4kGA9YUr2/JkLDhNl3xmO4BQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=F5P1cFfj; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=F5P1cFfj; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WzGFk5MF6z2yR9 for ; Wed, 4 Sep 2024 18:42:18 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-20543fdb7acso28484515ad.1 for ; Wed, 04 Sep 2024 01:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439336; x=1726044136; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=F5P1cFfjcB/qPN5d8YLnueLr6Sd+dqPDCCHsmYxXxHmJsmL8qYCW+scdJfJ8AKGZ4Z /b7Abkr+qGE4gasaDDteqFIPWM7i6IhP/4d5AESBQAwsXb8sdnml9EjNFqQLsBZShE8/ 1VFboBus/mkdWEC/bJphFzV5q6aeYx4n1K7491AXSS6JeDzrIlRSbgwNLVnokmVvCxMj zXGc6uhwJ74iPhh6l4I5DgvrfAiu9o5qSOXocpgaRuQfFIMayQaZVtm6Ob4uMU5pl0of L9KSOXSkHwYzbuaM4kPpYi28cHa2ltTKYCcIrG/t9G5ByzMqccs2LyV4wUp2SmfZe6cy 9a1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439336; x=1726044136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=Upd2sXPHlJGHXRZp50M98UmLvFJZgLaNw56ro7LNjsgFR8nZscrL08qvEuGR1aTHdy eIJtgWpdQcQshjRb/me8+c0MQu9mULI4L+1df5D0NM2/uPagEykOZhp2uwNZIKvVwBuF 3P2Cgs7rLx3aU6T+ZrjQ7W2EvO/lp6xupzrfteLkvc75pX64da3Ltns3QwZJb+5T59Ze EPPoTN4Igjuw5SYaqQ+h6MODraW1AIKD1+gpjZ69AnQT5lzQdKynVZwVVUvGVfxQm+ko sl/7g1bS0A9vnTdXUS+Hf5oQFGrxlWVa9x7tHCbxeoLJNxaHluzSov3ErWvaXsiax7aF RooA== X-Forwarded-Encrypted: i=1; AJvYcCVn7pYp6QjUEd9sT0Zow1HoCThyo5j6lCsYM0EukCXVcrJJmaxc276LkMxKi9rwpankv0ug6s9cMWWAJxo=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx6YpVMH9JDgse4MHxYpc8fMeju2I5CvFTI/y3Ky1HCPy8QcRw7 6Vbx3DKztjhV4ZLx4ePNneypDTgDgemQN1MAJxhuyZoVRewCIhTMm1ZZ7BofeGk= X-Google-Smtp-Source: AGHT+IGRRbNTDVHBE5FEDByUohEYiQ4MupDiF5X1fYOKyMaX8J1f74Xy7lHxJ4SbBUityp3jftUwsg== X-Received: by 2002:a17:902:e848:b0:206:c43f:7896 with SMTP id d9443c01a7336-206c43f8dd5mr2410565ad.21.1725439336115; Wed, 04 Sep 2024 01:42:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:22 +0800 Message-Id: <20240904084022.32728-15-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a117d35f33aee..318cc3eefb040 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1724,6 +1724,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1759,11 +1760,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma