From patchwork Wed Aug 21 08:18:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974815 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Z52+5f4G; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-264-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjY00Bl8z1ydn for ; Wed, 21 Aug 2024 20:40:52 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjXz647yz2yGY; Wed, 21 Aug 2024 20:40:51 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::434" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Z52+5f4G; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::434; helo=mail-pf1-x434.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfPz5y9Pz2y1l for ; Wed, 21 Aug 2024 18:19:35 +1000 (AEST) Received: by mail-pf1-x434.google.com with SMTP id d2e1a72fcca58-714262f1bb4so278193b3a.3 for ; Wed, 21 Aug 2024 01:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228373; x=1724833173; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=Z52+5f4GzfOVWjwpzpAwSygnOUPREjFKMBTRacTbh3/Nebmsmc2Vt3QXpBU0xSXhbm dA38ckmPIcg8PcraKBO3QeHX/k44K+NuHUQzpCwkmUEbY5tvrG2XUZlguFTM3DBxpE0K k6rSXhukf33eqimMKRgxNwnTQROOdlquzWrgpPvdjVv5UZ53NJIjZshxBW9oq34Ea6MP YkHDmJxB4dHmCAvdaajdn0ua/1ZilvUrLDd3RtIcYRlthdPqV0HxkygvzhKnSnT0CbZz 2FTIkTBVpXeSbJchctkPRIIbahMnwyCdzPlTMNoB6m/DXvvF2L0Txu0VHUnCwT5236gT /2+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228373; x=1724833173; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=bAA90eCjgOCnlzk6JIbejc7SQjl4MgC4BUMOP6iKZkDVaAfdlhF5PtsYY+yh8/uVUd AyBIYS8GKUo+cQ2bxzs7x+Jt0wUxciS4LnKeYZdlwyeqEAUoMnZt6BY8VltVT8Pp9wNr maqOIchxyY7EdmJu0ogB+ksR0eUf19Z7nWI9Go7B5mxMPFwrVVph/B54L+ETS1z2Rh4s 6rZb56ZuLq3Yk3PPuvIKYZgShD+wq6Y+bfNJAufi3cY9uGvtMZws0zfn9g4eWdTYO+/+ lSMKJkjp7VhoaxCZuOPyDnr0hlvjRo6fS9B4RX8trYExRsdqp4QOR4fhTscNomGO1uVt nEYQ== X-Forwarded-Encrypted: i=1; AJvYcCU/zlx37qdwVpNGNifjODLvoQANXROuBUgRoWEi52A3yhDiIjPSHrq3kAlHTcjhC+PXxpuj+Bhi5Bo89hM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxfwpNNCXZyTgWVc5FWSm8AvNYJT+yDms/wauLeqmpp7EmaXnd2 TGB21pBoyTkdSSBvGuqhwrFLrzeaKBxPp+j4PW0FpyMoq2GdjEGPjZe0WtLIrlc= X-Google-Smtp-Source: AGHT+IF11KqU9EwWDBxqOuV+x8Or0DtjlDyZMn8HSaNUPAUvyVetfGjOkbkYFHx43yYUuJqRZJ73CA== X-Received: by 2002:a05:6a21:2986:b0:1c0:e49a:6900 with SMTP id adf61e73a8af0-1cad80ef085mr1849540637.7.1724228372801; Wed, 21 Aug 2024 01:19:32 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:32 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 01/14] mm: pgtable: introduce pte_offset_map_{readonly|maywrite}_nolock() Date: Wed, 21 Aug 2024 16:18:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_readonly_nolock(). For 2), in addition to changing the name to pte_offset_map_maywrite_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases we can pass NULL to pmdvalp: either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index e4f6972eb6c04..f54f717ae8bdf 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_readonly_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_maywrite_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index 00501f85f45f0..1fe0ceabcaf39 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..29d1fd6fd2963 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, return pte; } +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + if (pmdvalp) + *pmdvalp = pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementation * __pte_offset_map_lock() below, is usually called with the pmd pointer for @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs that - * either the mmap_lock for write, or pte_same() check on contents, is enough. * + * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pmd), + * which might by that time act on a changed *pmd: pte_offset_map_readonly_nolock() + * provides the correct spinlock pointer for the page table that it returns. + * For readonly case, the caller does not need to recheck *pmd after the lock is + * taken, because the RCU lock will ensure that the PTE page will not be freed. + * + * pte_offset_map_maywrite_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_readonly_nolock(); but when successful, it also outputs the + * pdmval. For cases where pte or pmd entries may be modified, that is, maywrite + * case, this can help the caller recheck *pmd once the lock is taken. In some + * cases we can pass NULL to pmdvalp: either the mmap_lock for write, or + * pte_same() check on contents, is also enough. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a page * table, and may not use RCU at all: "outsiders" like khugepaged should avoid From patchwork Wed Aug 21 08:18:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974816 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UhhCcR5z; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-265-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjYB2FX7z1ydn for ; Wed, 21 Aug 2024 20:41:02 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjYB1Lpbz2yGs; Wed, 21 Aug 2024 20:41:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52b" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UhhCcR5z; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52b; helo=mail-pg1-x52b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQ60XFbz2y1l for ; Wed, 21 Aug 2024 18:19:41 +1000 (AEST) Received: by mail-pg1-x52b.google.com with SMTP id 41be03b00d2f7-7aa7703cf08so4626432a12.2 for ; Wed, 21 Aug 2024 01:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228379; x=1724833179; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=UhhCcR5zLDcnq7OoDwum4hCVWHsCHXStZoG9pda+087OZFZLnu5e+V3VNrBU0gVh8U Wc8nS6p925hpnt3f5wFohQB80kyHk5XFHZN9nNTKU/9QLHWghqDKfITY2iUulo34p0L0 OjOTKZHS++01KjzQdZRwSYYtLyha2w2Bf9Ha+If/EVmoJgohkWtnbtRWlNi12+DY7jg9 14SXfk0Q6ED7clFg7vp9CTrMi4wgvDfn5S/wAR5tvhwxzr4RS1GB0qPX+12JwoPvbaQz JhNXx0A4b9NfDeI883UviI6S5HSAx/JfYVegIhSLkhbdxJ3Ddb0PDusgV+qfZmfv/swe R0Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228379; x=1724833179; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=dRpAQughOzUnn3jfbdra7tOYDf7b7fLvyfe9htPxvOr84y3ACtjwmQXmV+OayPSSqK P320qe6yhl5DJIEXhTjGlLcvwUu1xCRW7QTnfFiyhXwFcYeJGMdJGUTbatAEI5xtl5ns Hu6CBtpbnXAF8SM7pp5JUUBzSHy8zsETLrK6Po1bfoFR+R5ZVs8v43RdGFb9CklJjgu7 7/iJa4NPCTwqesfbqMWP4bGI/eECqoW8uM7sAI4yfMrjzPFLbeIsT3mF10SWlZYs8QeN sSNm6C8gKm6Mu/9S5zZIDynV0GoWB5LG90smfvnzjHivZZRwonMCqOorTawSWXNlzAnB U3fg== X-Forwarded-Encrypted: i=1; AJvYcCXmWNrnNYz6kILJ5YbtXwKbD4ge7MLY6tqo/DDTIyf7Jb3wLLu0YCLiJHxTP+EsclSkxUjEW/He2Bs28R8=@lists.ozlabs.org X-Gm-Message-State: AOJu0Ywngme3RwmkddanxJjtsggnuOqT5v5Xf7DGUdaZpDuxVsWV+BGU MKf6pBXd++385KmyqCOLLsFZ1H+RzTAffMS+VhkQOu3pDqnuF3UMPcqkOzmYN1g= X-Google-Smtp-Source: AGHT+IEvHWRxH/KV5glvGK0Xiv2dJ13pnLs9scNXFJwmOSFepkjBPO6rW4EjBZpYY7GPQ4yq94i7jA== X-Received: by 2002:a05:6a21:168e:b0:1c4:d05c:a967 with SMTP id adf61e73a8af0-1cada179499mr1734402637.51.1724228379371; Wed, 21 Aug 2024 01:19:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:38 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 02/14] arm: adjust_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:45 +0800 Message-Id: <3d151aaf876b93a97fd3f070780345a3bf572123.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_maywrite_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..5371920ec0550 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; pgd = pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, unsigned long address, if (pmd_none_or_clear_bad(pmd)) return 0; +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte = pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte = pte_offset_map_maywrite_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } ret = do_adjust_pte(vma, address, pfn, pte); From patchwork Wed Aug 21 08:18:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974817 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=iKgm06tW; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-266-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjYR3Wb0z1ydn for ; Wed, 21 Aug 2024 20:41:15 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjYR2Mzvz2yJL; Wed, 21 Aug 2024 20:41:15 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1030" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=iKgm06tW; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQD4Rl2z2xTl for ; Wed, 21 Aug 2024 18:19:48 +1000 (AEST) Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-2d3d0b06a2dso4606872a91.0 for ; Wed, 21 Aug 2024 01:19:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228386; x=1724833186; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=iKgm06tWty5hEXR+VcYLCYi01JvZyyDU3IdYWkESd9He8mARjmyAie7qlaT+s7y/b3 /O+G7Ii5djZyTY8aEdmRB/MAN8PI/pAfeohZm8thbUPgLP9JaVPpE2C11yrXHECv6QJo zYzjRGhlKfbf8rKPRtjvw5r8SXVZ52+EOrnznJsseQOuqQ51KtO+obfZ5JhmKI32zgz2 MkaPA4jqKxWjYVuQhr1v7P4MmaBo4JOxN6yHnDStkcsRvm/ZXEzmg1fJxxbCx1WUQsHy u5BTXfif9LSgJc5K6sYxeatv/8jH89ikFnPciuhQ7E8hUCbvIkZyz/L2AYO2vsGkemao H42Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228386; x=1724833186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=Qfq50c4lpNHNKUnW1gfDTAUxGqpXbnSFfMYz280dl/QmCXjMdnzfA7BluAM0ZwoIP6 L0f7oCGX7dIIfQXgK3P71s6QcfWp17saRjKlrg/YbvWQ6kY+uVsYB3u7DNVqPgaKRxhR TmS2r0n2cLStAgjsUgxUL9AfWymnbw+QRfjw/6YO3sy1OAQWtA2HcTKWZXwavNuKA1pM Saf0tfKy2RKjpGt3oEmjozQOzRFuMcNDhWsl3uAnWoDWuvKelqP2MSXv6uvJ98Hfasq9 6I1k/8ef1hkOM+P5Uha0P3V0iANFqRzgWAwXozRkeOK+4RkuNOv3H8Fx5HTWewqcUI8t lFBQ== X-Forwarded-Encrypted: i=1; AJvYcCUAa1m01UgeZHy6+s7UxqrWZ78iDapTIp++HYNsb4t/ArvdQehWq3zhuQYBKUsB4CU23KeODywhYosLFKQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyA//U9wWetrQV6KtZQEy3J2V0w2tQO0ICC5J+vf6LttOfBJzrr edauQOs78RMmX1X1YKah2zVc7RrP5urqnbfV3ZMx89jfkKXsjAZBhjjjKWkKkY0= X-Google-Smtp-Source: AGHT+IFyY3NlNyztu6E70clqH4irJgHSXlPUHA4AtOW6RrWWSbwAKaoUYDBEK6KZMX7nlR++kviHPA== X-Received: by 2002:a17:90a:e7c6:b0:2cf:cc0d:96cc with SMTP id 98e67ed59e1d1-2d5e99c8b75mr1585510a91.9.1724228385781; Wed, 21 Aug 2024 01:19:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:45 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 03/14] powerpc: assert_pte_locked() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:46 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..ae4d236b5cef7 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr) */ if (pmd_none(*pmd)) return; - pte = pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte = pte_offset_map_readonly_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); From patchwork Wed Aug 21 08:18:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974818 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=DAsxMyJh; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-267-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjYc5YJxz1ydn for ; Wed, 21 Aug 2024 20:41:24 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjYc4BM0z2yLJ; Wed, 21 Aug 2024 20:41:24 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::533" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=DAsxMyJh; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::533; helo=mail-pg1-x533.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQM09Shz2xX3 for ; Wed, 21 Aug 2024 18:19:54 +1000 (AEST) Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-7c6b4222fe3so3874666a12.3 for ; Wed, 21 Aug 2024 01:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228392; x=1724833192; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=DAsxMyJhEFu5aq71U61pl/jeoTZ4a/y8zicy6KSYMMkIsppjhbftHcIfYiG9XUB4B2 VA2rDLXta6R61mCgKEOd7iM3Pi6cnfbNHGQWG/ou8uQpgPePfZUo5hD0Pss74RsUnASh hgM+X14nbh/wruzKs9ip+Zy2sQTZP7Sq5iQYYxY+C/zevMumhZwQQ9bpLKLhUpHpwqKc tCFFS9HRF3yAcK9gWtk1lc3is/9gpgCwvMUrK+2WEB8HjfjWWSXPM1b9N+lPV3ZIGTJo aBnaJ4PYkzi8QG8AE52lMX0APsZW3xzeCZvLcWv2i8YS1rd7CEum3Mcb6MVIujpdP+qZ GfWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228392; x=1724833192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=DHgJ+3js+lKB71c5dKgvqyB4jipSstd0ryDSt7anzzbqxxJgpUzqvz3e/XO1STFbrO u5mxj4tAnv5jGE5gK+NUP6yJxdAwgRrkm37Y0bIK7AlRM9hmzWZCSsa0S5I+iGuaMtCk VqbuZaiWHc6y5tJbe3LY+U+diYY+yswIxAwtRNoYAxUJVqx9VHOotc0lRIgNom4p96+j 0D5g5CI36k+ijLJeskib+VP1umMQe/e95Q+9gEPGJ+7vHbNzSZbLpLYirvwMCYap+lkk Wg+EFTBiDi17ljHhrQYvpRznSScu+iHxUbWdW6MfPBOrRLRgT2SxNF79pPMEwQp65tGn KByQ== X-Forwarded-Encrypted: i=1; AJvYcCW6IArIpFEyz5wjJ3797QOzywpHvZvidkOnf5tpeYrLLxR1jgrqyeXDNZdK8VeCu+Kg8DloGgeWw/2hyQU=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzzLhSgoVydxYs3mz2zGtE9im4jyps9wH0lLaX+YTT126n7CPTc nh3wdTgobGpnYP0Z9R1G3/FAxUyUDXEzS/7lpfG2O59h6wfrEFTWxX91FOUrR8M= X-Google-Smtp-Source: AGHT+IEDKdhDrkAt8Xw9HjhxvAn16yLBj0ykk9gzeBgjUCoqqbqdejlnt+3gntiVrRuxkT/2XR+s6w== X-Received: by 2002:a17:90a:d98e:b0:2c2:c149:ca4 with SMTP id 98e67ed59e1d1-2d5eaa9d141mr1321075a91.43.1724228392525; Wed, 21 Aug 2024 01:19:52 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:52 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d87c858465962..491eb92d6db1f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3228,8 +3228,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; - ptep = pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep = pte_offset_map_readonly_nolock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; From patchwork Wed Aug 21 08:18:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hWqPruzl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-268-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjYn425sz1ydn for ; Wed, 21 Aug 2024 20:41:33 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjYn2qXJz2yM6; Wed, 21 Aug 2024 20:41:33 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52f" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hWqPruzl; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52f; helo=mail-pg1-x52f.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQW0Pwbz2xX3 for ; Wed, 21 Aug 2024 18:20:02 +1000 (AEST) Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-7cd8afc9ff3so221023a12.0 for ; Wed, 21 Aug 2024 01:20:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228400; x=1724833200; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=hWqPruzlFvWE3XsY5g+oGLVih72ZpmgkYC1ERZlZXrx5reMNWguPq6QPEP5Gecl9hL ibrxLOBrgnd6M1ZsCWwW5Vu93mIVw7DUMwmS1DrYk0OKNxB9E+rQM4VHdHW0GRhuuLII mUlzuIgLU6aA9QMGFIEWopKk+tevaXO1sQRaeJhNrWo7tG7zC+Xg35FumGMyg5q7RnBX /SCNRO/EP57e9ZL5eSeHteqrjn8CRQv8rrmwGdCAswXJlyZ2grqX1SpSf23Q8qPn5Qae IvLrh/URJa/NzyhMJV8cAqInFQMTfz1LDPG5ODcmcu0fHg508szBHm164hBce0kUnnUp lLlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228400; x=1724833200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=MlWF3+kg1GpZaCPaBxJvUeBdUCV7UZcA48G/x4We84nTsZ8YUTuQd3zhSWGCwuj5h5 0I0UrPxFgfuQ66TreQLEgNqyEzyJPU1bxN/vMT/Dz7YFX0A/DbC9LI5Z/cOYRYKEzTbm 8x+PvaoWScwLaulD4dKNe0l9pmU4h36tR7y++AID9h+77bFj+PjsbIhbn26WeOtoeFjs no7CQvdSoXrsUxKIWrIu/2kqdGXlJALFNCaU2xwsoK9mw8j/SCv8Nwz4Q1H5qnofbHuQ gi/DJjOqAedYQSExvkcDYkO4fTEoq0gRLOFxehgj+PjX6RV3QwSIUz+5Wr4UF5eMVX50 S+wA== X-Forwarded-Encrypted: i=1; AJvYcCUc2msz+gveeHKs5S0PDWib5xQashwW0nKVojyf1vXFsS3uj2CBWHFUNocVllYQ9zMOO86aerfPQdbPdhs=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyfsmA/9IL9zxw5XGUkVpf/I9UtsZaOz7IaW3bDRASaEzNcu62d 0bnQyKDVLtQIdf5QTM1fDVBuCC+PQAWb78CmzUDSebJY5XsRwVVZsGjjnqGZKh+4Lq0DJ5Qdgng x X-Google-Smtp-Source: AGHT+IETLNjbMtdResL0gCQrMShCwCpeGL/L1zL6e1zJRM+ER4woSJMvk4wtR2Cz+Z4m4+aIJL98tA== X-Received: by 2002:a17:90b:4c8d:b0:2cb:50b8:e59d with SMTP id 98e67ed59e1d1-2d5e99a5e05mr2805975a91.12.1724228400217; Wed, 21 Aug 2024 01:20:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cdd1d8655a76b..26c083c59f03f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1009,7 +1009,11 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte = pte_offset_map_readonly_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; From patchwork Wed Aug 21 08:18:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974820 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TmUvXwx9; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-269-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjYy6DHPz1ydn for ; Wed, 21 Aug 2024 20:41:42 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjYy54Fsz2yMh; Wed, 21 Aug 2024 20:41:42 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52f" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TmUvXwx9; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52f; helo=mail-pg1-x52f.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQf4SCxz2xX3 for ; Wed, 21 Aug 2024 18:20:10 +1000 (AEST) Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-7cd70404bd5so269366a12.1 for ; Wed, 21 Aug 2024 01:20:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228409; x=1724833209; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=TmUvXwx90dvruGyFof51UbRE64Tk8AHaV8akKqnktONGFDdW4dDEE0X6N2YS3ssE1S v9Ld4uTJtmmBNyC69KhKb6BXPHP9GIqXioPGwGJhsKSZsT8rBpMQFFn5i9Q1Njp2vLzs 1gZy/6+J7Q3Jr4P+pFnRQKzmXjmFuEtR+fLeevkCwSSkgYZ+kNMY0Ocw45EG+Nmfn4k7 3hSIT0PXbg/HyKQ0GjpDMayZw+yDsjxcv7d1hmGf7F+/XV0hzsWOn0txrW3ori6DRixW 1zdaZKW1UKbjG/Buh5l/0dExvJIu6XnwBSBsb7qy3ahLXNFaTuxg0WeZGY5A0PhuboUh 9t5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228409; x=1724833209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=AcTkHE+x0GjzazJAPL/MR2MvbvhiH/HLcjBrM0JSlk7dxMHRa+Sut+KE9dcyfbNcYM GdTqXDrF9vrQCCCzILjTGHEGqoQQbjgwi4JHxobdWy7SltTP1gs0UVjHDSpYY/abrhvf Hj4s1ZQv/xTZgO9XyFX0+BaC9I8LaYOvaIb2c1Kxt7KdXrf9v0AQq2XaSPMU//GAPelY +vIquMZ/6WPDgqRkFPoT9CMox6O4VePauT6qcZKH+gmev8/tbBBME4Mhm70l79FVhuL/ dbQvkGjk5O2MlarmmGzYh3r1v+0hi3mLNxdwumhzK9Qo90fNN/iW6fn/nTL/avEOCDzB dt5w== X-Forwarded-Encrypted: i=1; AJvYcCUnsXprZEmBBRBvfhNxhIRJ6MwgvJ120DniaxWw7oUODx8POOHzkfSHMU5WbEf12dKDXv1TRbY82oXl2Kw=@lists.ozlabs.org X-Gm-Message-State: AOJu0Ywvux7lmVfcrxWYU1c/dKFWNBEBksWQ/7+g5c92vcdGaYL0Kq0a xIxfeaWuT+4NfeFgJcs3vApZr/zCfndZwXPpPQDLSPjVRH7Q7bjdTLBLsjgXGgg= X-Google-Smtp-Source: AGHT+IEz6hEUMU7QDWWn6q3mNXap3/YJ5XtpoR3UgLNTk8qimJ6ZhRH05YhkApxF5Pk84JIDmuV6pg== X-Received: by 2002:a17:90b:4a10:b0:2cb:5dbb:d394 with SMTP id 98e67ed59e1d1-2d5e995c932mr2717849a91.4.1724228407428; Wed, 21 Aug 2024 01:20:07 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 06/14] mm: handle_pte_fault() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:49 +0800 Message-Id: <239432a0bc56464e58a6baf3622fdc72526c8d57.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..d3378e98faf13 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5504,9 +5504,14 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval. */ - vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte = pte_offset_map_maywrite_nolock(vmf->vma->vm_mm, + vmf->pmd, vmf->address, + NULL, &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); From patchwork Wed Aug 21 08:18:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974821 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UKjFS8Ys; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-270-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjZ83S77z1ydn for ; Wed, 21 Aug 2024 20:41:52 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjZ82PcCz2yNB; Wed, 21 Aug 2024 20:41:52 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102c" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UKjFS8Ys; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102c; helo=mail-pj1-x102c.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQn1BpVz2y1l for ; Wed, 21 Aug 2024 18:20:16 +1000 (AEST) Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2d41b082ab8so2511550a91.3 for ; Wed, 21 Aug 2024 01:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228414; x=1724833214; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=UKjFS8YsjVO6O5pn+se1rrFO4hMfOb1Cd8CQu95OhBwI74BCS3qeeIaIBDVyVxv6xH etGYk+lXDOL7fPVeZx7etGUGT9wlOwqPS4NmT4X8RkocmYkNDusonI0PpiYrx9eIoYqI tj8sbE+dtu8Da45RgDuKjPy86bDPOvHNVm8LlEd8ifUd96Z+NLfH0dkzf2CxhBBSuFYR zTbaJ8xRfHrWuGdoj96fzbTyXgE1nVvpkO1UJyoWqL+Jnhigrgf3FSIh/ldeGtKhzMVc VWzejsC7Y2MBMGB3mpJbIwyXx08qFhItAqhcnBGfaz7e+OmZZ/jEccgpmlSaK+G8F1pr JP+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228414; x=1724833214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=fcOog+cRgEpI/NHTOrPe1Bc2AXP1gjwl4yE2CkBB8vKfyjh2Srtr1xA25lFsb+Yi8C rKzYefp3jS/FBLdE6tNAe4FCQ7fv621vusJltKZ72ehQlBsaJxinq4kbq4GZs5BYkPD+ bQ/7se11kwxhImUFLTiAFaju2Yxb4mugMvtv5ZasCy6acx6YQV3Nd3XEMaEbKuNnGUvA IDGyEi8Jpg0j+tWtTD0TkAd6qRqraEUO8g87BZOzDvEMschWSv0B+QGZ2D3/KNZqCl5k 6BxiBWJMcbjHHxLApmvenco/hrGMKe3i95KX4aDzGowJQFyPFjXiwvY8vOpizlBC4E9T N/Fg== X-Forwarded-Encrypted: i=1; AJvYcCU56TzC773xUdTmkn7MrNajZwnhoD933tR/rV2ve42esHE1rCd09dwQCYv371hRZnToVvcIHabewvlxAYU=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwH/I4G8T1R4aHTgN3P1nBKpXHgMHgIxZ4kdJQjLftg6mwHRGot ubU+JJrxq0Kh/QMPmmaZZPh/2df3/23qsJYXHo8edghzbiHnROVgs+Tch8PykQA= X-Google-Smtp-Source: AGHT+IEASO/tYfbmKEqODBO5ptoLkIYqajvyb9bjTzXqrTBGFe58odiOJRUvoUKiSb3oI+1Cc8Ov6A== X-Received: by 2002:a17:90b:1c01:b0:2c9:9658:d704 with SMTP id 98e67ed59e1d1-2d5ea4c9ab9mr1366119a91.40.1724228414402; Wed, 21 Aug 2024 01:20:14 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:14 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 26c083c59f03f..8fcad0b368a08 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1602,7 +1602,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml = pmd_lock(mm, pmd); - start_pte = pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte = pte_offset_map_maywrite_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1610,6 +1610,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, else if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i = 0, addr = haddr, pte = start_pte; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) { @@ -1655,6 +1658,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml = pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1686,6 +1699,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml != ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: From patchwork Wed Aug 21 08:18:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974822 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LywolRsl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-271-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjZJ61Wwz1ydn for ; Wed, 21 Aug 2024 20:42:00 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjZJ45Q4z2yNs; Wed, 21 Aug 2024 20:42:00 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::534" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LywolRsl; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::534; helo=mail-pg1-x534.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfQv3zHlz2y1l for ; Wed, 21 Aug 2024 18:20:23 +1000 (AEST) Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-7c1324be8easo376478a12.1 for ; Wed, 21 Aug 2024 01:20:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228421; x=1724833221; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=LywolRsl1+p6Mw9bZ5CDjTrcTYqHA07RAfllbzn6N89+uHWB8BFv1/f7tBa/SQn8yr 4n4WTwE0hf0fdYEnTl9oyh0EGpJ6KXq5oTo+RxnBkS1Gi1waAA+kUEp56e6I7Y8OuFd0 rCaVuSUt18Rnbe15KU/mbvnuLIU+bUvZR5vG7qjp/OXukRd2yEYlJ7BCgald4+A7tWxI 89oGqpUGr43PYdidnCrndWqX5WbF6FgcyRfcmab1H6hGpK2ED2x9WrRUe2cab5et/VSL eeuFOrw95QtR/JArpOXes7RCFTeeShl1ghUtWBMNqP0EYXq4FNOhNreJ5uh+I3MpFSYs vWtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228421; x=1724833221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=ZfrHwkUHkbNZmXZvkXMVNDu1ANbIirdlyY/jEXDRvIiwmAJh78Yo5uhro0pHfqrEa6 GGMchBXCp99QdzZgDKe6byEUqF+vagQjPoVKSG9Jye4y2Xxtjc0IxAV+yLbjB56Iueb/ abQ/aLZMWYmXFRqUIZkFirv8Jf5CTP/vMB3fcbJzfheCeXiEapAtp71ZhfNui5FqnZ79 850tX3oubzChvQUIElQWZDSCpK/gUnnEoRf2fCjV1eOpQEaOjQCja7F88rbMLfqWljlX FVg2uTT5jmgB/rtmMBToMOg8ByCKJVokfJbtZCEk/M2pOQIjJDHY+fMzbBgbGo3CpyNu H15Q== X-Forwarded-Encrypted: i=1; AJvYcCVrIi7TvASIrgVD5qB7TiWrvuRX9gl4yuSeSn3shYkxuNq1qMyoyXU2V0ACL5H92ZkaFmoRbrdYodBqios=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz9s8HfvP3RLVycIG8fnmx5an5JumuSAUU7ldfgWghjwcbHij+Y MZHplp9PdSRFmrQOtn7235I6iMWFVHhE0MVQdxDQQcSquQ3lRVdQad6RNdD9Dac= X-Google-Smtp-Source: AGHT+IE93wrdhvj1jPWy5zvxTPwSbhlcdosDYnf+NOlDda6ytSjhdKrpXWKVADSrOUal4AF7Aqrd8g== X-Received: by 2002:a17:90b:350a:b0:2d3:c084:9d93 with SMTP id 98e67ed59e1d1-2d5e9a2f0damr2610948a91.14.1724228421082; Wed, 21 Aug 2024 01:20:21 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:20 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 08/14] mm: copy_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:51 +0800 Message-Id: <05c311498fc8e7e9b2143c7b5fef6dc624cfc49f.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index d3378e98faf13..3016b3bf0c3b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, struct mm_struct *src_mm = src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret = 0; @@ -1108,7 +1109,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, ret = -ENOMEM; goto out; } - src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + src_pte = pte_offset_map_maywrite_nolock(src_mm, src_pmd, addr, NULL, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret == 0 */ From patchwork Wed Aug 21 08:18:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974824 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=e1M645wF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-272-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjZj6gbcz1ydn for ; Wed, 21 Aug 2024 20:42:21 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjZj5Jybz2yPR; Wed, 21 Aug 2024 20:42:21 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=e1M645wF; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfR25q40z2y1l for ; Wed, 21 Aug 2024 18:20:30 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2020b730049so38806825ad.3 for ; Wed, 21 Aug 2024 01:20:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228428; x=1724833228; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=e1M645wFqiketQzZaIP7J0Myg5atP2kSDGwZgF3zMP3yk7xwefNw9LwipWY14q8rql h8mHKEhysol9ZL/yNmgyV2/bPIqomP2KtQvkj3ZrzObJPuqhAyhQvgY7CyhofhSRAjbM aSZA0R8XK0ZqqZvH8jxAhv16qMzV7kc6zHRrlEQXB8CsjDQXSIJ5Ij2K5hyekC6UHo6l 6Nkl8WYfqr/CcwWQ29jpIo+ynb2PdMAZ4xTt8MgLTolPXh8HslYQRCF8IqHjsSf58cNW AIssKkgfLvo1DKU/u3D8Zh08H2c/Hep18BzmwuMkuGIWbM/jY/gf4uzNUvEH3pGMa+gX yrkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228428; x=1724833228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=wfeCzlSIEjjh+yQw/WXW4jTBGiPqXOhvjASL/xo0geRHfWfxaJZWgNvf1u3+pIvcBB w07Sf03rm/vcLmTQIjrgRkauv20z6WWJpqN+doI/VrqIu+o7+bBL+QI649r9TwlJzxc2 z6Ev5LPgz0DTf3RY3sHIrqIrcaKHEEblhqE5vYz3w3nxmfvEOvgNiDv47QyiwCqEW+qg L9DBMcP3febcWP3PMr/F+7orZLmcDRnHYp7CCmsHSKaL0wB2A1o/ljpvtQrebfufyHOv JYFYZOBKhEQM6h6jkNdScJ/QN+rtJI7/IqoJ+PFuKTK0UPJ2p9uTDlLU4Pr8R+hToiFm PN/w== X-Forwarded-Encrypted: i=1; AJvYcCUDDHWsFCy0S6uVqDzXcXtpFL0tSPhZdOz8o5QsWJ9sn5kNauZJgo7xegCa70YCMpilxANib0ljz/5H/I0=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz5O6GAnbK7mvk22U1Ubjoa9uvdf2/wkIYL1rYS4QDOwmI9ve6Y SBG3gxYKY9BfOUqRUVptyjcSeeOFpX/EE0Zw/YogBWEydkPNQvMuLZU5Wb/3VOI= X-Google-Smtp-Source: AGHT+IGct1lfg7s+Av92JTuFI7wGr/k4q1ybIYiVEVcgmdImld0nRYlRbV5Vh9eNiEYSyV0nSbS/hA== X-Received: by 2002:a17:90a:7893:b0:2c9:635b:7271 with SMTP id 98e67ed59e1d1-2d5e9a24ea2mr1616047a91.21.1724228428100; Wed, 21 Aug 2024 01:20:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:27 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 09/14] mm: mremap: move_ptes() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:52 +0800 Message-Id: <89bd1d48db1986ec83dcae228defa748f38238ff.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/mremap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index e7ae140fc6409..33a0ccf79c32d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -175,7 +175,12 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, err = -EAGAIN; goto out; } - new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte = pte_offset_map_maywrite_nolock(mm, new_pmd, new_addr, NULL, &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err = -EAGAIN; From patchwork Wed Aug 21 08:18:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974825 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HjF5ItjY; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-273-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjbK5Dybz1ydn for ; Wed, 21 Aug 2024 20:42:53 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjbK3rMbz2yQL; Wed, 21 Aug 2024 20:42:53 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HjF5ItjY; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfR91Gz6z2y1l for ; Wed, 21 Aug 2024 18:20:36 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2020e83eca1so43752575ad.2 for ; Wed, 21 Aug 2024 01:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228435; x=1724833235; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=HjF5ItjYseB91OrfwVkLp26Qy6+Cl58KP7qv5e3mKZdc9YkP1BLerdnlqzXNWIlH2h u3RKVz/lqnDgQXLMt7O3kCN6oalEROW6fV+jEOKMFCt6Oj8QFBu6txEKvN2TQmXLBHt5 4htoTdNju6qLwxUvmBKSGF/da49ZXidw52DprKbHF436ULE6AEWdk64t4kvpMzt2AM1y wZuyHtEzN2A2EOj7YeVRCo/Al3X9XtESocEBIOUGaUWDrI2pdWP0FiewBNYwF+Pf3Ss5 l26DFn5sWrDF6sP5OBZ7ek6qazpPjUdQEQ3OXpoe2Ah5wSoEt3HCYKhruaK9rS6SZ0zY M5eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228435; x=1724833235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=AhZTOROwZPERzNDl1pjBYbHOQLqQ0hU5oe6kYnSGqDY0gGkZ8iCWrR1TzrQNbBBmDo B0xHBXEC6F7o2BuFiLZtdS3xWgLeEjzp2aQDEBjm21KsFxzSf+E0KCeq7cSCBwa230jA Hg9B/IO9eB2V/ezpgk+F95OFhhd3M2RRAn5Hgve4/FJ13ssXEyOBEvP+wp68EW9BeZ/b +lasiYvzHMQMOglaVbw5A05TxpC6hXpiDeLdxsoUmGWN1NXni1HYqjA0lFVmqoNbef/6 lcY6aprAn9pjCecXgaumtJS427rmUi0AUnSZzKyeJ6Q+ZpLMEet3kFM39Qmikn2RFihD 0H5g== X-Forwarded-Encrypted: i=1; AJvYcCXshCK07oXzoYa7MaoURzpaAJh7poceiV6jQE4MWSnpFaw3I4HFnV7g+QwE8UqLzgHep5CWMBiSZcHT10k=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzFqL7giYDPHCwpp4idCz/hPuDa4jBQ5tJZHPyhNck2Rq2661OB 3jXQob1iwjKrT/2/Akvnnxb6ICJDJKJtmMCKNewjbcmuayFGQ5ETYObywehHj4Q= X-Google-Smtp-Source: AGHT+IF3P/4155uMvmGYpoowuXrhy2v6aHwudYM9PJFMCLkX9j2rCBHO6+Z7X1wkaLdhxTlExnQxpQ== X-Received: by 2002:a17:90b:3b41:b0:2d3:b5ca:dedf with SMTP id 98e67ed59e1d1-2d5e9a0db5cmr1626393a91.17.1724228434854; Wed, 21 Aug 2024 01:20:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:34 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:53 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..da806f3a5047d 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) return !!pvmw->pte; } +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte = pte_offset_map_maywrite_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp = pmdval; ptent = ptep_get(pvmw->pte); @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (!pvmw->ptl) { pvmw->ptl = ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl = NULL; + pvmw->pte = NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); From patchwork Wed Aug 21 08:18:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974826 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OB9jfpFk; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-274-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Wpjbf5Qnmz1yXf for ; Wed, 21 Aug 2024 20:43:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Wpjbf4DQCz2yRF; Wed, 21 Aug 2024 20:43:10 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::42e" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OB9jfpFk; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::42e; helo=mail-pf1-x42e.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfRJ2hS4z2y1l for ; Wed, 21 Aug 2024 18:20:44 +1000 (AEST) Received: by mail-pf1-x42e.google.com with SMTP id d2e1a72fcca58-7141b04e7a3so935242b3a.3 for ; Wed, 21 Aug 2024 01:20:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228442; x=1724833242; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=OB9jfpFkETcyMURpbr4iLdmB2nAdc7YdAPaLZFgFk5Sz6rTPz9bhDpXpL/1eRQrmEs RWE66Yq6HRgvuQ9Gc/N8GrhRuMFDIDKYJRdUszXpUSsK6vy8GcStCTv9rhkWJnmP+UGh oKRzR4ABl6haeODfeOc2v/NxC/krvwLntHD2mscFh0B2+QaPoaWjRgg13RoE3IxwrWv8 prGXHmHD28zduNWAI5Yp/v3K/RGOP0RSSi2v5dkKBa2oaySOd5CcFAaCY02llaM61//D vxdW26Z4Nj/w/EVtcKCpi+WTQoIi2avA6biqUj/zBndu2ZJKSzSl1ntL7vJTbg20GBs6 LUgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228442; x=1724833242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=J/z5L1L6hxobhPBi7iFMmdZHSkBQktMu3LB7OALKGiPZ/hSjWz09FbtPFHY/x8sygF WREQllY9XrOa+7X25h1ks0CmIibfl4bjn1emJHrr1hT01Xqdr6bVT1YhZ5aCq6cynI3H 3TRlKfic3rOANp4u9S9RUuCcO+HZZA4kNTue2+GwqKfuEUXx5xi99rpU+M6xBbOGnbrc uxo+7QNJ3CxBuXwdABkkMkCqN8qsWc1Oxpcfoy3FkD5DdRRXlrzpOY5HizJQ1p/mMV/y lzkufvqli+CVrv18uhN2ngjRoqFOav6fkI3Ca4Nh8J4rgd0hK/i32k/UzPih5ldod+48 9SfA== X-Forwarded-Encrypted: i=1; AJvYcCX2HvSFBAdEAm2zosYtreNn9fryi54C8OICzwLwWHFuF03e3U7P4SWSKgaNFV2TrLRtUP7aJn6MGm4N+/Q=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwD4nss+eZGEd4WXX+ZNYlheTmIUN4gJ4BFbn9klqGdX2ZOrhtj SHP+jnKWztNvLtt9PbUwjum0A1HxDUPNV+RK8GGukAzuw3NhfwAmq05227lenV4= X-Google-Smtp-Source: AGHT+IFM+x1tDjPFZswDxs94u/u9nyN6BfWsdj/MLtJd+fBRDvg5mEN2O6omshjTkNyUCHM1AQzwKQ== X-Received: by 2002:a05:6a20:e605:b0:1c3:ce0f:bfb7 with SMTP id adf61e73a8af0-1cada055937mr1798278637.21.1724228441599; Wed, 21 Aug 2024 01:20:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:54 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..310289fad2b32 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1146,7 +1146,13 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte = pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval. + */ + dst_pte = pte_offset_map_maywrite_nolock(mm, dst_pmd, dst_addr, NULL, + &dst_ptl); /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1160,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, goto out; } - src_pte = pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte = pte_offset_map_maywrite_nolock(mm, src_pmd, src_addr, NULL, + &src_ptl); /* * We held the mmap_lock for reading so MADV_DONTNEED From patchwork Wed Aug 21 08:18:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974827 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hY8g6DfC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-275-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Wpjbr02c8z1yXf for ; Wed, 21 Aug 2024 20:43:19 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Wpjbq63Vrz2yTs; Wed, 21 Aug 2024 20:43:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1034" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hY8g6DfC; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfRQ3RSVz2y1l for ; Wed, 21 Aug 2024 18:20:50 +1000 (AEST) Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-2d439572aeaso379997a91.1 for ; Wed, 21 Aug 2024 01:20:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228448; x=1724833248; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=hY8g6DfC3A0gVZXI3GqfPMSCioVJDLkG4mPu014MePtOEc178nemzmmAN44EUleeCv VX2cCkynDOWPShBtQzyu/zU8dl0Q6Rer/5501ufTf6QvrHeus3RcO2CxR3g0c8kiKETL BOTVVhIjPoWg4m0G0OCOci5RmqKrKamdgFzzl+gZdzTgoIf7DCucEQKa5pf0kB+MTJmC d/dvskqsD9LLYu4bVkvoR2ApAmLz2ph3vI/Xay6w4YPbN3VK/bW07ahO5cpA320eGSYJ TtjTNVX7e+kCaLvc6evvwPfaILUN0cH+c0FRi8rghMq2Qgz5syzRTV/UKdiBvh/r2yVt E8gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228448; x=1724833248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=lDDxY9dgGcECKXc/H1JkejVrlJh9CRys8TZ1MnG9nv0oAtrC4yU3MIR9SlkZDWvg6l 1WDCGqkuoJpqk4PDgkpTiOUCZxgDGgtEatIpffk2goAjRVuIkJdiSPbt86Au1q5cJ5uA 0k3K06Dt4fTX1KgUHWhJXbMKIq8+Ugn+kLBVOp9w5Wc7QHYJPinKozgTuYzTe5wjlsxH A69xRXwq0IO5s/pXa0TFr82WPWyvUG1nmzJXdLPbulShEO8nEFrdVoTGiATdksHIanZy vBJjHwusZtjOJMLmiD5Jw0jYyyogpbTH/noMFMu94SJ8fdhbFxjdMTJ+ghaKuBOCjmEw 490g== X-Forwarded-Encrypted: i=1; AJvYcCWFG1Ii8ZfjywLm0zvyfCLxMLFHrqh5pl39Pn/zwJcsMwgf1aXhDMQTbAQaEV4QayWHpb58fn7M5PJ6h1c=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzDyvSmSDmkCZKApFDYCho+Htk7Pq+qu4i9AaQjPo0ikfCPPTXA uRVjfARBAjZ8AwotNHITSoqJioM+fZoZsaRX8dQULw7cA8c4MO5pR6ChpvLgwP8= X-Google-Smtp-Source: AGHT+IFBw9G+gRGiVw886HN5nxLIybNKSDx7re7KNY7nJCh1bBtNFP6GTlUB4MiVeFJVy0IWyCdYMw== X-Received: by 2002:a17:90a:d987:b0:2cf:fd50:a2d8 with SMTP id 98e67ed59e1d1-2d5e997c3demr2593063a91.7.1724228447873; Wed, 21 Aug 2024 01:20:47 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:47 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:55 +0800 Message-Id: <1bdb1ec4a30acf6b3a376f746c96236328904768.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..a6620211f138a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(max_seq); + pmd_t pmdval; - pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte = pte_offset_map_maywrite_nolock(args->mm, pmd, start & PMD_MASK, + &pmdval, &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, return false; } + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { From patchwork Wed Aug 21 08:18:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974828 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=chMYZpW8; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-276-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Wpjc12JS5z1yXf for ; Wed, 21 Aug 2024 20:43:29 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Wpjc118ckz2yVF; Wed, 21 Aug 2024 20:43:29 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::531" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=chMYZpW8; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::531; helo=mail-pg1-x531.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfRY3Qxnz2y1l for ; Wed, 21 Aug 2024 18:20:57 +1000 (AEST) Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-7a18ba4143bso4178394a12.2 for ; Wed, 21 Aug 2024 01:20:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228455; x=1724833255; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=chMYZpW8PqTbTfqWtrKBS1Id3jbOpkk+N8kHcZptNEEJA7ALYWnAfxGXoaAXYWmLE/ zLpGWOXBdosJZQ37va1w8h0XmyMeLDTPiRgUhcabgwPrZD+wzZJeFt/Y81gpwZ80IING NIaxjlliVqX127zoPtrYqzQil37A+/j4dBLQx8vxVKjFra2vKAz45uo7Sz8qcF5PXn7c AMtrX1QzanGNoNTCMP3x026bJZiAT87dFwDLNzS1ds86mv1IItPw8YMheiyOpmBVZeW5 OA99jEhZqY7qQbPzMMNm3PPcYorzKd1Go2BhW26r4Skj13a2dMdiCuGYWW0ivLcNNv/X l37Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228455; x=1724833255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=YY3lSRBc+Pi4HQ4pec7/kce2+lpH7w4XTYgpQmHyv1YTWs9iqev6Q15N8zbPqQsSmX fyjzY8BjTTJKW23zUn/0zIVHgN1VrLcfjvHoWX2tdwZK42CmJvBOCngwIx5r4BXgQB1W /ShEtAFcdCXGGBnuqVhCMf3d2bewO+y8hHCmWi+xFsnB5kOAvGJP8/qMY319Hi1kavQV y5Tvy3HOnb5a4wk2fphxSTtaa2MSWknd8Z/RlI7gcAJ+2w7ez4Wmhjj181aJWi5BhB2s WKwnoRE4VaX0t0WjweB5U0nYYUfQ+rKHbFo8HFFb9LUNjpjiE3DksPmyUTayXeXO/n4C drAQ== X-Forwarded-Encrypted: i=1; AJvYcCUZorhOjocDwkoeUZDGRU7p8DoY4rwg/k0OeQeYHMQnxgRFcI6nsParROGVzGm4Y9LEoRYA3h2Shx6gbMI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyxGg+VT4qg2RhVhD7PGwNmSOZMP7ABTGVOAGFHWTnh2JdjTM07 i7+aFKC3EQQk+MrCDeK6fjv52XvtA5vND9TIIUpEpqwAC655I9RGPa179zRqVVo= X-Google-Smtp-Source: AGHT+IGrVkekNFESU70hIpUuBMPasbPeBaSvK0LFz7NLCJlpTpTqHUrWHKc0kKTsT//aSMihe2hbaA== X-Received: by 2002:a05:6a20:ce47:b0:1c4:a49b:403 with SMTP id adf61e73a8af0-1cad81a4248mr2186429637.46.1724228454750; Wed, 21 Aug 2024 01:20:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 21 Aug 2024 16:18:56 +0800 Message-Id: <129f11e3434c5af898096c1cf3fe378f50bc94b3.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index f54f717ae8bdf..596b425fb28e6 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_readonly_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fe0ceabcaf39..f7c207c3ab701 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, return pte; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 29d1fd6fd2963..9ba2e423bdb41 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) return NULL; } -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte = __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp = pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unlock()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_map(); - * but when successful, it also outputs a pointer to the spinlock in ptlp - as - * pte_offset_map_lock() does, but in this case without locking it. This helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spinlock - * pointer for the page table that it returns. In principle, the caller should - * recheck *pmd once the lock is taken; in practice, no callsite needs that - - * either the mmap_lock for write, or pte_same() check on contents, is enough. - * * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case without From patchwork Wed Aug 21 08:18:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 1974829 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Wous8dAw; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-277-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WpjcH4S6Wz1yXf for ; Wed, 21 Aug 2024 20:43:43 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4WpjcH3Fs5z2yVd; Wed, 21 Aug 2024 20:43:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102b" Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Wous8dAw; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102b; helo=mail-pj1-x102b.google.com; envelope-from=zhengqi.arch@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4WpfRg4lKTz2y1l for ; Wed, 21 Aug 2024 18:21:03 +1000 (AEST) Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-2d3d662631aso4423363a91.1 for ; Wed, 21 Aug 2024 01:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228461; x=1724833261; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=Wous8dAwZv+RRMRhtj0Knl7jfXJKexDcvcVrS7SapK065lDoqy9fuqT2VAprPgnqhd 18XyWwR55H1uHVCMOExYbqoEj4F83YaQ/tJ1dZg/+nLrZuzILEsgKUBDOcfzCaOFLc7I jpUfavj1hJfl7qKiBd1wCtJfY106Ch/N661O6TSMQUX5qyr4UYCwrL9/UxsfEEbA9Mw2 Mu+AsOxmBn11A9SrZPVWcQrEffX5eY0pOL32/eVzb5SbGL8DYBkEOaOJ6+Z+XyuW1xdV zovHVGrsEyo5q0ZYA5jOcY51Xso8oduRa5/J0S1hSoWv5xX9MVKN2L0RCHXcwug7MSRn GpHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228461; x=1724833261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=iAwph/ObdgfAjKblCtwrelyqFLkdzEjnA+4ZTzJkt+m+u02YcjpXnf6o8bV1iSnL5W LlBKsGwhDdoRIlyaaegbXgcZUQwJp8skEEwfiCzL1B7S4wZc6WazPLRNlXQlsSP36YU0 yRzC/0wCkDcYSvt/ClqXS9yz8vPs6YhuRzjp68ax5BMofGHN0vb2UdMzwkHy4kPGd2+Z JIU+nIqfWasS3vqO0YRzLMblawfV/oX58rv+6YVWDInNRANzsNFmpMES+/etsL6ohw0j ELr2HRCzwZdr4OXOnX6llvR1vc9Fp2N8450zeUv+FMqUiCQkeRQDJy8GhxTu66TUm/Wd ycKQ== X-Forwarded-Encrypted: i=1; AJvYcCXiP5gQfjXA6esY+S9T+oysw84aBNfqc6rWScbOwYQfM4UBo4aZ00pFqswXKOkLvhLr2qcjPn2bs8jVjog=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yzik5FthZ1qqE0p5kOJzfwm3YT67Q7Y8/WtpJs9QYLrz60Yu8Sr GR06a9T+2vu/Azf+SjqWVoln2p89ng+VxE8+KPkBJ3deZPDr52VIX/OIIi4Twak= X-Google-Smtp-Source: AGHT+IGVCW7LuSOfGhGGSqonUfZfQEf1Y51LpMxCXHDPBrQX3ltYUdmnoycYhvy3ErzvacrcFVYd1w== X-Received: by 2002:a17:90b:3851:b0:2d3:d239:24c2 with SMTP id 98e67ed59e1d1-2d5e9b99f57mr1469884a91.19.1724228461120; Wed, 21 Aug 2024 01:21:01 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:21:00 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:57 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Post: List-Subscribe: , , List-Unsubscribe: MIME-Version: 1.0 In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_maywrite_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8fcad0b368a08..821c840b5b593 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1721,6 +1721,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd = false; + pte_t *pte; /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1756,11 +1757,25 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); + pte = pte_offset_map_maywrite_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml = pmd_lock(mm, pmd); - ptl = pte_lockptr(mm, pmd); if (ptl != pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl != pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma