From patchwork Wed May 13 13:04:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S7KEWUMy; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20849-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFtzj0bnqz1y5L for ; Wed, 13 May 2026 23:07:44 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFtzf1ldRz2xpp; Wed, 13 May 2026 23:07:42 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677662; cv=none; b=VFkswSi2F4HiO1tftzZwpiHO4mYhxCSGmJYXMUn+XH1aRIrGb5odoIv7LtAWXvNz/6sA/YbJT5F0l0lolXnZ5mHv7voGCBpRh3cNyOttsGaV3mx5ZYy1/OpxzlcWS2/R7x2aAH+UZtWv5s5S1neQfPg3j1td3Bhj/9aUhAHLsTD1cuEEIHgsdRfJe8PBBQT4+TlhRCrcXthUaLe6jjAqk0XLY27x3Y1wWJF34whjX7VGSnxKOyireXm+N+WtM9MyCl0EbYJJtn9cQWRmRzc0egqiUyQ0itBSdlgu3x1JTprLbEg6EiXm3GGwqjD0xcB4WqL3Sj+NgPY42Qznd3PucA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677662; c=relaxed/relaxed; bh=glZCSgCcQPUR5mPOULt0p1weLkVndlFTk9nY8tD6isc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HUbjP4/etSBpivwHHoGaNpZpCwaQMeooNwwRHh8jjBMNNuJQIJY0mA5ggxYt0OgXNvXTEKG3a6w9IvCBNayaaPgjV1RkdXsRFzMz2/DbTLFoBZMs21Ho/LNtcIGuCPNuqOvpNWYbKvkSX8HtCjJUKLxhx4ATiOgCjLRlMngoqWv5UgyrPJSLcbJ2WtTd8QKhf48xBQzf4xbqC9ZBrL5xSpHUBWsJKUzz+uFj2ne36nh2HX8JwiUdUmtUX7W2Kh30SeiLAz/JX4dydVwEEj9ihalMp5RZX5hBh+bjK9UmvmK1cYTCNpmjz03lcq6mQYsjza9J7GL3KNzD5ho+t38OdQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S7KEWUMy; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S7KEWUMy; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFtzd2QhZz2xld for ; Wed, 13 May 2026 23:07:41 +1000 (AEST) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-2bcd3ac3307so19190005ad.0 for ; Wed, 13 May 2026 06:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677659; x=1779282459; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=glZCSgCcQPUR5mPOULt0p1weLkVndlFTk9nY8tD6isc=; b=S7KEWUMylbi26brgV/OM/s/2fS1esVFiYB9WLqdXpie6fUSCiazk0XkqwWlQ4G65Ax cUTX5q1hqCRGKw+4FHesMlSvbu6nT9DDDSnVqKxfEhjRfx3nu/tMDYgZHJyqyp9Rm81R BDQdTUgXQEla+/4RsStR/DhOqzkwDrzeJY9JkQXim0rQPPjIzDyvOoc9iepC17e7SA4M 8oUUO78ez9Xjs4yvDdgCXPGVj3Uz7vIknNAGukmsYkHSPXtGlrvn/nIeDvddH5NdRAfh dHlW05seKs/fFyXH2tJIo/OxEL837SphTtQAK1xPW0lFwFgNRkmlKztwlWsH0d6iRW6n gfeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677659; x=1779282459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=glZCSgCcQPUR5mPOULt0p1weLkVndlFTk9nY8tD6isc=; b=J0eR7X1WlJD03UaPH6svmaTy0t5uFUrOhYpaWmFGf0IaeDJ7qv9iFCYeNk3UcAJJrw 1U+ypPja5rhNg6sdzsOzElh2vmqQN6aF9h/1i/3/rIQg/V53kimbQ2g7TbyUYbIb22nQ v759VRZj3Vx3jNva0Qs9RrFsN51rOo5oRRwJP698l03cq+WuMDcGBbkWppKUQmbW7CAn uNY82aKwkQcu7lYvgMESXfHGI/B3nE2rL3r1pvdRpsMiyWTCa4EUOCgE0UF3DocWdpNx b6nX+1at0AukInjtM3eD8H0C+yBiFU3/ufvnHwF5WB1bxb4qvB5KOBTZbwOIYEVFQ5U3 Kk7Q== X-Forwarded-Encrypted: i=1; AFNElJ8c7SWWlVRnqB4BhmVx9pf3WYO3lsnszBQfNWpBSwV8nbTpamI2u8fB92ntgV66v5GFfm6hgyB8BkhN+Uw=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx3sFXsHNSK3OvhqL5R+L0GATCBm3CQXaio61BUeqmL3Tcbtlhp XSE9IkzvQf/dG3IsmTMQQSB16vSK1XRKNzPxsPavAkLcm+jrJgx14nRAz6hZCqCiAwM= X-Gm-Gg: Acq92OGMiKn1/X65eh+9r0xJYvqoxDpaRLaUOOGSgtcggn+WX3kBr0qfXzaZmrwWTw8 WSPrS6z8/zn8jPXitjMf8eAZt9dhrfvSvo5ANzoqBf2aBbetkI+efzup/ADSEOzwVFmtwfDHAiw /h10xXKyjzXS8K+9SQuxtuDZJs8Xnmnf2nlamBwEh3lvrZhN2dI8OXYSltY442yTgs8DJomt21U 6HZrDUCiMy7gvUasFkZ9nmiQzxuu8zWljAMM4NF0POJVJYGuUAkFUEqM+7KaeHdRe41ydgXFTgk QzmrppO7g2NixoVcXGclHoTuOKDzdbNVO+6nQQCia0AL2f9sABb50DANrEwAdh6734hDai6tkqK hgOd3AlTYWLYq4P1AvYdjapiqNLTJ2BtNasQogN1TNen063EDLv7SZdy+gbdGmGd+0FZNlPkASB EM47lYULDKy4zTC27k+0aZMTa6OWUrjkpVNROCJ8xHD5Y9xZtixQcCXE4wWDY= X-Received: by 2002:a17:903:907:b0:2bc:e216:db47 with SMTP id d9443c01a7336-2bd2713c36cmr35966965ad.8.1778677658559; Wed, 13 May 2026 06:07:38 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.07.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:07:38 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 01/69] mm/hugetlb: Fix boot panic with CONFIG_DEBUG_VM and HVO bootmem pages Date: Wed, 13 May 2026 21:04:29 +0800 Message-ID: <20260513130542.35604-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Commit 622026e87c40 ("mm/hugetlb: remove fake head pages") switched HVO to reuse per-zone shared tail pages from zone->vmemmap_tails[]. Those shared tail pages were initialized in hugetlb_vmemmap_init(), but bootmem HugeTLB folios are prepared earlier from gather_bootmem_prealloc(). With hugetlb_free_vmemmap=on, prep_and_add_bootmem_folios() can access pageblock flags on bootmem HugeTLB pages whose mirrored tail struct pages already point to the shared tail page. On CONFIG_DEBUG_VM kernels, get_pfnblock_bitmap_bitidx() then dereferences the still-uninitialized shared tail page and can panic during boot. Initialize zone->vmemmap_tails[] from gather_bootmem_prealloc(), before bootmem HugeTLB folios are processed, and drop the later initialization from hugetlb_vmemmap_init(). This bug only affects CONFIG_DEBUG_VM kernels, where the relevant assertion is evaluated. Fixes: 622026e87c40 ("mm/hugetlb: remove fake head pages") Signed-off-by: Muchun Song Acked-by: Oscar Salvador --- mm/hugetlb.c | 19 +++++++++++++++++++ mm/hugetlb_vmemmap.c | 17 ----------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 31b34ca0f402..d22683ab30a1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3382,6 +3382,25 @@ static void __init gather_bootmem_prealloc(void) .max_threads = num_node_state(N_MEMORY), .numa_aware = true, }; +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + struct zone *zone; + + for_each_zone(zone) { + for (int i = 0; i < NR_VMEMMAP_TAILS; i++) { + struct page *tail, *p; + unsigned int order; + + tail = zone->vmemmap_tails[i]; + if (!tail) + continue; + + order = i + VMEMMAP_TAIL_MIN_ORDER; + p = page_to_virt(tail); + for (int j = 0; j < PAGE_SIZE / sizeof(struct page); j++) + init_compound_tail(p + j, NULL, order, zone); + } + } +#endif padata_do_multithreaded(&job); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4a077d231d3a..62e61af18c9a 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -870,27 +870,10 @@ static const struct ctl_table hugetlb_vmemmap_sysctls[] = { static int __init hugetlb_vmemmap_init(void) { const struct hstate *h; - struct zone *zone; /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); - for_each_zone(zone) { - for (int i = 0; i < NR_VMEMMAP_TAILS; i++) { - struct page *tail, *p; - unsigned int order; - - tail = zone->vmemmap_tails[i]; - if (!tail) - continue; - - order = i + VMEMMAP_TAIL_MIN_ORDER; - p = page_to_virt(tail); - for (int j = 0; j < PAGE_SIZE / sizeof(struct page); j++) - init_compound_tail(p + j, NULL, order, zone); - } - } - for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { register_sysctl_init("vm", hugetlb_vmemmap_sysctls); From patchwork Wed May 13 13:04:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237816 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=PiEcvlXA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20851-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFtzx4kMgz1y5L for ; Wed, 13 May 2026 23:07:57 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFtzk6HJsz2yQn; Wed, 13 May 2026 23:07:46 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677666; cv=none; b=VCcehzMLx8ss+HcMAyvHJs0tgV14p9S6l4m8aZpsSxXqJD2cNqB0vIC6GDeWKJNc7i5fPX5c75YFgByPjHeLw2ssXFOyaBpJvD5xP7OMLm1LOy/YmMyXChF9wfNzUnU7MeYgA8rKIaXjjJgIbRsxGQ6OegtG+X046PyOdwIjlTjGXXyTzZMb+baUbZ7B8E6cJllVJPKc5WRxn9ZOjXBonDuLgfE5skri05lRkQ9KZt9WKitPgjf6j3zbadfT4MUkgqrqzqZTFkWy5W7ObHp3P2CEIeWDyomwxHKo6ND9SYfhRLVjb9GE5NB2ZuHgtbYWy4TsLRVxxQL87MrarDVtTQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677666; c=relaxed/relaxed; bh=Bn/gOs/8i5hYCJdtJbGnpMg4EbzU/bTW8mznM7ekqW8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NUHXNH00BR0KfNJ+tf2cZjXiZ59aFGNCcJvXRLL/sABH/GmMzYxSQizE8fDDVL///uuZK/AutQfSbDo4eAhjuiZNP0j21J8+rvBw3Jrp0FwupNQK0RZ8Bp9WILf3EKT2f/pktaodO/QmTcjlRAlJIr298q6eOKLAqlrnNTS8lYVjJIHIxoJ9GKmu68aFKzwik843eojwfMZ4gAvt1NC9iqSSAj6hAKMCj3x1S3nZD8f0oa0ZdNeDbHTgX+iHLpas0V/DFQb3nSIRL82nqVnFWbvwtN52Jz6WJEhBxWwh8bIJpgNTtJzSdhh9lazi1tTdelD65o1H9zohQbnnOxuhQQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=PiEcvlXA; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=PiEcvlXA; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFtzk1YWcz2xSb for ; Wed, 13 May 2026 23:07:46 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2b9fcf7c91bso67430585ad.0 for ; Wed, 13 May 2026 06:07:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677664; x=1779282464; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Bn/gOs/8i5hYCJdtJbGnpMg4EbzU/bTW8mznM7ekqW8=; b=PiEcvlXALLFUcvqNzp90I1u0wOcGjPdF5v0js4mTdGti512Tx1J0A3faArhtoLXYfT bs5J0/voRvOsYrS3adaMJ3BC5qApb8ieSOqu73WC9JNrr/kTcLkzCySMKD///5d5wRDN 3clJZF3ZiqJwe8Hv2g1xOd4tcsH2WaXlIEVbwUWtGx5cHL03aXjK+rvQ0rz+jJSaB769 QQtr9ZRKZFMxTyFXNCAaos3Rs1WtgvofxVdPT2i4DAMioNuQqH6iQxoRvj1tF5PmoNXq CvKHUtJaOWc0xRnoXo83OORhw910o6MWuMWpzN+5UN29lnQOGr7T3Acm4tQK8NhIpVHu ZzdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677664; x=1779282464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Bn/gOs/8i5hYCJdtJbGnpMg4EbzU/bTW8mznM7ekqW8=; b=E16frLfBMGk2oxeNDs+Qfa5TpdWiIOeF0XEwPqUsJTHHHK+oR13fQVnOknEyVIooRl 8FdCkfFGYsoU9g4rT8KwV8wyzcTeDcs7PvJ/uFN0jP4dGkmg6unkmaTor1nxd0bbAAry zjwnB1h2uHh6nR70A7Huv4E1Pw7r6/NOwz7uvQN2WRDGxEvW9U7EZbBvWG1o1FHcFgOw N9Xe3JY3LjWfflCvDxyxLMhyN/3JM3KoHBgldrTeH0eANzWJE4JB7GzKOB2JOisv5B0i TBU+uYrzZhkPvnAcw9YYEJOoaWKIDkuf/zNcQSV/ujKfOlQ8rT1b1ndW0F374CZTFPH0 /8Qw== X-Forwarded-Encrypted: i=1; AFNElJ/mIIChubq69ldO9r/HnA/gUTvtvaAqA6Min0juyO6kKdjSIqvEyx8kOLkZjj57y7/12rUWuJ17HDN3xmc=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwcxmKlDg1d/mhtaMuBihobHGkWXBu1Lygyuljqq/HsQvnMKHQa Td1UeGvHL1KeNgNNBKAGi1hObs5E1yA4Plx2wLASZgVDkunMZpVRzPsE/0/Q19bnChk= X-Gm-Gg: Acq92OH86qqNaAaneI1+PuYLwHG18eU1jW1xAfVUKaXysD4PWT7vYS4NArMk6OGyP83 o17L3EPZiTx57/BE3ijAdmgb3ekHDaLuJW5DdoJBZ7FUUSY8JsX44EIjptDRSnTu6LGKDCcLOmB IenTUjWvFfX7hA+zSmki7ZyzX8SvCyMvBP080F8e489rWa246oUb/47MJQ1Xcn/YpIuXdCOg36K E24KCB0L8qrNFB/+lG0A3u+8FDB+CYLyu1Z+SbILoBLrffknPyTKVGAtCynsNKfExDijniLpwf9 KYjozRCzcVbhplm6ivvKyLnOVhogxf7x7wUNDPsXlSI91KiEGDJJ1QjCl5m+iXDHyDGEbK9Ki7R +EsSYGjQcXv4dARtqgbpRf1kB87VRCL/TPeafcBOkvu8//wOqMl18kKUow5anVsoGgjR/msnVAM Btn5Mw3XaWxHaZhiMxjlNYTaXVVWsy6wOUnQ57Bn6qgGnMF8LUsyQj3Xzf6AA= X-Received: by 2002:a17:903:bcb:b0:2b9:cd2d:6f14 with SMTP id d9443c01a7336-2bd2f4f740fmr22511335ad.2.1778677663953; Wed, 13 May 2026 06:07:43 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.07.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:07:43 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 02/69] mm/hugetlb_vmemmap: Fix __hugetlb_vmemmap_optimize_folios() Date: Wed, 13 May 2026 21:04:30 +0800 Message-ID: <20260513130542.35604-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org __hugetlb_vmemmap_optimize_folios() uses incorrect arguments when handling bootmem HugeTLB folios. The section number passed to register_page_bootmem_memmap() is derived from the vmemmap virtual address of folio->page instead of the folio PFN, so the bootmem memmap metadata can be registered against the wrong section. The helper is also given HUGETLB_VMEMMAP_RESERVE_SIZE even though it expects a page count, not a size in bytes. In addition, the write-protect range is based on pages_per_huge_page(h), which does not cover the full HugeTLB vmemmap area and can leave part of the shared tail vmemmap mapping writable. Fix the section lookup to use folio_pfn(folio), use HUGETLB_VMEMMAP_RESERVE_PAGES when registering the reserved memmap pages, and use hugetlb_vmemmap_size(h) for the write-protect range. Fixes: 752fe17af693 ("mm/hugetlb: add pre-HVO framework") Signed-off-by: Muchun Song Acked-by: Oscar Salvador --- mm/hugetlb_vmemmap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 62e61af18c9a..4f58cd940f61 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -635,12 +635,12 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, * mirrored tail page structs RO. */ spfn = (unsigned long)&folio->page; - epfn = spfn + pages_per_huge_page(h); + epfn = spfn + hugetlb_vmemmap_size(h); vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), HUGETLB_VMEMMAP_RESERVE_SIZE); - register_page_bootmem_memmap(pfn_to_section_nr(spfn), + register_page_bootmem_memmap(pfn_to_section_nr(folio_pfn(folio)), &folio->page, - HUGETLB_VMEMMAP_RESERVE_SIZE); + HUGETLB_VMEMMAP_RESERVE_PAGES); continue; } From patchwork Wed May 13 13:04:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237817 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TI4mI4Pg; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20852-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv046bWhz1y5L for ; Wed, 13 May 2026 23:08:04 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFtzr4Njyz2xpp; Wed, 13 May 2026 23:07:52 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1034" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677672; cv=none; b=KUMAtpxxUyLLwo/u9s4jgyalL6uQqxKE/TFjtpoLBO0Oyvh5DahBlmqdTEbahA6nNLJdwLBCX5OgsDjrMevzr7feWl8VCYLFt2bR0RJoLVNLmjrCMq5zahXNeftVc0aR3y3tjdzb6zNRjnqd0kgXhTnIn97gWy+N5QdaqEUtt8sQZsvTjGRbsnwAUfNk0S74VbRENcUzF2L3v3cbGB6Oyni/KEJqA007/f3z3XpXoWPG6J85Se+2JmEro0BDgbJz4bedwiTVUshXBcXxX4AObKZlgaESyXMUdlhzpIkECZFyAEKmicUvlInmMtb7WxQ4Bbw9F3Hw58f7xC/FxT6P6A== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677672; c=relaxed/relaxed; bh=W+0ze7EIzQ8zDlwa2Xdpv+yLCfHzqyRtA/0pp8bbhmY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nIhWPvQ0ROJS3GkCCPRm75b5PAmwHfztd9HioeCPJ0oiAdSDDn7cUKhEtxiCKhLtHnYam5+29DmjgiroZINNmuvFSUkLLpFNjpCGenY7PCCpJqsSy5VJdOQOPsz+RiUq+p1jbDTJuO/eKZA1hoG26REzVM+hvAMbiy2HBJ8JBW8mo09GQyga/Y5im8DNb03YOrNUTg3HNwnhKnw6/USUPLZ4qIYP8H6o6m9wKaH2gUrKEtiwaDgjomMOhLY2tlw8L9f4mXeGhX8/XEuxYzFojFY2p8w3ImOF6m3YF+aShNDKApQgsHsEq20IkEzLZMupBaW75DFOsCC4nxvYQqJzmA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TI4mI4Pg; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TI4mI4Pg; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFtzq6BRmz2xSb for ; Wed, 13 May 2026 23:07:51 +1000 (AEST) Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-36900945df5so330561a91.0 for ; Wed, 13 May 2026 06:07:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677670; x=1779282470; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W+0ze7EIzQ8zDlwa2Xdpv+yLCfHzqyRtA/0pp8bbhmY=; b=TI4mI4PgWcCgWsFxubc33jt4smjGxm3ac7cpXI5kSPDz2Wr/Sj3/l0iCgGFzwA0RiY RQowrV5ZnRFXXy/EdC8d2s2t58B3nr2FzumSR+dCxK167fXhJSK4qZhlaFSHMVOLQyyZ Wz+w0THUoamEy2gvH6U9xjTfzw3VHeDWJpbUDM61S90YqUTEqY5q0/kGM8B8NU13NEjo tg77+EhQjhFA4vTsHRNgy1UX3c6GFK8VogjwscRv38TUx3iplRqmkodNrFq0v6yOhoX+ yyRq+gBD/sqfhLvaRWdOC+yshCxtpNig4vMz+2R/8bif/K7TlP07o7bOFobmxVeJUDZy o4oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677670; x=1779282470; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=W+0ze7EIzQ8zDlwa2Xdpv+yLCfHzqyRtA/0pp8bbhmY=; b=NJhPexRZ3LnM6/2tjcQpo0frX6HDwoEFH20OvzBUe3fHcKw2SauZs0dhzKNQRkzh0S Mh6+2x4wykkIDFaj0hBlYsQJTMGGvLCNoNFOhVTPH3G2lv+SY3yHFwSDSoLelQ7kQROi B+pZB0cgJbKLbHgFF0MZQVwK/yK2EFpl+zqU0MCD4R3yk1qJroAnVoy6K00rRtJLlYTT Tcsm0yiQcqACtQT0zXSL4ClKHzdKOLMpq0B0s1UFZmmtWmGKQwmKxrBZJg+28AZFazSh 3CTuT5EE/JRRICPuTWW7A0WUI88HNoRo9ymRnT3iyffoFeVv7w+MzquldVQP8zkwCBe9 JPeA== X-Forwarded-Encrypted: i=1; AFNElJ9BTTIpss3EPGxtFcX1XF+5MiZHfw+330HR8mCEc3LyfKl2kNUIvp1tlW9+PypPDcpIAAfPGDLmqTB7lAA=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yw71wt1/DKWhV/rNrIMVU6c6mHdq4sgol84qoCufTGQrcZPJrf6 bmsr0gwStVDAe2fHMRkf5fhRAAgZOZB86a2ofvWUTcmBuLyhgggUxcOObkpAZBR7DYA= X-Gm-Gg: Acq92OGFjfhl1JyLQw/N6VwQxSuh1NOdDYIF1T1iiPLo1z1S7sr500BZCs6MYvlvWyw zdGpo76JG7gt4hKncT0AX5pCFWc2AZpZjZRnKMQMM62OKb6GIGBv1X/+aK+w9IZ4JHM6U0qg5Xs QjFd/7YcBmVh7f2Ql5Zu0EEVuiilPDY5qPRK7DT1RSGKgDWph8nH4C1lJjaaYxMO4tvF/u/ynWY AZh0MGmYfTTZPC9//M0buLSqaQg5CUONCKrDOx6wiL4nLJowUfrsrgm+/cFxRnr96katVSJEXz6 O4KKEuR3PgZIcmsbwAeghTzofkFjVV0G736+j2WpgAHGkA+hGYYdLOzm28r856k50fKIOFd49cn N4dYUj3iIlZ2c6XuAyKqk+kL6VPyJ4uN4aMh6+vLQG8WUp4BikKTV6c8p/UahyKdRBTjuaPAQGA RChg6WSO+l+VqDmRlAh3opMVEuE05S2ROSiLX/TUIhn+aEIV9gctVGjyXMlhc6mpdMqzMCJg== X-Received: by 2002:a17:90b:3f8c:b0:368:ed92:6f5 with SMTP id 98e67ed59e1d1-368f77f6980mr2850344a91.4.1778677669469; Wed, 13 May 2026 06:07:49 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.07.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:07:49 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 03/69] powerpc/mm: Fix wrong addr_pfn tracking in compound vmemmap population Date: Wed, 13 May 2026 21:04:31 +0800 Message-ID: <20260513130542.35604-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org vmemmap_populate_compound_pages() uses addr_pfn to determine the PFN offset within a compound page and to decide whether the current vmemmap slot should be populated as a head page mapping or should reuse a tail page mapping. However, addr_pfn is advanced manually in parallel with addr. The loop itself progresses in vmemmap address space, so each PAGE_SIZE step in addr covers PAGE_SIZE / sizeof(struct page) struct page slots. Since addr_pfn is compared against nr_pages in data-PFN units, it should advance by the same number of PFNs. The existing manual increments do not match that and therefore do not reliably track the PFN corresponding to the current addr. As a result, pfn_offset can be computed from the wrong PFN and the code can make the head/tail decision for the wrong compound-page position. Fix this by deriving addr_pfn directly from the current vmemmap address instead of carrying it as loop state. Fixes: f2b79c0d7968 ("powerpc/book3s64/radix: add support for vmemmap optimization for radix") Signed-off-by: Muchun Song Acked-by: Oscar Salvador --- arch/powerpc/mm/book3s64/radix_pgtable.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 10aced261cff..cf692b2b5f7b 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1314,7 +1314,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * covering out both edges. */ unsigned long addr; - unsigned long addr_pfn = start_pfn; unsigned long next; pgd_t *pgd; p4d_t *p4d; @@ -1335,7 +1334,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, if (pmd_leaf(READ_ONCE(*pmd))) { /* existing huge mapping. Skip the range */ - addr_pfn += (PMD_SIZE >> PAGE_SHIFT); next = pmd_addr_end(addr, end); continue; } @@ -1348,11 +1346,11 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * page whose VMEMMAP_RESERVE_NR pages were mapped and * this request fall in those pages. */ - addr_pfn += 1; next = addr + PAGE_SIZE; continue; } else { unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); + unsigned long addr_pfn = page_to_pfn((struct page *)addr); unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); pte_t *tail_page_pte; @@ -1376,7 +1374,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, if (!pte) return -ENOMEM; - addr_pfn += 2; next = addr + 2 * PAGE_SIZE; continue; } @@ -1392,7 +1389,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - addr_pfn += 1; next = addr + PAGE_SIZE; continue; } @@ -1402,7 +1398,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - addr_pfn += 1; next = addr + PAGE_SIZE; continue; } From patchwork Wed May 13 13:04:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237818 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=N7XoMOxl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20853-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1 raw public key) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0C4zSsz1y5L for ; Wed, 13 May 2026 23:08:11 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFtzx63yjz2xqn; Wed, 13 May 2026 23:07:57 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677677; cv=none; b=Qyd4EQ/xD3E3wEe4TPPZq1E26gfy/chVhZOMGwLzoHEyCUKsLMT06quortkLARp3SSIsVM1oiBMUdbsxdtHRTv6gixqlVa3IdMi9ytLd/Dte4ppcdQmaJdaK3N+7suxfhL66tsQE5e0pDkbNv2348nlDDOFPxuhc1iVockwjeV/zbr9+umvKi4t+/C04+Ax2B9XjjFawxgf5pQRxJkPrvBKgW5nvRYaxwc6WrD8ZhSK7zMdiv6/UG86BGuQU0d3oJHkJKQxbkttoKldrmC5T7isDhBHdhQQHFN+FGEOUyenfHLi5Zy7ve5LynkHnSkHMYxtgzdCp8dm1yWA4LoMugA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677677; c=relaxed/relaxed; bh=vB6yS1z5EFpHn1m3zCB1eARc99Wr/QTdVrJKUDK61SU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mHqIuL1jRgdDAcgmevnkU+dCodqHJ6b14KHQROxBJewlS+UmP68QeeOwAx7VtiyoNeCzbNvr24KbiaYZI+7HZQKTaaziK7dOW03wCXaMjKOF2QmxPKr/4sYyGuxhI8vgG7aXRAYWhX4GraKOFB1EVvUYBHG8BFtamGYuh1KBwwjIawHYCOO0fP1/2+XvdAVMQSohEdzD9mXhUonTtLMPrH90JzjQga6CLS81b6km/nGq5cBKwhkmxJvnony3G94crll7U4RM4SyY3k3XYQ28v6Eg/2r40nm9tl20d1NQfkTMZE1zsiHJMXscYy8YySrMbGLKK5YVgkhEPOWo4yOL6g== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=N7XoMOxl; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=N7XoMOxl; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFtzx0sqWz2xSb for ; Wed, 13 May 2026 23:07:57 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2bcd3ac3307so19192395ad.0 for ; Wed, 13 May 2026 06:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677675; x=1779282475; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vB6yS1z5EFpHn1m3zCB1eARc99Wr/QTdVrJKUDK61SU=; b=N7XoMOxl19CKYAbZgplHleDwtmD4JThiuujr2abfe0YN6/g+vhR3IM4sYP8eNwWFKh XlF2PF2XQGzT87ySppNWAEnQUKTuVVyVwrVFfaM7Nnmqta+iZIfTabpWybygJiTGLen4 6vmGHCTvlRO6iC3Qi0mwVIUUVsGzyNeMPvG7lGR5SDWO3Tum7CjjIY0zkOTVcS0R5jLF UyQRTVpit9BiStZTkCZEv709YUhqz1Fvauo2AJwSzg1LKXEKc4Ii5XkpzgXyZIlkcr0A ur0slvVDbQ5r0xxtP7O8Hu7TkNcuC4Q2mEk9RzlkH43e6lmvVOLVpG2UPE1QFmx2VnOO 85NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677675; x=1779282475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=vB6yS1z5EFpHn1m3zCB1eARc99Wr/QTdVrJKUDK61SU=; b=TEtnxzLwRIUENu8nszDSMUtedpnl5H8cmZyrddwQR3zzAKQ66bOIQXvpVH2V5zr4eJ 3qI0aR/u8g+VCwmqFGN6YMFX+w5VHjV8qO8Qi9MzHxOItqeylr5KKuG43zs8GnVnZ0na SsXV+uXcxYGSGaMP+GzLcX7wl7W42M/DX2hSYIhwXdbyb+JlJIvnyX11G+do5kB9kPZt OKNblnGGsHy+GB1KCXCPxlhYkgORUAthkLLgtMOlE/UlWvP86p0+Fv1KxOzsyDQT4TBU ebwfjE1gJSmgiTfuQan5zL43J5t5iAuS5FBhmur+ZqKiBBdUhiLjQ56Az1txc+/lC4fw gI0Q== X-Forwarded-Encrypted: i=1; AFNElJ8xCCEiF/RMVNBmaKcOq8guXGkab+eIJAt5fghEN7zRrm1zYY0yRErYjk45/SBxre3ROLcrMnl3W7l+l7U=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx46ET78x3xJy5WqNYlpodUnnM6LdRdBWvnoC5iNidTKJ03x+z+ jgXtOTFJev3Pj8Kn7ViY+Dla38H3TL2W6BLOhdyAFxcjcaVFrPzLbWf6kOLLTaVe22U= X-Gm-Gg: Acq92OEuN1Na2spACwDwQoQ3/z24QstVMY9gKMai9pQ0FFSPSaY+0Uw36nd89ALUDMZ Rr3OrfaBwb7kZW/SHt39Z2rvekADydal2BW3DOQrPN4zxX5TL8dZFPfBuSfms8KwKuBy00ShFE2 QRp3qdYuJElGbeSy7ArprHY2Ijs/sYiybZDN8vtcYNkp0X+J7CBW0JJOk5G6QtZJvXwfm8XvcdQ 07Myf9E/r7YQEZRM+QQLBZ3mjyLeU9eWw00ahBW7XfICr8z5YCkzOaWAN0BQBynzLHUqqGtHR8G JztW0LpiPI6usyZ10VVCbSeYk55JoMj/Mz9zmKw7w3nKAB7y1JiX65Ic76aK9kG80nLKvBLcQ2O G0noCcdWRaH/XblEf75CaroRUHyRNqZOxV77691AijMAY2NDsJ0cVD8TP/FHi+yqir5PCp6d0x0 dc0tJly+jSYUyEVK2N6gyKhAvMGkE56w1Z7wDwefVdRMJrVNNmelHudpwEjLY= X-Received: by 2002:a17:903:f8e:b0:2b2:ebed:7af5 with SMTP id d9443c01a7336-2bd271773f9mr36693205ad.13.1778677674866; Wed, 13 May 2026 06:07:54 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.07.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:07:54 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 04/69] mm/hugetlb: Initialize gigantic bootmem hugepage struct pages earlier Date: Wed, 13 May 2026 21:04:32 +0800 Message-ID: <20260513130542.35604-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Gigantic bootmem HugeTLB pages are currently initialized from hugetlb_init(), but page_alloc_init_late() runs earlier and walks pageblocks to determine zone contiguity. If a bootmem HugeTLB region is marked noinit, set_zone_contiguous() can observe still-uninitialized struct pages through __pageblock_pfn_to_page(). This may not trigger an immediate failure, but it can make set_zone_contiguous() compute the wrong zone contiguity state. If extra poisoned-page checks are added in this path, such as PF_POISONED_CHECK() in page_zone_id(), it can also trigger an early boot panic. Initialize gigantic bootmem HugeTLB struct pages from page_alloc_init_late(), before zone contiguity is evaluated, so later page allocator setup only sees valid struct page state. This also makes the initialization order more natural, as struct pages should be initialized before later code inspects them. Fixes: fde1c4ecf916 ("mm: hugetlb: skip initialization of gigantic tail struct pages if freed by HVO") Signed-off-by: Muchun Song Acked-by: Oscar Salvador Acked-by: Mike Rapoport (Microsoft) --- include/linux/hugetlb.h | 5 +++++ mm/hugetlb.c | 3 +-- mm/mm_init.c | 1 + 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 93418625d3c5..52a2c30f866c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -173,6 +173,7 @@ extern int movable_gigantic_pages __read_mostly; extern int sysctl_hugetlb_shm_group __read_mostly; extern struct list_head huge_boot_pages[MAX_NUMNODES]; +void hugetlb_struct_page_init(void); void hugetlb_bootmem_alloc(void); extern nodemask_t hugetlb_bootmem_nodes; void hugetlb_bootmem_set_nodes(void); @@ -1307,6 +1308,10 @@ static inline bool hugetlbfs_pagecache_present( static inline void hugetlb_bootmem_alloc(void) { } + +static inline void hugetlb_struct_page_init(void) +{ +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d22683ab30a1..b4999653a156 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3370,7 +3370,7 @@ static void __init gather_bootmem_prealloc_parallel(unsigned long start, gather_bootmem_prealloc_node(nid); } -static void __init gather_bootmem_prealloc(void) +void __init hugetlb_struct_page_init(void) { struct padata_mt_job job = { .thread_fn = gather_bootmem_prealloc_parallel, @@ -4163,7 +4163,6 @@ static int __init hugetlb_init(void) } hugetlb_init_hstates(); - gather_bootmem_prealloc(); report_hugepages(); hugetlb_sysfs_init(); diff --git a/mm/mm_init.c b/mm/mm_init.c index fde49f7bba6c..5a910cc5534c 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2335,6 +2335,7 @@ void __init page_alloc_init_late(void) /* Reinit limits that are based on free pages after the kernel is up */ files_maxfiles_init(); #endif + hugetlb_struct_page_init(); /* Accounting of total+free memory is stable at this point. */ mem_init_print_info(); From patchwork Wed May 13 13:04:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ABD90Red; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20854-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0M3Qf2z1y5L for ; Wed, 13 May 2026 23:08:19 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv032bCjz2yCL; Wed, 13 May 2026 23:08:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::630" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677683; cv=none; b=XaLE0CeYzRyB0QyL3Q/BWwPHsPbg3gKHJsW0xtmcrMTvq9TVpB9Pd5bKRHOa0SWb5z0wY/fXjF5LVjJbfhosMA1/uSPETS9mu3sBh00g3L4kWZWNdjV+cXyjNhCgvXfATjqGALoE9HYFchvIk0Z4jWYjPg3DKEIOLsAc43Jz3ZseUhfMUFbYKxi6XsRpeat/I48yLXiks5cp6hjn8Ew2+H2MOqsVJ46Yjj5PsH/bLxk/ugbrvshfyeUpQEHFAr5ftjMpbnjLTa1QnzhWGgmUFVOA/Rp1uMfC8w/cyv/WR31DW3fy1D2gpMiB62VrY44WCeCgq3qDAydVKUHYZU/Jew== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677683; c=relaxed/relaxed; bh=UmVa4U0u+W/oPX/GlMpJ9lsDnm/CfX2AcDL6FiyA9T4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AqLflOF4AQIEnjshW0waO47XWbSXIV+4Q6O5zKDZSWHZORssvrg+BuRQJFRqRg1MJNL5UyKS9xYFrfbrE6wOIr4fh00mJBPY5l+vzX/j9xIg6GtpPTcimZ0Ngs2KHhih3OFOhxu7jPJ0dLm6i1bNhWVHBr210mxVI73msStXJWWOYP5gPhrOhApbe1QZIUMir9aQx/lhOwKiHlgpuPzTDfBZB7sYn9qbKGzrQh5htrttAGcPBZ7o54uevDV8KZ4Qwv01qcfhU7efAG+auptmaVy3u9tNDjzGIfrRhY6/uwC+YZvxHd3fU6rRsShJ5iv5BHdOxcLGzremTkUV/y4hTw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ABD90Red; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ABD90Red; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv024KvXz2xSb for ; Wed, 13 May 2026 23:08:02 +1000 (AEST) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-2bc763e2ba8so21640955ad.3 for ; Wed, 13 May 2026 06:08:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677680; x=1779282480; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UmVa4U0u+W/oPX/GlMpJ9lsDnm/CfX2AcDL6FiyA9T4=; b=ABD90Redeaz8D5RIoPoEDkqjSgG74+AZ7ExRH8akKTRd0TReDMUq9Cc9nEm7X6THRl vTlNt7C0Gk5q1uaSPZ0fVV3DqpN1dgRRaD6qoPtPLYWw6QnSvoE2tcnNF3fMNmVGMnqR X4jMChk6Jnw/Q5y2DKHGzn6rdnQMwjytKrq/dq5gxYPaoKAT6ISWZ25mroSIoIN43kCT RKOidYjSHOwkocX52eEKGZl4BUL3IpDrkH+3CEolX1y+8kITIfUo7PzfTHTv8iQ6mcXj bhRN7VSrOa9faHv1nmvwWEMolAoLPto2qXmWJOS8cRW1BoT/VZt00QiLGAwIAnl0tFmV 323Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677680; x=1779282480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UmVa4U0u+W/oPX/GlMpJ9lsDnm/CfX2AcDL6FiyA9T4=; b=i0fWM95vJHWFvkvuoHX+KV3eTECuNYqZAT5rOaTqBF7bgOQC3eaTEzFCgQNWZUBNZI yUCsebGH3oRfs6ufo9zOBTODZEVl0dOlXCG3QMBKDb1donElV0HylG+neO+8chhzGB3t 32sah27KQiZ+nzZOg0pTJARVWeNX1GAQ71rRvZajvZJ2tS1UqSdfpVH28GZ8dWzOfEvW WrVjniPOnri0m1PXYBZvAUsAAcvE8at5ikfkFwJ0GE9V1AO2aspiKq1VjW+IwWjj2Cbl Y5XbKVUWdmxtqahYd4pEa8w95Ga4QGlDqd/Xwzne8ateKgmblohqaOYtutW9UTqPE5aF SrKQ== X-Forwarded-Encrypted: i=1; AFNElJ8Kljcyl1P+ITkQfw0bFSlQHPbNER8tud24R2aBMBGIAZcdPW+yyxDbOuLJdp33wliYWhQ/QZhGYj/Qvig=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwLmQCe5sfOG4tmpiQbj1KTh+9w9hWKg3Yq+g+k54N32RzpKEWY w/3YO8nS/Vak1ExWtzHO7N0juthpe0StzzCRCrIAh3MZvzDruhU2gAJUPj0Q9ti6IR0= X-Gm-Gg: Acq92OHkitgJZaKJiN0AS6e1eDxthTodZ/P7N5Rwq1QkHAdXRkCpO2gk6Q+tia5oPqR JXHVMjXZBsbOe0NGpaoNnEOO5StaBfFX0Jn0yl+vPc41HJUGH8b3LJ+uzrgLhDGbCF7u+GVNnRu XjLtbrFNZzhWpE/EEnz+F0zfRLGqgzHws8J/CaRjeZ9gl+dZgbwQAVthv8AQ0nZl5cwh2r4o1Qu phD/xDqrkn66TOoTdqEem5rmnMAS8v0JQMIIYxPmouIhSpeNbd33hAUmaTFruDV+GbaAOqRETPK AdVtm94sEL4aR6Mw0T1aKv1rd5IjplsdEqbpOsuAvGhA4k43KkwLIBfVF7GsGpxBcgdhfp1Uqjo Qb2BgcHH2Ftbb3kyI0aMpk7l5hn+uYZO2/2Ue/cDwdybIO9fzXDiLwxPJ7sd1i6A7N73IuISvQ4 6n3CyFfCn0exir9XMGFQ+i9CHdFxfDllJKWBYhMLkTywJ3ZT9+bATJOqJQG00= X-Received: by 2002:a17:903:94d:b0:2b9:f59e:aca5 with SMTP id d9443c01a7336-2bd27702f8dmr38772485ad.39.1778677680317; Wed, 13 May 2026 06:08:00 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.07.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:07:59 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 05/69] mm/mm_init: Simplify deferred_free_pages() migratetype init Date: Wed, 13 May 2026 21:04:33 +0800 Message-ID: <20260513130542.35604-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org deferred_free_pages() open-codes two loops to initialize the pageblock migratetype for a range of pages. Replace them with pageblock_migratetype_init_range() to remove the duplication and make the code clearer (Note that deferred_free_pages() may be called from atomic context). Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador --- v1->v2: - Add Acked-by from Mike Rapoport --- mm/mm_init.c | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 5a910cc5534c..96e0f2d8c3ea 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -674,15 +674,15 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ -#ifdef CONFIG_ZONE_DEVICE +#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INIT) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype) + unsigned long nr_pages, int migratetype, bool atomic) { const unsigned long end = pfn + nr_pages; for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) { init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false); - if (IS_ALIGNED(pfn, PAGES_PER_SECTION)) + if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) cond_resched(); } } @@ -1142,7 +1142,7 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false); pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1993,12 +1993,12 @@ static void __init deferred_free_pages(unsigned long pfn, if (!nr_pages) return; + pageblock_migratetype_init_range(pfn, nr_pages, mt, true); + page = pfn_to_page(pfn); /* Free a large naturally-aligned chunk if possible */ if (nr_pages == MAX_ORDER_NR_PAGES && IS_MAX_ORDER_ALIGNED(pfn)) { - for (i = 0; i < nr_pages; i += pageblock_nr_pages) - init_pageblock_migratetype(page + i, mt, false); __free_pages_core(page, MAX_PAGE_ORDER, MEMINIT_EARLY); return; } @@ -2006,11 +2006,8 @@ static void __init deferred_free_pages(unsigned long pfn, /* Accept chunks smaller than MAX_PAGE_ORDER upfront */ accept_memory(PFN_PHYS(pfn), nr_pages * PAGE_SIZE); - for (i = 0; i < nr_pages; i++, page++, pfn++) { - if (pageblock_aligned(pfn)) - init_pageblock_migratetype(page, mt, false); - __free_pages_core(page, 0, MEMINIT_EARLY); - } + for (i = 0; i < nr_pages; i++) + __free_pages_core(page + i, 0, MEMINIT_EARLY); } /* Completion tracking for deferred_init_memmap() threads */ From patchwork Wed May 13 13:04:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237820 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=flKYATqE; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20855-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0W0g5mz1y5L for ; Wed, 13 May 2026 23:08:27 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv08648Zz2ygL; Wed, 13 May 2026 23:08:08 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677688; cv=none; b=H6iNUqfSYvFWeaA2KywecUPG7Uq1q9p2jwC6HfgHmq3A0Aua0552UyRma35XadxwCF60V8KVlz+keV9N/mpVF9N7ivmhL83T59VgG0IwplAoPdI4IrNkYk+6V4DI71Na3w8KggDBYP9zvVLeuwvMhoU1wdTF+b6pRZW5Bg1gygvJF6kjiyEdt9rGkVMtoNtPQy1u12eQw7hsusAFs8JUkKXK69FG6ce+VHxS7I5v/9F0txwXHhWdef8sRj7YGevS+mWRblwR4v8i1pHmGUlp3W9tlQC+AjkiRwpyBZWZWA5HzPBriYnPPK2VNt7j/Q9ZkUZtcx6SZzKazcptrdyQAg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677688; c=relaxed/relaxed; bh=4FGAY5sLEWcL6nfZSRVARbstApRC9v1d8J6Ir2cANyU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V5hiHdvakgIywwJbEmq9vhga8+XUf58pIUOTCtObOlX8JiQvX3JtYqJ5P880C8CqB6JLFf1L8jP9XFgt6ICg1dDFHbuJ7P0BcdPC938VrPq3Cd3Qdd0jRzEyUaDHa3eMspEiNPrBjpcLjgYzZf9TWU3kTSlmuBBjWdJQ8TiPvBmVkL9dEPAHxyyMOxgbdK4J1GYXCeikPIbxWVwKcu6xjfWU3tgYMyj5j04UZMNRE7+taez9CS51Olwqi0V263/EshUS3m7CYzdcsbRLZwP906Qf2V0Q6/soeUU7fEJyxrMb51pXHS7PWFKK67CFCYdxhJVWLmIDirziGLa5w5lSsQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=flKYATqE; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=flKYATqE; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0800Kmz2xSb for ; Wed, 13 May 2026 23:08:07 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2bc85eda6b6so20766745ad.1 for ; Wed, 13 May 2026 06:08:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677686; x=1779282486; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4FGAY5sLEWcL6nfZSRVARbstApRC9v1d8J6Ir2cANyU=; b=flKYATqExUc4VZXroNw6x7AtkWcP3hv6SraXTtup6u24Wj55d7Wp+VP4VJMJr3zzOD gkmFTalonCdkC/XBbx6vMJ589XU6tstWTrMh0RbfrMs5uiXRO6sxqrRwuLGfmgTT/xO/ aDiF6Od/ets4gJr2exxi5eWEAcdvHZK6nXezcYRxYlYZBzs6RJ7xG68BV6x5LeTORW3P iHVaKP8iYyCCm8OqocAVTsIWKMaXa29lgSpr8wLIGPrz8V7DYVJm0ts+mQjW25/5x8mP 5xX7dTXtJI4T6IHyYHob4GjtV82H4uMzPD8mOCvbjSbJrqqCf7jrA5Wp0rAO/F4oOS7Z e9fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677686; x=1779282486; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4FGAY5sLEWcL6nfZSRVARbstApRC9v1d8J6Ir2cANyU=; b=McTk+EDD3sfjDkfEdkA17udaecvs5gaYbkfxkF6Fziu6/KWw6l/x4lG0v8T7p+vgo6 VNun4D7l7Bx0FbV2AXV+Ydn0pK/6pAeqO1qCG5W7LWIP784N9GEGJecMZcCtYIkOgcO4 rVjtTLb94837vEIIKdke/nOgzwieYoY+4EVaAC1TIWeCm0Z8qazQQfKLpWfZNFtdwqr7 LxSCxYE+0oDftjp8xdYSO9NKTSQOw2TLEOTKRI1ZdY99HyxbBiHEIStzfy/JQZ3EuTRL MOpxLd0qWVWaHy2F+qnpl2S0hTByMlDT22Tb4q4oZ3qA1OF8Dd0qjP1GkUG2/lISyl+r 6GQQ== X-Forwarded-Encrypted: i=1; AFNElJ81KAuMsXCNiQxHSQT7AmhHz3BJwuiD88L/Q+0VCNYo0VpNfkjjMBmX7ADaprPCzXYpnV6hpQfHXjIIUWA=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzFF1HeG2qaVgi7n6Afpp+XCqbvcEKvTzQgalZWfiLeZxGP+C9r AAPGiue9vShUnDWVtQ+J3uFbV0byhk06KbSqNviHRl6G0p+5KPgGYZKusMNayn9BK6U= X-Gm-Gg: Acq92OHp+ek7lCg+tLLOxK3FoYt0EfjDPfocJ54AeuVlsAo/FqMMkR2WkGuRiY99m5r nK1Jp+fHanv1gGc1k1P/ueLPcswItZmrb9IOoOnwmKUIWrHOXRMrZUCNNIwVigtmTHlRWfKK3p2 +e6RZe8E+Hft06LMHJrxH42cF2Arl0bXhfbD/cr+QW/5ksD0uW0vZmS4t/1dw9jx917w6XKCMPO 6EPWDINs0t2XR0huAyN2ZxrvPpLQ/FqCToQAtx4KNBBxnzRun1EM5XTFAi3fOPV04CMCocbSNjA lKKyAX7OjwTnmKgl0utvQN/Pv41byqiRGJnE/c3IEgmnoPcxUwWosxbwR0/84SEiee9fU9cuOQ9 NS91qmv0smPBZXP1/JFImKMA0RTUmG6qJzjeFjgI/DTWyKKGtVluhjLy2YJaD9rIKxhR2I1GqSM D+CzGIbx9bUApcQ0ZphwtPQ/6mcaawBANqG6GMBnAOKsvzRHtVCHMDkr3zkaQ= X-Received: by 2002:a17:903:bca:b0:2b0:ac1e:9737 with SMTP id d9443c01a7336-2bd27121cc3mr28132885ad.12.1778677685927; Wed, 13 May 2026 06:08:05 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:05 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 06/69] mm/sparse: Panic on memmap and usemap allocation failure Date: Wed, 13 May 2026 21:04:34 +0800 Message-ID: <20260513130542.35604-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org When vmemmap or usemap allocation fails, sparse_init_nid() currently marks the section non-present and continues. Later boot-time code can still walk PFNs in that section without checking for this partial setup, which leads to invalid accesses. subsection_map_init() can also touch an unallocated usemap. Auditing and fixing all early PFN walkers for this case is not worth the complexity. These allocation failures are expected to be fatal anyway, and other memory models already treat them that way. Make memmap and usemap allocation failures panic immediately instead of trying to recover and crashing later in less obvious ways. This is also consistent with how other memory model configurations handle memmap allocation failures. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador --- v1->v2: - Add Acked-by from Mike Rapoport - I refrained from adding panic() to memmap_alloc() as it wouldn't simplify the code. However, panic() is still required in sparse_init_nid() because the architecture-specific vmemmap_populate() bypasses memmap_alloc(). --- mm/sparse.c | 44 +++++++++----------------------------------- 1 file changed, 9 insertions(+), 35 deletions(-) diff --git a/mm/sparse.c b/mm/sparse.c index 16ac6df3c89f..c92bbc3f3aa3 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -239,15 +239,8 @@ struct page __init *__populate_section_memmap(unsigned long pfn, struct dev_pagemap *pgmap) { unsigned long size = section_map_size(); - struct page *map; - phys_addr_t addr = __pa(MAX_DMA_ADDRESS); - map = memmap_alloc(size, size, addr, nid, false); - if (!map) - panic("%s: Failed to allocate %lu bytes align=0x%lx nid=%d from=%pa\n", - __func__, size, PAGE_SIZE, nid, &addr); - - return map; + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), nid, false); } #endif /* !CONFIG_SPARSEMEM_VMEMMAP */ @@ -300,17 +293,14 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, unsigned long map_count) { unsigned long pnum; - struct page *map; - struct mem_section *ms; - if (sparse_usage_init(nid, map_count)) { - pr_err("%s: node[%d] usemap allocation failed", __func__, nid); - goto failed; - } + if (sparse_usage_init(nid, map_count)) + panic("Failed to allocate usemap for node %d\n", nid); sparse_vmemmap_init_nid_early(nid); for_each_present_section_nr(pnum_begin, pnum) { + struct mem_section *ms; unsigned long pfn = section_nr_to_pfn(pnum); if (pnum >= pnum_end) @@ -318,34 +308,18 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, ms = __nr_to_section(pnum); if (!preinited_vmemmap_section(ms)) { + struct page *map; + map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) { - pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", - __func__, nid); - pnum_begin = pnum; - sparse_usage_fini(); - goto failed; - } + nid, NULL, NULL); + if (!map) + panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page), PAGE_SIZE)); sparse_init_early_section(nid, map, pnum, 0); } } sparse_usage_fini(); - return; -failed: - /* - * We failed to allocate, mark all the following pnums as not present, - * except the ones already initialized earlier. - */ - for_each_present_section_nr(pnum_begin, pnum) { - if (pnum >= pnum_end) - break; - ms = __nr_to_section(pnum); - if (!preinited_vmemmap_section(ms)) - ms->section_mem_map = 0; - } } /* From patchwork Wed May 13 13:04:35 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237821 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QzacGz+W; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20856-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0f44njz1y5L for ; Wed, 13 May 2026 23:08:34 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0G1knrz2yFd; Wed, 13 May 2026 23:08:14 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677694; cv=none; b=P0843o36pjo83Z9bvYEJHubTsDq8Yf+GMCmYoywNUaHg3o5ynu9/lqrF7dYTHAT8iYvhdwT6EqBi23y0mMOGcE6i+ngxbBU2RhfD9xDKKaZr2Nf+rDL0BGsizxS4w2mnUzWHCiiVRcuamxmrmDqLH1cpu7T5DC7cbXZOAwvO7XqVA7OZvLBEGhs6PPtxKPW/ZfJTxXjDrqjvnBJWiyk3MhKelXco4ImvISAFtbFHORSd3rIvpW2q5N8TfiSx6Yx1J38cDTH1p99fmURBhmBZ7c65boIE8yKz42bgGtz/p0Oe20CeB/l6+/WKiy51eQkoU90yoCyNuZQmpc433gzkYA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677694; c=relaxed/relaxed; bh=5UyK7iOrAjDiLiKTgZmma+Goz9i15/Iarg4XyDf9KyI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QzRgPDwhboxiCl8dVndKHS1bHwZQ32/RDzSAUNA+NHLn7FyFGPLWdsapp5J7zUzs1Yp5yNjg577kcfE4yt0/Q53Z8D9VpVZyFsTf/n083S2aaD/w+pCOfqoI3CK29aKh8PDKR8YST8Fn3UayVxoQJmFFyEd3Al4gEN2FHkbtWqcrF/XSZE0DzfbNFwyCQWo269hpPeeOSyn9eYpvevC6cE0QoJCr1S+CEiVKd9gnwLyRxOov6fuvkNbzityfN4akn4lnFowkkUzZPGjkfNWQltegow+H8uQkLYbBuBR5/f6EQ3u17FFI+DyhWaFSMDFzGyo6+2isJBi4aP0uBp0r2w== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QzacGz+W; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QzacGz+W; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0F3Bbtz2xSb for ; Wed, 13 May 2026 23:08:13 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2bab2548e8bso31477855ad.0 for ; Wed, 13 May 2026 06:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677692; x=1779282492; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5UyK7iOrAjDiLiKTgZmma+Goz9i15/Iarg4XyDf9KyI=; b=QzacGz+W9BjjvDxDXT4a8v4OMyLtPZpud/NwPy4XoyEwmt9pAAsBug5ZfbIJNeA/Vy PwgCJXi8QNtkp2xNElRBctJ6aWemFo3z/oFcYf4q0XhE1tVDhfBZudJoMthM2YVJhREJ TID0yzCNCvjDj1Iz8WtNqfpR1Oi6j8f0sIlXZGuTHmkS/ijGm4S/ucGCzlm24y39vYZO /P6CmaTj0uHcStb2Anms1VjwhAuD33ZC62Jz8bUxqrOYdJ25cmdUzZx3CtCVL2X9MxBP yluVTR1bOsYYbhwtOnqGGLVnSkMFatOx5aMwYRJD3qhH2x6fu4uINAQdYdUy92qgkGWU 3bVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677692; x=1779282492; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5UyK7iOrAjDiLiKTgZmma+Goz9i15/Iarg4XyDf9KyI=; b=Kkhf9eYA6fEHL9KVdOgK9qtUzHf0uxMtRZYD6UxAj2rpPqu6lJaiRn7IpGHxhoTg1O NuFY3gwSvmlaCnM2Aa2WdR0lL4fSerQQAgjvwnpiTJoBTA3AR45JZ07SdFx7D45xBlCo gW1fjjiQHkrs1mKxm6QTRUL1MmQuPLsvbb4hKUWtrfz3e4uRwGV/ZvVcR2iXWHpFOLIS KMZQ6BOAsy8p/fnc85xxgs6milqOYhMBtqGdY++bfLjvt7O2NBgmeefjC3VrlS/jBaOb AgdQQZoh0Bx7YokNCaous1MMelyuwaDg59/vYnsiCZoGQd2ejVp+nhP3lmqxm+n94H2d SXLw== X-Forwarded-Encrypted: i=1; AFNElJ8dsO0xWc50x1OZXu1DuMSpy4gpUzJM2i4PZ0kDfa9cFcpVs/yG17GtCcINofCEv9MRaSWTV0gSUGjAhyk=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzWZdDMxuPwxg0wRXQkrjxu/TO1vKzXAqpXtFTGHbpscYYn7PPh EbXn/odtREqilPQKeQb9CDeHDzmO2dKyaF+t88ycgZLux4xgLAz5+PEGuXouKYDKRpg= X-Gm-Gg: Acq92OGLjtj8MPGXEy5nQ+GmfVwZX/E0ZT2Up5qSjLEd/5/vLKQNtDDwTKT0wtcMhJE 1heWayLYAN1H9SCqwMs0eL0rJWVgtojFOJDg8/0EQcFczp26gTqo2VNZC6EzCEjWLpLLmACk8uD r1E0FJnmpGHYhwb9/2Y8vO7KaAi8P5bmonnyxvVxjYaIuTDz4Afe03A4+jwwyLZhbfPfg0uzdTw tJaWuV7YmmO6Qgn1B4iP8vvRxa6tob68xGV4Qho3ugfXLoUQ1isqW5179Yz9w963cZfoSRcBSxO qurE8fCVtSw9BbW17XEqpmz1mD+m+nVnYaNl8W7uOJztE5wH36ktapTqH7Bl76KSBAgvEz7GtnJ voltjQWbYzCDn9Bzn9AHp0WdX4jBaONcavx4dRu7bbqbm6lA1rW5+7DkKZwhbN6o+4v2CTbQwnN IMCWkT88U/hSaWsMMhyG64+IkYMm4UXymi91mlxyHWMUY0JEvoRsqsNb3G7gU= X-Received: by 2002:a17:903:290f:b0:2b2:539b:d2b1 with SMTP id d9443c01a7336-2bd2f6071femr19430235ad.16.1778677691580; Wed, 13 May 2026 06:08:11 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:11 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 07/69] mm/sparse: Move subsection_map_init() into sparse_init() Date: Wed, 13 May 2026 21:04:35 +0800 Message-ID: <20260513130542.35604-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org subsection_map_init() is part of sparse memory initialization, but it is currently called from free_area_init(). Move it into sparse_init() so the sparse-specific setup stays together instead of being split across the generic free_area_init() path. Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador --- v1->v2: - Add Acked-by from Mike Rapoport --- mm/internal.h | 5 ++--- mm/mm_init.c | 10 ++-------- mm/sparse-vmemmap.c | 11 ++++++++++- mm/sparse.c | 1 + 4 files changed, 15 insertions(+), 12 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5a2ddcf68e0b..28d179cbc451 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1003,10 +1003,9 @@ static inline void sparse_init(void) {} * mm/sparse-vmemmap.c */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -void sparse_init_subsection_map(unsigned long pfn, unsigned long nr_pages); +void sparse_init_subsection_map(void); #else -static inline void sparse_init_subsection_map(unsigned long pfn, - unsigned long nr_pages) +static inline void sparse_init_subsection_map(void) { } #endif /* CONFIG_SPARSEMEM_VMEMMAP */ diff --git a/mm/mm_init.c b/mm/mm_init.c index 96e0f2d8c3ea..12fe21c4e26c 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1876,18 +1876,12 @@ static void __init free_area_init(void) (u64)zone_movable_pfn[i] << PAGE_SHIFT); } - /* - * Print out the early node map, and initialize the - * subsection-map relative to active online memory ranges to - * enable future "sub-section" extensions of the memory map. - */ + /* Print out the early node map. */ pr_info("Early memory node ranges\n"); - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) pr_info(" node %3d: [mem %#018Lx-%#018Lx]\n", nid, (u64)start_pfn << PAGE_SHIFT, ((u64)end_pfn << PAGE_SHIFT) - 1); - sparse_init_subsection_map(start_pfn, end_pfn - start_pfn); - } /* Initialise every node */ mminit_verify_pageflags_layout(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 112ccf9c71ca..fcf0ce5212f1 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -596,7 +596,7 @@ static void subsection_mask_set(unsigned long *map, unsigned long pfn, bitmap_set(map, idx, end - idx + 1); } -void __init sparse_init_subsection_map(unsigned long pfn, unsigned long nr_pages) +static void __init sparse_init_subsection_map_range(unsigned long pfn, unsigned long nr_pages) { int end_sec_nr = pfn_to_section_nr(pfn + nr_pages - 1); unsigned long nr, start_sec_nr = pfn_to_section_nr(pfn); @@ -619,6 +619,15 @@ void __init sparse_init_subsection_map(unsigned long pfn, unsigned long nr_pages } } +void __init sparse_init_subsection_map(void) +{ + int i, nid; + unsigned long start, end; + + for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) + sparse_init_subsection_map_range(start, end - start); +} + #ifdef CONFIG_MEMORY_HOTPLUG /* Mark all memory sections within the pfn range as online */ diff --git a/mm/sparse.c b/mm/sparse.c index c92bbc3f3aa3..85557ef387c7 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -361,5 +361,6 @@ void __init sparse_init(void) } /* cover the last node */ sparse_init_nid(nid_begin, pnum_begin, pnum_end, map_count); + sparse_init_subsection_map(); vmemmap_populate_print_last(); } From patchwork Wed May 13 13:04:36 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237822 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TCNxrNFM; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20857-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0n5LdNz1y5L for ; Wed, 13 May 2026 23:08:41 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0N27KSz2yFt; Wed, 13 May 2026 23:08:20 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677700; cv=none; b=mVnO/Zq/sIUQQpZJJ5SJWDzbJnZPbj8EnF1/JSoKD7bKrsctGevConMrzRqGSa25HqYmdS+cijPVIbLKiuGIE8sm3AQSsuJV4DVKMXzAiBUMabWl9gL0xubui1FKXq/1CW79lMB5jv8ZbrNu+lhgpWJF4lkp7TMS5kmmL6Vb05v4PJO5EnKQwu7oVizE821OrpWQa4w+o/anP51eOOQb92gMPB9G1fkZeWo2Gpcc5X3S9oypaT1fx6OvlIW/WbHYN5i7VST2H1ui95wB2ViDCiAByYvBRXLh3uE7ayNG2XKoxDMOffqvRxAc7lNYPK36sk/rAFSNHeMcqBBCtx+q1w== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677700; c=relaxed/relaxed; bh=DYuPvzchRBegO13WUUAUtcrTx/HN+dPBsLEtJBVtJ7M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bUl4OSeL7OIRyo274GziUWhJ8GvLd+fH8uU+8bHswM8uzjq5J3GCdJJkhArYgwLAVnQ4rDQCQicz2dLqHOuWxxRDNXwIC0D5i6JCfFLrTmZ82NkCGKJM4hUeH8WeWNroBKHZhkbTPxRnHL4ossCtTA9XtUmARuf6I5m3m94iBVEG1I1cykGC6WxBETM2tF8TLM5uyxhGx3CSifON0hSKCH/uKtKnJsrQtTStjJShVlyqoei5eVjgASO3g0RdzrlmOuqA7fT0ullIre2ltxDth4VPot/0RW9iBm1kYKHXiQwQHbXJ6fk5oip9jyaWVqXSXGAwiuqB228LyOLzmxmWCw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TCNxrNFM; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TCNxrNFM; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0M3gwXz2xSb for ; Wed, 13 May 2026 23:08:19 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2b9fcf7c91bso67439625ad.0 for ; Wed, 13 May 2026 06:08:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677698; x=1779282498; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DYuPvzchRBegO13WUUAUtcrTx/HN+dPBsLEtJBVtJ7M=; b=TCNxrNFMthRVxg56THa2FWyQRa0ifVScI1j0DczGoRMFMtf/8CfGO9En759m6alVNE HADvf+4zPL5nNSt+CYyjG+vDBUEY9f/dAABMLTCf0+e6fTxEZP0mDpzmfG7mwZpbA7Ko ZhjtAdWM1dPctYhmJqtqb9PZ2oHIGkq12KsquevlG2f7H07wP1vh68pqvGRRcWuo9QRi 0dJ3jPOUfI/UmeZAZtp61yh53iLKqb/5hIhvU9gmxUaMGjxS+schEQaVYe2pnga78wqj QrBzR3IDIM18WX867Em6RMRM/n2WPXaIPEHx1d9CDtbh/0yjQBvIDoRCLIQLFo9V82zv HxFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677698; x=1779282498; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=DYuPvzchRBegO13WUUAUtcrTx/HN+dPBsLEtJBVtJ7M=; b=R7UqoUNMYJmmdBUZwmiPe+FgLdwAXZYs8jashWzW8aWZxCrVpyEZUw8C0spJivWRQj wguwPTvuDjDz/m4TC9ptH2qtA3NN4fndvw3hkMA4ZSKpBft+qt7akwbxp/ZS6iD5XdxK CfGljXeymTXO/Vohtd/1IuVSgGaEarWHZBluB6SNp8u3NbBZu2BFigyINWhOcJ/6OsR5 NwSkJvM2N4WZoa8mTcbNI0HCENkNzGzNIj5zmMX16whkew2AZ0uXcSR3/XBQkhFYh0hb FldYvaFqPg4AfCVyhBCQ5iyG9ygPX7GaVV9JhWDVGLwujZIEOZc6jMupTFuempNJwi4+ 1+dw== X-Forwarded-Encrypted: i=1; AFNElJ/2PNRCdyaIy1xAdFtY25U8VvugGfP0VNWg9aRPPKL++slfT/XUB+tP/hMoUUCNBjMmsGW7LoqsYtpJ5gs=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yw2r87DVmv0XvtlauZGpC6DX9sMZGebH1cym48OIpcFYufQHP8R Myb+BlRjn4s/I55+99whoMf1XNsbEhyFu5ubcnHVQbOTBBMaRc50g+9sNPR6RkJhwRw= X-Gm-Gg: Acq92OHlAGhImXCqa64cKQeby9m20qjB2pA+NVnaqEg8jkayppRtgPh/PZ5JDYllaS0 3QY+/Uf4pJf7uk2gYhox1jtVl20JoTXvGGb316x9IIj68JUIZ24qckpGkhkO/+0QGPJ3+xr4jnE WQVwKkR3yhN6PvPAeC9uD3kgtu5gAy/7ILuFw9+k9JDN/EKwJhy610K2ae+UjXhd1fnp9iLnN+8 KpHPes6yev7Rz9UAkLzUDPQdwWvF71Xs2hruVaY1uCZH0dYI71EwaI5C+U3JY9/xPfuGvWfth/c OthIKt6buKmfJdlK86xIxZMwpcuTmYpYF+uxOHuIuijEbfqLFtOnFu6kk9R1DqM6QrHhTzQFHm7 w0nKr3ggaHQiz/MmryyKtmZ2geNPHJ0LVcAUIM1vibbl7BHoo0QOjGtlN3o0GtOv8qPmiTc6yt5 2/QZkLK8Aue+HHDjM/pm9xphtERxCkZh3mJNbajfGhmpDsSCQGKaZZKqCKHPE= X-Received: by 2002:a17:903:46c4:b0:2bd:2de3:51ad with SMTP id d9443c01a7336-2bd3043ca95mr32656455ad.34.1778677697132; Wed, 13 May 2026 06:08:17 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:16 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 08/69] mm/mm_init: Defer sparse_init() until after zone initialization Date: Wed, 13 May 2026 21:04:36 +0800 Message-ID: <20260513130542.35604-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org free_area_init() is responsible for initializing pgdat and zone state. Calling sparse_init() from there mixes in later vmemmap and struct page setup, which makes the initialization flow less clear. Defer sparse_init(), sparse_vmemmap_init_nid_late(), and memmap_init() until after free_area_init() completes, when zone initialization is fully done. This keeps free_area_init() focused on zone setup and ensures that sparse_init() runs with the relevant zone state already available. This is also a prerequisite for later hugetlb vmemmap changes that need zone information during early sparse vmemmap setup. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador (SUSE) --- v1->v2: - Restore the set_pageblock_order() change suggested by Mike Rapoport - Add Mike Rapoport's Reviewed-by --- mm/mm_init.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 12fe21c4e26c..c14491c2dad3 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1826,7 +1826,6 @@ static void __init free_area_init(void) bool descending; arch_zone_limits_init(max_zone_pfn); - sparse_init(); start_pfn = PHYS_PFN(memblock_start_of_DRAM()); descending = arch_has_descending_max_zone_pfns(); @@ -1915,11 +1914,7 @@ static void __init free_area_init(void) } } - for_each_node_state(nid, N_MEMORY) - sparse_vmemmap_init_nid_late(nid); - calc_nr_kernel_pages(); - memmap_init(); /* disable hash distribution for systems with a single node */ fixup_hashdist(); @@ -2691,10 +2686,17 @@ void __init __weak mem_init(void) void __init mm_core_init_early(void) { + int nid; + hugetlb_cma_reserve(); hugetlb_bootmem_alloc(); free_area_init(); + + sparse_init(); + for_each_node_state(nid, N_MEMORY) + sparse_vmemmap_init_nid_late(nid); + memmap_init(); } /* From patchwork Wed May 13 13:04:37 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237823 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=De91V/AU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20858-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv0x2F7nz1y5L for ; Wed, 13 May 2026 23:08:49 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0T0wnZz2yGv; Wed, 13 May 2026 23:08:25 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::633" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677705; cv=none; b=JVHRzZvMNUO5eOEtQWz/ZEKfz5J/f7XIsaPL6yxHLRi0YF6TZxFseqvKbdvFPbm8qCwCvWdNSeRm+21HJvKWtdvkPERBELACXOJ2XtcVVxDadpzdxmZBiiYkC0igEyOh4c2tshF6CclxSUE9gntv4fBS6L/3QZRLc81rR67lFME3/Aq+AYIyUtmvjOA0a7vVOPh1eY7zRNGodoNOFOvrwFmWiEt2YKRAgupCTVOUpM4go4B46A4UcT3zTx1J+b5miPhyv1/uACDoSaYCtXmCYxOV0Scp749vvEtSpjfiZBGLKBIy/Bddmu/hXQ5P3FNKi49FMHjurMi2Ja5nl78stA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677705; c=relaxed/relaxed; bh=agVhcKOFJ9pAjcqZLpJ0s/37gKQkV6Sy+5Xwv4eBZ3I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lT1nrWY8N2jksqOQ+yOi8/1ABDF+I/I7Ap4ldr4D+h1omf4CNLy4PVlu+URv2BNpzYxeAtnXeHYj4E3cUj2eXArWUROKLUyKPf9i7W2YRwNe38fyKPEqddk5+Rm/L7W81ZG/VXtPfUuyPMVY4k1FRWU2FqczPx6e5SUJS9fw9aEuOp0IdiT9S+JT/yzEEiRGMumvQf4w2tASOkBKpUPsF+Wke32JKVfnHQYr/N/u7SYFYkZx9YwfJM7EGAjGHPkMPPhkWZNoOE7iKLzAzS5freZy5pQqkxQdrpQrKC+rMiL/FYBLgD2+gXKU7xg2Iitg4cuA67x/wJrzTZLp55KKsA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=De91V/AU; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::633; helo=mail-pl1-x633.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=De91V/AU; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::633; helo=mail-pl1-x633.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0S2Xz5z2xSb for ; Wed, 13 May 2026 23:08:24 +1000 (AEST) Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-2ba4efedbeaso47445155ad.1 for ; Wed, 13 May 2026 06:08:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677702; x=1779282502; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=agVhcKOFJ9pAjcqZLpJ0s/37gKQkV6Sy+5Xwv4eBZ3I=; b=De91V/AU9llw+DHQQKA34uh0J7za14EP03QRjxMNAtpX7rvRONXeqm+Rjv2IRfvjpK QN3DPNdZf50c8kRNmmUlOr1AJHR/o0xiNDl1W6LXpkRa76oGjPfOjTFaUa2Y5HIQSFvf 0V6pWiwzR9Ws5DnAlvxHZXB7PaP4cZh3veJlc7V7dU8OqPDk8dHn3DNtxv6PP3QhYOyx yTB4kRDsSU9LgydmCM0xu79nhNBcu+5voJsOVezRp2T2/PgRaxIDlyX700FpcxbFImnX Vps5S0dWCCjudaRo1ZepFH3J/aG8oVwWiqRkHJW0+nmfUOnzL6LBPJP4n8/Apiwnl07l 6HCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677702; x=1779282502; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=agVhcKOFJ9pAjcqZLpJ0s/37gKQkV6Sy+5Xwv4eBZ3I=; b=aPGpXyXATZwFLqqNneeEUnR0SRpKirSNXdlrABFjvWMcAN8DXSPHxQWxqyvSaHxibE MoPhdfJPRvy0+4J84aQNHFABECk+q6llmrDHX6ig4q41kwzHrIchvx9ixxHrFFONmzcr IfG/F3P3E6VF6nddCoOHg7ejDKefZ3ej/CKBs5b72UDH40N4n7sk0QDreDVEISqoOPP2 8L+ix8ypbmsl6XWUmxZPpHWyyxsZKpJgRlF56HDqtZAhvy1R38IS25eUxV6WJ8RntJ+Z ee0PT58up9FusQokW9bc0d2tcFFzyneiGsMxRvzCGzvrDVNYqpoPt/H70IqE5vUMX2fR hsrg== X-Forwarded-Encrypted: i=1; AFNElJ8hrBVsuYpbYhBdeDJGe2qb+VnDHwwJ4MvmBBgroAkIptqIpqaId8xHL2L61gN0mFkzJocgTdldXoTPoU8=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwndvncEB6YHlrOA1o0t4IACqXP2Hs/luVnyv2xUXbWOPceLAnl r1LmANw1DG1CPPDIopxF7nE6/i9XYmtuZXvryIccXIPfDrsjzeasYEXyAJ0F68bQwR4= X-Gm-Gg: Acq92OEJEHRUTQYk0BV4wEuHheB/WQNLibsvp65f7t0i3I0Wb6PYyDbX0FJgsnfHDQE 74/m7HKPC/8WUJWtOz5aMhLkU8bl26lbJaEDYg30bd/zE0as48jl4tPFE9N6UCw5y4FNgOFJ6pd 3Zuj4Kz4r5HzlpJ/3tz0yRs83te2j6MMHuHLjFWu6FscyvcULeil8iRz4/cqZLzEqQmICLQhx5D tvRLRRuVy8aB6UT5v+y2zENplgD4WwObagen8z6XrtnCX4nEkcIRYUS26n4/9AWFL+JmlfcTcO+ pPbUPCD98npaq+ejCwMkrpqRuwX6t7SxgnfFcPxfRYmofZY5Z2zW1rLs4tvREevHSk6QtNardUn hUBvau4B7fovtAoCYx/oELNGvUZxpBte1yKP6J7ZcM/xm3BnVDXVsFjfVmlj23Y3DYMY3m67qd7 PoxUe98oLPdLzSFhPob/ibwCFe0wz5P1i/9UOUCSIjU2hbGr+NBAGBOAmeJX8= X-Received: by 2002:a17:903:120f:b0:2b2:eaec:c810 with SMTP id d9443c01a7336-2bd2f4f76c1mr29145735ad.8.1778677702439; Wed, 13 May 2026 06:08:22 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:22 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 09/69] mm/mm_init: Defer hugetlb reservation until after zone initialization Date: Wed, 13 May 2026 21:04:37 +0800 Message-ID: <20260513130542.35604-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org hugetlb_cma_reserve() and hugetlb_bootmem_alloc() currently run before free_area_init(), so HugeTLB reservation happens before zone state is initialized. Move the reservation step after free_area_init() so the relevant zone information is available before HugeTLB reserves memory. This is needed for later hugetlb changes that validate boot-time HugeTLB reservations against zone boundaries. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador (SUSE) --- mm/mm_init.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index c14491c2dad3..75f98abfed97 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2688,11 +2688,11 @@ void __init mm_core_init_early(void) { int nid; + free_area_init(); + hugetlb_cma_reserve(); hugetlb_bootmem_alloc(); - free_area_init(); - sparse_init(); for_each_node_state(nid, N_MEMORY) sparse_vmemmap_init_nid_late(nid); From patchwork Wed May 13 13:04:38 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237824 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hDKoqrpm; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20859-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv144qrGz1y5L for ; Wed, 13 May 2026 23:08:56 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0b13Jjz2yHT; Wed, 13 May 2026 23:08:31 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677711; cv=none; b=c60Lk1YiKZQXQns5qOafZFG0sihYORCqQLSNBD2qkNEiio8nw+tpll/zFsEJTx18ABpBZ/rkJnezRm9EBKFx9BxqKGgQf/voZFsOK09m7bcTCPLGtFH22DO9e2655xlWsiiXuO43xqpXxN6arJM2jZco5brrZvOogaC+2Q+QQIWVzP1PAfRtNBaPjMIEpHg0i/EizxXTP7iIcXWTikQeX/QNcdEXAesD8KFI++k0+Ww0J+8kLbhw7SnC7piGCdlHQ/+FUr64x/I0q8odZoFHzBDsQ5Ww7kMPO108VtNISBxfi0vPWwkP9VjA++X78bL36e2l8Jo/FPlUcerc7IDe5A== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677711; c=relaxed/relaxed; bh=/LCdC6mqaLMXTH7aZ0tMWbwialgwYp0mO7Qo6rGSxZM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n/ffbf9OZxkObyejWFZIgbbPuvTnY8MSFSfN+1WL2Q42aY76s4ufAatG0wO+xRt7T6432gqOFfobUSjwIj0sQCYuZMobJU/JxO92mX7g5L/8yW0uwMlQ3Uw0pFYQVYIvJVSCyCZhXO1Y7mrpko607OEX3gpfkedWKaN53/9AJk7Cif0Du3WOEAFbWXEYIj4SopV+XyvL2xeYdldqxkasf8bWINzGIqPhUzBjYSSf9D3aHt49GxlXZJjhuIZPAnY7vQoO5Ul8JbFWBJrXKSVgX8sISL1KzgQH0ATHjdAh7TC3E38NXUffKbdfj5OigrlpO92oyRB0pYD9jBEsRgBwVw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hDKoqrpm; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hDKoqrpm; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0Z3Bb0z2xSb for ; Wed, 13 May 2026 23:08:30 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2aaed195901so33207465ad.0 for ; Wed, 13 May 2026 06:08:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677708; x=1779282508; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/LCdC6mqaLMXTH7aZ0tMWbwialgwYp0mO7Qo6rGSxZM=; b=hDKoqrpmoEDMujzJuuQjYBrYGlMKk3tLtPTCL5VMTysQ+6lMQEAcwB+tKHFzYnKgP+ H+FzoZGgJjkDTbtR7s4tiuVd7FYPSsKVnoNaaQFAPZXubRmif9D2UE2UyKsPZEWnaxTi XCNeuvwL1jIkSLNFfPsz5VhL5lBz2kBdIGB4at37jz4VLYfoXFe+c3yHwJcsoUybMnOI 1doRZz6fu2OtNloMYsElfrq/lKp/EsTcTYKarVyA9WO9X1U+GOJnJNwN+p2JzDDaWG+T vwe9qpCV0nSJSBsSegdk+GBaoqWfdBKB/9Fp4gYaYfA9IUKdAi6cwVA9DMLayEpo9hmo DjXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677708; x=1779282508; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=/LCdC6mqaLMXTH7aZ0tMWbwialgwYp0mO7Qo6rGSxZM=; b=LGkAxTOgQt4EH4eCVHjZSRDkrtila+d8Afqz2cYz6ZARwjs4QNxbFDpsqNBCVmAb+k icIzHg3ij63enVV9BP1DVeHWcD5iQUuMPGRP6W9C7/KUUui0Tpcg68cImhX2b2lyVEQU SddZgnqdRuz/y++/NSY2o94VlN3zBt1b234xgcD0j+3UAsTtf7Sq25OIpnDLWPxM7pe4 amw7t3SqO0yl+c11EGhqG/qweQ6MhVLNgWNz8vZt9HhhSfpwjHcyFq/z2oHKORiJnGaK YNsshNO63B4bISQjlLgiMnD6N3D9xfXiEApzN4boYb/Q7aKUJX0DQuYbPo72Q1L0C9ZB tOiw== X-Forwarded-Encrypted: i=1; AFNElJ/nqtJHFyEV9nYF5lBeK63F4wnz5diI2YA4Qah2tb0SDbsY8M8X16wOmHnLm1gbkpQVMd+xqypliQnYxI8=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyWz9A/9CqrqcqvdPUDVJzJPirT7BZpPFmHjgcQFx3SjNuOELtf cW9/+EezZjs/WY1RkbtmzppDc3ELWdebTvs2zH+sSEEh69yGFLGMHm/9aBO4oM5PyZQ= X-Gm-Gg: Acq92OGb4HtWyxeRS8TQ33zu5c2lPgZfaSSeU0+Hmo2zxYJ3Q7L44ExB7HU9EwR2TyN T4uWnhOQieD6w/ZNfUGurZE4SUbNQaeELu9xPkgWaLpWyQ+fFCHt3UjfQmuP6rKBNXIDNRMVF39 U+6Way3V7OPXEogLONcH8yVLoZduIEuMj6xSsMQoYO3FVbc+BJkUhkZg+vGXUEmhpyD5FCPkKdx 3TUILdEs5qMk8nX2NGgvaIT8XjGhW+d5Rrrr3ZmmN7lOh0NLjPlGJhTz0TpHJEr439+I+uwnXNK hizMSe+d6FX5Wse5m6qoVe1YefqoTQLvbUfh7sCMV3yN3m51i2O7hLOOBG64Uhcf5t/jB+wPOVM 7J1fp/DUL3B/8g63yHEHxageEFORmMlB8zPekVrhQZXCM0wxmvI5XLfJvLw/RnbvdcHpS7LUmfG rXShG6pwwAKbrabsoDtSfzSLrxbCKfoWlVy4ZQfH7RaBRjtCf2WAkuizUA7Pwr2FOwgeCyXA== X-Received: by 2002:a17:903:4b03:b0:2b2:a267:784a with SMTP id d9443c01a7336-2bd275c818dmr34598845ad.24.1778677708025; Wed, 13 May 2026 06:08:28 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:27 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 10/69] mm/mm_init: Remove set_pageblock_order() call from sparse_init() Date: Wed, 13 May 2026 21:04:38 +0800 Message-ID: <20260513130542.35604-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org free_area_init() already sets pageblock_order before sparse_init() runs for CONFIG_HUGETLB_PAGE_SIZE_VARIABLE, so sparse_init() does not need to call set_pageblock_order() again. With that call removed, set_pageblock_order() is only used in mm/mm_init.c. Make it static. Reviewed-by: Mike Rapoport (Microsoft) Signed-off-by: Muchun Song Reviewed-by: Oscar Salvador (SUSE) --- v1->v2: - Move the removal of set_pageblock_order() into this patch - Update the commit message accordingly - Add Reviewed-by from Mike Rapoport --- mm/internal.h | 1 - mm/mm_init.c | 4 ++-- mm/sparse.c | 3 --- 3 files changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 28d179cbc451..6bd9aa37b952 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1436,7 +1436,6 @@ extern unsigned long __must_check vm_mmap_pgoff(struct file *, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long); -extern void set_pageblock_order(void); unsigned long reclaim_pages(struct list_head *folio_list); unsigned int reclaim_clean_pages_from_list(struct zone *zone, struct list_head *folio_list); diff --git a/mm/mm_init.c b/mm/mm_init.c index 75f98abfed97..6646d4b47796 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1508,7 +1508,7 @@ static inline void setup_usemap(struct zone *zone) {} #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE /* Initialise the number of pages represented by NR_PAGEBLOCK_BITS */ -void __init set_pageblock_order(void) +static void __init set_pageblock_order(void) { unsigned int order = PAGE_BLOCK_MAX_ORDER; @@ -1534,7 +1534,7 @@ void __init set_pageblock_order(void) * include/linux/pageblock-flags.h for the values of pageblock_order based on * the kernel config */ -void __init set_pageblock_order(void) +static inline void __init set_pageblock_order(void) { } diff --git a/mm/sparse.c b/mm/sparse.c index 85557ef387c7..324213d8bdcb 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -343,9 +343,6 @@ void __init sparse_init(void) pnum_begin = first_present_section_nr(); nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); - /* Setup pageblock_order for HUGETLB_PAGE_SIZE_VARIABLE */ - set_pageblock_order(); - for_each_present_section_nr(pnum_begin + 1, pnum_end) { int nid = sparse_early_nid(__nr_to_section(pnum_end)); From patchwork Wed May 13 13:04:39 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237825 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=XZ2r/iWl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20860-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1D1fVSz1y5L for ; Wed, 13 May 2026 23:09:04 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0h57Whz2yLB; Wed, 13 May 2026 23:08:36 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677716; cv=none; b=D2KnRuFi2wrXUwXhn8uBwD7oGokKoCeY014Oc1xs2SDwZZTOa3FYotYnmNvMvHe15l1XFQgdxO3lkzCsZjJmP9+S/CAGM0Lt/VmpAH0UDHdz1VdlVY/DYPAsVEiTHp2MJsw43uGLYQP9ryqMG355PekODoABHg2EzyaQ3VcTz6H0rYicpJoW14jpsf3DWvpAXLFbXM1e/6XJ9W01XUwD6S7IS2KoN1ZgeYoXwerKgg9QCTyq3/tUh+U73cz05/jRpbxloq4tNbhLyJstjuQ3HX2FyLITyXFyW5DLOV+xQa/rwIl5VjxEkt8XdTszm1FtzM38xcFGPjGZkSlOdRhCQQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677716; c=relaxed/relaxed; bh=Dehsx8fG7S5ip+vlcPIiCfZ+EzOUnANbQVn1Jj1cBa8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RmAcaBCGHaojY3AtnecI/3xuRAPfXbPw4U1XecdqoFWf/f9a5vspPYKqMj4jN9lHvTiV4NUq9BeD+mBkxaZmXIS2SZj9BCJtX5SCTE8JJnh+RzL7vB93jzzrcORMx3n4cf9jPZWkI46dKU7mEKanWUKt+krYgtVHY7b2Qum1TsYi6yGqYRRkNyViUvJ4wRq1IiU2hAA2d2pt4DL6i1RIp/xLw9eRJWDzJh4O1irQ7OVkyh4XPUgJCDk75+H8AVlWSXl3V6aJYGikBblHi4Tsyhgdkw+x3MuMyqFhnvkHAn1njODiraedCJn3Cfng9hBbfrhqKC7VK7nC76diiYs5Ng== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=XZ2r/iWl; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=XZ2r/iWl; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0g466Yz2xSb for ; Wed, 13 May 2026 23:08:35 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2b788a98557so50687705ad.2 for ; Wed, 13 May 2026 06:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677714; x=1779282514; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dehsx8fG7S5ip+vlcPIiCfZ+EzOUnANbQVn1Jj1cBa8=; b=XZ2r/iWlYNLNsgqujm819Vg9OmbxRTtBwi+NPuxBYBh/eal37Yg0kMshhw2ld2+hOZ PVNblz5j2xazT/EEMrytpzz71PNypdrH42+F2Uygbtue2H8TW3em8uXuAKi5r+VRw+k6 p2RcoCwGz9uGQBrfVzLJcUrdhuHwK88NifLh1z9VpjkgZUosdng9Tcu7lK44dchTHbcD fWDldlpRIALiZ/HTVr99zDYZ7uO187DyOMTRhDdW6I9fJjD3M3/aLuKYVayPAasYfOXV UlTxDZWUR7PK1fbDuQnTfrn6JwgZUcL3OwEXyw7qo02992tVhZHcw248PSEtmhFsxUpW YHlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677714; x=1779282514; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Dehsx8fG7S5ip+vlcPIiCfZ+EzOUnANbQVn1Jj1cBa8=; b=KacjImci4xYap0y4wkNkAPkoE/tu9fTiuK3VrzNsR0d4SCdv99kWQcJpTdURwWphFE haK3uIeo+pL2HQKM6ekjHLuxUhG7uNAJth3ppjr2YmWLGAOg1XCJJP6NAQpqjZSfSYZe k0m3DmUfaDNyzLTgUdT5gJ6mzmA+AUppsXu99jpMsOF/dSgcWk2mUaEEUmA6LHTmzFRO 2dWMEdAFIeagl6lSrNMu2lq3TiLHzyjmIJ3PKiplhsBm6Kg6qBHf9R825/e86KRpi5Up VmuWzwOJzGheOYyWiJgvV9ycFid0OKq50s2VrtFSnpFFCwT0Rwm6k9xNYgTEpC72N5jE nQvA== X-Forwarded-Encrypted: i=1; AFNElJ9z10Zizd/1jYlHw9NKONW1xbBuhtWEBnnaJH4UroKsZCX51LIXXudHedmKobJQr16P5waEeXvstkqCIwU=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz9jaTyyAxk4Ff5V1PfsdCroBqC6934QWO2uftjdJO4kdWdIO6c 9/31Rdb3rXcvwYnW+Ic1DzjANo5reN8ZBo4cCKJNv6rMJiByVlaIUEBofbM+Xwn3H1k= X-Gm-Gg: Acq92OHFbM+ZlgpBxCWBYgg3u45VGSiHRjEkuKjgv5BSP4oXJ4wc/4ZciE9UkB6r9a7 tgcjefC4VsWVHD+iws6JBhEKbfwTAx56yTvrSa34z4oUvewE4jjQTpkEzaL2arrZpmFRsT+N0fR WVAQMAzVO1yto2S7z7N5gNHYQW1U9F4NawRFL0PeJ4fA2mQe5FJ5mITF5VHn98VLFL4TVICqrDe 5AmBSNl6fmA5HfQpt3Xt3BkeTtMxCmSn1JuGO9QiM68QSDnHujqXUObZTPovpdby2G3UcvXIKa2 wmII2vdER5+gDMsi7fPJf08b4UzdSxgEZ7SFAO1u9qt3Swh+hIffwXO5Qb7uh3FYqccg729d07h 7QRiYDrCJvXnlXYPhq2ZW0uWOZRl1EqxxUp8dQOJtj5fUHmHiyj30WpsY/i9A/aWIkM8vAI+s6I SbHOakdbdP3fH57+yb7sXZGg46CiNNXbMmm/Q6NiFS/hpmPH+prZfwjfm4rhA= X-Received: by 2002:a17:903:1a68:b0:2ba:5e44:ce8f with SMTP id d9443c01a7336-2bd26f13117mr34896815ad.0.1778677713521; Wed, 13 May 2026 06:08:33 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:33 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 11/69] mm/sparse: Move sparse_vmemmap_init_nid_late() into sparse_init_nid() Date: Wed, 13 May 2026 21:04:39 +0800 Message-ID: <20260513130542.35604-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org sparse_vmemmap_init_nid_late() is still called separately from mm_core_init_early(), away from the rest of the sparse initialization path. Now that sparse_init() runs after zone initialization, call sparse_vmemmap_init_nid_late() from sparse_init_nid() instead. This keeps both sparse_vmemmap_init_nid_early() and sparse_vmemmap_init_nid_late() in the sparse setup path. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador (SUSE) --- v1->v2: - Add Reviewed-by from Mike Rapoport --- mm/mm_init.c | 4 ---- mm/sparse.c | 1 + 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 6646d4b47796..165b83c9a9c3 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2686,16 +2686,12 @@ void __init __weak mem_init(void) void __init mm_core_init_early(void) { - int nid; - free_area_init(); hugetlb_cma_reserve(); hugetlb_bootmem_alloc(); sparse_init(); - for_each_node_state(nid, N_MEMORY) - sparse_vmemmap_init_nid_late(nid); memmap_init(); } diff --git a/mm/sparse.c b/mm/sparse.c index 324213d8bdcb..3917a47153d8 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -320,6 +320,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, } } sparse_usage_fini(); + sparse_vmemmap_init_nid_late(nid); } /* From patchwork Wed May 13 13:04:40 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237826 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TVSV1DI9; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20861-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1N250Gz1y5L for ; Wed, 13 May 2026 23:09:12 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0n6xYjz2yLP; Wed, 13 May 2026 23:08:41 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677721; cv=none; b=aQh6u+T9Sqa/suCIyp6HNdNWso0fNqx3oAS1UDLsumKMriGtjX0MQjjJ57EAe8cEsKpk7w3TrpG1JQQ/OSs6mJ8O3/nYCRKbmCsMmESBWQkvb8MQe8DJh2vPdgcRMdarLtil0rys9n16sOZh9HU1neoKl2rfV+4ZlaqMGhQRTWY0Pl7Wl1CJQ1izeJ2/JKxxidJsKwzkms8dBHMhEmG10bt8E5Mew9h9Nld3x4hk58EC6azxpjpxU0hLv1+ka79Nrqq6RHN8PqJw3QWxFB+54LjJgTJ5pzYbf9s+UZCVu83w5c/KdQ4l1V2EZDeEmU9CQ/f4fc4HorIfzf5m216+sw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677721; c=relaxed/relaxed; bh=ryF8YQ2n+wDSXZLLv2wjofP7Z1dSt9c7zUzJLRCYPQ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dUp+u2DSnjd2pRGc9UjiXX1TBhn+E06PmeK6koMpveNjm5zrG/Nw3urdnFIW/xwqCodTyCE4j/YMG+MYca41gzb/fo0Sa/klVl41+6fyvcBKae9ysYvUpsrzBFNsWsCNxgvjRreHWe1m39kREIfes9xn1aLzhg5/qy0diB/D43sGSBP5gV0F+j23pPs1Ml7uJfqbID+5TytH9w8+TfZ3tJ12k6HY1oJB4vlDTYupyfKYmGSn5yuN6SwdrjYTyi6Efd50s0NZmIKqWXc5+tYgrgedMnglfZvcxWpQwoRgF7fSco741xjlqblKFylcsLw4YYB9NLNlWzNd9dhq6ptnUQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TVSV1DI9; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=TVSV1DI9; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62c; helo=mail-pl1-x62c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0n1nxHz2xSb for ; Wed, 13 May 2026 23:08:41 +1000 (AEST) Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-2bd2c147abaso5474745ad.3 for ; Wed, 13 May 2026 06:08:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677719; x=1779282519; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ryF8YQ2n+wDSXZLLv2wjofP7Z1dSt9c7zUzJLRCYPQ4=; b=TVSV1DI9bTRSzQwJy0jHTxysiEZT30hwOoi1V49fZw+LDMYpMo/LlrdGwpTMszxYRA UECPHSQBsiIYnWKaV+yZOqdorGA5Kt/xgaNpcALBPgph4z42wA6/k8SqxPAUnjZrEreg tkRTjOZyHN+UMqVt62FnRXJ6SAx/9EuXyqhQeAmocdKtay9DAytsayW8jvJEBoK/FD1i dVLPVskXCguaru2wU8hvoj5Uj6AFlmdvCcoH9lMjN1Zr8ldo+c0s5C1Sm7K3Qu9wWdPH 6VUfvpe72NEImn/lWyU6NUDm9QvKNw2qWSYUjnMyKjaf0MbfxbknoslWGonXE/4eNaDX OLRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677719; x=1779282519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ryF8YQ2n+wDSXZLLv2wjofP7Z1dSt9c7zUzJLRCYPQ4=; b=bkAkEEijsiUBZldnfgMrlbGFGoexPFCOn5ru2R0BZgMLeTJm6u76BO6edsRKAnXqVH tUFPn2uA6qEhtjjGvrD5j+2YxHJ/PIz8fCzugV3j9GZF915nAG+7eubxyXMvFdEuVnOA t6Xd47PDhF9zcgglTHnnisOG3E9LvnKNYcxEItxa+VHaEBrFMnlN4W0ZneSP3w9nH/Ro ElEA25KUY3B6hzH/MNw/ouU2jNMmIb+/tqD2t/KsTXnSBcbX9qmmcNz3UkxHAL40Van1 hKha1MgFwTaHcoVffxXG2tufoVC95vO6tapYXPNC/J7mTECYgjBxH6ffNUiJ+/tvqMDx Ag9g== X-Forwarded-Encrypted: i=1; AFNElJ+7byXYf82AwwKjkhC++tRszwBlCq4nE3ijuZyZOHAbZ9ZJQ+usoJYyW8XHjCujNbNdnkNCFVbGFbFkJgg=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz4ZnepAiyNqjFXmUv1xyBBS4S59uRKKQLUymk10mHyvY964FLq gML5/FtckfaleGjj5EZqD/IYIzBpZARjJupTMUT9mbrW+E1KN2KaTIkwRX6Qrgx1REk= X-Gm-Gg: Acq92OHcy+mKmUf68OdIWN6GP5vZ02WlbZ5CvaelPmrUJzvW3juuCW4PFhdAOfiv+// 2cwsmWeWwBvTepoBzXjufX6UD1c8qfGQpq93L35k7T1Q68T8cf04sYwuYUcrmhKMrdrdGj6thp+ ZwR5OgLQM7U3MdG+YN32AJYrMIMzMwQgprQcu6jR7UtUS5ZmgGvDQjjDv1rVITBW7cnevbp/E8u +/JS5/PIaJn5pO+gDlI1tVAWoGyLfCVbp35rFF5pLzdLjaaV6ARb53E/pWvhWiePgIwzLVl5NXe YHvLyIh2q3c/l9jBknMjU1XnjssfszPf4RgaWEKpGH8gA0QBmJ5LA1H+ABZDsmfIMXr0nBGDJiH 7F6KXiwldVZfpVIg8zoBYGrCaOt8vneHsCOpgBWkWNCRmVwExjzTjGT9z1e+hm3ttFBIlWUVIDC 9NgrrO8JkIx6kBhK9So7iNFSlE30Q8KOZ6b9hGPo6hnghsUNSq6TigbbhWDNM= X-Received: by 2002:a17:902:8601:b0:2ba:839e:15cb with SMTP id d9443c01a7336-2bd275cd01amr24527775ad.27.1778677718942; Wed, 13 May 2026 06:08:38 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:38 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 12/69] mm/hugetlb_cma: Validate hugetlb CMA range by zone at reserve time Date: Wed, 13 May 2026 21:04:40 +0800 Message-ID: <20260513130542.35604-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Hugetlb CMA allocation currently has to cope with CMA areas that span multiple zones. Validate the reserved CMA range up front in hugetlb_cma_reserve() so later hugetlb CMA allocations can assume a zone-consistent area. Also drop the pfn_valid() check from cma_validate_zones(). mem_section is not fully initialized at this point, so the check can trigger false warnings. Keep the sanity check in cma_activate_area() instead. Acked-by: Mike Rapoport (Microsoft) Signed-off-by: Muchun Song Reviewed-by: Oscar Salvador (SUSE) --- v1->v2: - Update the warning message for zone validation failures - Add Acked-by from Mike Rapoport --- mm/cma.c | 3 ++- mm/hugetlb_cma.c | 6 ++++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index c7ca567f4c5c..0369f04c7ba5 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -126,7 +126,6 @@ bool cma_validate_zones(struct cma *cma) * to be in the same zone. Simplify by forcing the entire * CMA resv range to be in the same zone. */ - WARN_ON_ONCE(!pfn_valid(base_pfn)); if (pfn_range_intersects_zones(cma->nid, base_pfn, cmr->count)) { set_bit(CMA_ZONES_INVALID, &cma->flags); return false; @@ -165,6 +164,8 @@ static void __init cma_activate_area(struct cma *cma) bitmap_set(cmr->bitmap, 0, bitmap_count); } + WARN_ON_ONCE(!pfn_valid(cmr->base_pfn)); + for (pfn = early_pfn[r]; pfn < cmr->base_pfn + cmr->count; pfn += pageblock_nr_pages) init_cma_reserved_pageblock(pfn_to_page(pfn)); diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index 7693ccefd0c6..57a7b3acc758 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -234,9 +234,11 @@ void __init hugetlb_cma_reserve(void) res = cma_declare_contiguous_multi(size, PAGE_SIZE << order, HUGETLB_PAGE_ORDER, name, &hugetlb_cma[nid], nid); - if (res) { - pr_warn("hugetlb_cma: reservation failed: err %d, node %d", + if (res || !cma_validate_zones(hugetlb_cma[nid])) { + pr_warn("hugetlb_cma: %s: err %d, node %d\n", + res ? "reservation failed" : "reserved area spans zones", res, nid); + hugetlb_cma[nid] = NULL; continue; } From patchwork Wed May 13 13:04:41 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237827 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YhETcY9B; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20862-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1X0TvXz1y5L for ; Wed, 13 May 2026 23:09:20 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv0v56tlz2yMK; Wed, 13 May 2026 23:08:47 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677727; cv=none; b=dBAGPdh5viWtSdEj4BmEIUWWuS02KWHNswLIzGoQqSykvaGkScYjLm0yyWR5ShscxuEMdHiMUIZAHrw3O8ZUh/WNxRbJ/PMc9JxZ/vlS5mc0VnS0U9NJKpjYvJTfPcIf+LKdOAhMVty9J96a6JjG8Mtea/IAndwMgn916RbBu97cwBw23RuSjgB7rwHVVRa3qK2vnIyjephfni1OuwPVDJYrJ0dyiocayz/KaQM11GEa4QaW6HxUvpwQIBCzajLSpoDDLDdZw0tQe9/zfm6IXmw/MzNrS7byljZaLfuaHGJY2IlLOkjYQ2cGVc3QJqQQWhigbrH+raFJt56iDbtNkA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677727; c=relaxed/relaxed; bh=8SgGCn4266En8ULwXbiJTViXgAGbT0n5na/Xl2vlhYU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j4zQQUV4E+YFjBSkE/6ql8NtE7HgAMCYioApSEhDF47IaxVeehZaHvQgC2hytLdhD//tMijI/W4WZML2O8DGwymrxsljLs2/4tZod+a7VvFhXzvBWL31BUXp5HF11G811sXW/cbPzemcmj9VMHKPXYTrL7t6XvMlbQD4J215ozVcBBO+6rPo2kH8DPcYnK/k9Z8apYRD5qTaRfReHZAy6GSe54JLH9NWhVZpuVzSJ90KjNDA1H3HmStN2BXGY1tzZhNzpJSHRSJk9PHP7j+mNNEZbvlQCw2ZzsxeZw1izuFtSBz/8+T7tnb3UNzogfYRgucTXvgNxaho/2y7LlV2Ig== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YhETcY9B; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YhETcY9B; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv0t6Hv3z2xSb for ; Wed, 13 May 2026 23:08:46 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2ba4a1a0325so46775635ad.0 for ; Wed, 13 May 2026 06:08:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677725; x=1779282525; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8SgGCn4266En8ULwXbiJTViXgAGbT0n5na/Xl2vlhYU=; b=YhETcY9BIX0ap5iLZhjDQnBTwSuJUfwQ9tVXWD/vgZixGltj0dmmLR9KWhd7R0aRek kh28omzFQIyhPSXSridtyFcSbUvhMvxo065Pzh62f2f9Q6y9xKTvZXLOszDz8tCsf79p 8AMWNgr9IMVfs6cC7cfO7mvLK+ygITO2C2PIxyetUdd1XLPtoN8CffO3AK28e7cxvJ4+ rW1QezGy6g6fZMsmotpVmil1rQiQN91mcdYy/fsCdPsWAO/iWw7QoTyQFWnKvWl3sId3 Y5kSIhLTq/IwD1HY6OvPODmRxfZ+OtFkeSeZ6y82A14AbElSEWB7mN5jkjxg5JoX3eBG 21Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677725; x=1779282525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8SgGCn4266En8ULwXbiJTViXgAGbT0n5na/Xl2vlhYU=; b=D+39nvUBw7XuYDam0P3KxA1hLE3DnRiUFiULTsIelYUH1A49douxkHd04NnIAr3JS4 S6SxD7n2+exbPXmJpXhoh6ypnnt6hHolSQ97fGrwAdNfkB8ZKVapqBUEWMgE54avLCwh bnunwt6AVQVZuijevdyRuVKrr54OIAqmj0bsotS336GlZFm5ur+NJMWv4LmrhaoR91xM /403F1HW84sOp1RKLc02R/tGp3phDh9iSWuU/NgBms5LxyfWAyBNDLzVZjYV3TEm48aR MQPgl0o1VTu7jFzyTo03Bb7dIxtovJTwU+kfHB8oDEUN0fKNERvFrQZcHjXVWzRNGym/ U3VQ== X-Forwarded-Encrypted: i=1; AFNElJ9MHkk+2NamudaYOWu9Zl+YufeR+dP+pzTYOx79xsHR9wv4Ze9Vq+FLUEj+XQ9LIZKzEsvlIDGCOOlB8JY=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yy1RtKdG1DjpyclQG9+wHEfA+qq1gR7MmlrGguz3Jd/yDfUid8/ aor0s1fxFVBpKAK7DrnavKEgjVNE0uquDklpI0L6fqIlqd3MZ3/YyworQQABoaRGiLE= X-Gm-Gg: Acq92OH7/nRnmk/g2rwMeJFsWx+uYGrVFhLC9EJTiZuzvng0z5YFMZTjC199KwyE1/c zoI8+qPo0BZ+gS+MfUfE4ShLX0tFCCceH9u2/99DkgqTXuBwhEUokiXhvj9Vpnf52NMKqYCvjIW /RTcjhAzvNFlw19rEovPY62sOWV+Yw03LF+ICPrtWRHpOhPJeq9StXtgcZJ/m3vBQeP1NPF17nW MVjXMx21aIYMCyZykpokt1H/JNJmHOe00Jvp3UE/RoCGQDPWLHsN2EGG9refBJgYf0YbGSXL9Tu X1mQIFWWsE0MqTPCm9tRgpXLE0UTGx6YBH66kYEYWLcVLBpfVPyMhlSst40xIY590rFCA8n15aT hYu1SwI5pZreYKAdOm7M+Kkv7d2HBhC59xwtZeewB72stS6+tRb8nfF9hqLIByDZUHlflD65n6S vNFNmPmTOdlC0kK59gVhnj8EtPqFconCSJX16s9Ts0PX9+smrjsxHSoPyv38E= X-Received: by 2002:a17:903:138a:b0:2bc:eea4:83c3 with SMTP id d9443c01a7336-2bd276f5ce9mr42050215ad.25.1778677724547; Wed, 13 May 2026 06:08:44 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:44 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 13/69] mm/hugetlb: Refactor early boot gigantic hugepage allocation Date: Wed, 13 May 2026 21:04:41 +0800 Message-ID: <20260513130542.35604-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The early boot gigantic hugepage allocation helpers currently mix allocation with huge_bootmem_page setup, and leave part of the initialization flow in architecture code. Refactor the interface to return the allocated huge page pointer and move the huge_bootmem_page setup into the generic hugetlb code. This makes the architecture-specific paths focus only on finding memory, while the common code handles node placement and early page metadata setup in one place. This also lets powerpc benefit from memblock_reserved_mark_noinit(), which it did not enable before. In addition, upcoming cross-zone validation for boot-time gigantic hugetlb reservation is common logic. With this refactoring, that logic can stay in the generic code instead of being duplicated in architecture-specific paths. Signed-off-by: Muchun Song Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador (SUSE) --- arch/powerpc/mm/hugetlbpage.c | 11 ++-- include/linux/hugetlb.h | 8 +-- mm/hugetlb.c | 95 ++++++++++++++--------------------- mm/hugetlb_cma.c | 12 ++--- mm/hugetlb_cma.h | 4 +- 5 files changed, 52 insertions(+), 78 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 558fafb82b8a..ff8c5ec831bb 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -104,17 +104,14 @@ void __init pseries_add_gpage(u64 addr, u64 page_size, unsigned long number_of_p } } -static int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate) +static __init void *pseries_alloc_bootmem_huge_page(struct hstate *hstate) { struct huge_bootmem_page *m; if (nr_gpages == 0) - return 0; + return NULL; m = phys_to_virt(gpage_freearray[--nr_gpages]); gpage_freearray[nr_gpages] = 0; - list_add(&m->list, &huge_boot_pages[0]); - m->hstate = hstate; - m->flags = 0; - return 1; + return m; } bool __init hugetlb_node_alloc_supported(void) @@ -124,7 +121,7 @@ bool __init hugetlb_node_alloc_supported(void) #endif -int __init alloc_bootmem_huge_page(struct hstate *h, int nid) +void *__init arch_alloc_bootmem_huge_page(struct hstate *h, int nid) { #ifdef CONFIG_PPC_BOOK3S_64 diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 52a2c30f866c..9a65271d167c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -720,8 +720,8 @@ void restore_reserve_on_error(struct hstate *h, struct vm_area_struct *vma, unsigned long address, struct folio *folio); /* arch callback */ -int __init __alloc_bootmem_huge_page(struct hstate *h, int nid); -int __init alloc_bootmem_huge_page(struct hstate *h, int nid); +void *__init __alloc_bootmem_huge_page(struct hstate *h, int nid); +void *__init arch_alloc_bootmem_huge_page(struct hstate *h, int nid); bool __init hugetlb_node_alloc_supported(void); void __init hugetlb_add_hstate(unsigned order); @@ -1152,9 +1152,9 @@ alloc_hugetlb_folio_nodemask(struct hstate *h, int preferred_nid, return NULL; } -static inline int __alloc_bootmem_huge_page(struct hstate *h) +static inline void *__alloc_bootmem_huge_page(struct hstate *h, int nid) { - return 0; + return NULL; } static inline struct hstate *hstate_file(struct file *f) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b4999653a156..e9ba0be2eb17 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3044,79 +3044,58 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, static __init void *alloc_bootmem(struct hstate *h, int nid, bool node_exact) { - struct huge_bootmem_page *m; - int listnode = nid; - if (hugetlb_early_cma(h)) - m = hugetlb_cma_alloc_bootmem(h, &listnode, node_exact); - else { - if (node_exact) - m = memblock_alloc_exact_nid_raw(huge_page_size(h), + return hugetlb_cma_alloc_bootmem(h, nid, node_exact); + + if (node_exact) + return memblock_alloc_exact_nid_raw(huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); - else { - m = memblock_alloc_try_nid_raw(huge_page_size(h), + + return memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); - /* - * For pre-HVO to work correctly, pages need to be on - * the list for the node they were actually allocated - * from. That node may be different in the case of - * fallback by memblock_alloc_try_nid_raw. So, - * extract the actual node first. - */ - if (m) - listnode = early_pfn_to_nid(PHYS_PFN(__pa(m))); - } - - if (m) { - m->flags = 0; - m->cma = NULL; - } - } - - if (m) { - /* - * Use the beginning of the huge page to store the - * huge_bootmem_page struct (until gather_bootmem - * puts them into the mem_map). - * - * Put them into a private list first because mem_map - * is not up yet. - */ - INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[listnode]); - m->hstate = h; - } - - return m; } -int alloc_bootmem_huge_page(struct hstate *h, int nid) +void *__init arch_alloc_bootmem_huge_page(struct hstate *h, int nid) __attribute__ ((weak, alias("__alloc_bootmem_huge_page"))); -int __alloc_bootmem_huge_page(struct hstate *h, int nid) +void *__init __alloc_bootmem_huge_page(struct hstate *h, int nid) { - struct huge_bootmem_page *m = NULL; /* initialize for clang */ int nr_nodes, node = nid; /* do node specific alloc */ - if (nid != NUMA_NO_NODE) { - m = alloc_bootmem(h, node, true); - if (!m) - return 0; - goto found; - } + if (nid != NUMA_NO_NODE) + return alloc_bootmem(h, node, true); /* allocate from next node when distributing huge pages */ for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, - &hugetlb_bootmem_nodes) { - m = alloc_bootmem(h, node, false); - if (!m) - return 0; - goto found; - } + &hugetlb_bootmem_nodes) + return alloc_bootmem(h, node, false); -found: + return NULL; +} + +static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) +{ + struct huge_bootmem_page *m = arch_alloc_bootmem_huge_page(h, nid); + + if (!m) + return false; + + nid = early_pfn_to_nid(PHYS_PFN(__pa(m))); + /* + * Use the beginning of the huge page to store the huge_bootmem_page + * struct (until gather_bootmem puts them into the mem_map). + * + * Put them into a private list first because mem_map is not up yet. + */ + INIT_LIST_HEAD(&m->list); + list_add(&m->list, &huge_boot_pages[nid]); + m->hstate = h; + if (!hugetlb_early_cma(h)) { + m->cma = NULL; + m->flags = 0; + } /* * Only initialize the head struct page in memmap_init_reserved_pages, @@ -3128,7 +3107,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), huge_page_size(h) - PAGE_SIZE); - return 1; + return true; } /* Initialize [start_page:end_page_number] tail struct pages of a hugepage */ diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index 57a7b3acc758..6b5c2aec4449 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -57,13 +57,13 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, } struct huge_bootmem_page * __init -hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, bool node_exact) +hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact) { struct cma *cma; struct huge_bootmem_page *m; - int node = *nid; + int node; - cma = hugetlb_cma[*nid]; + cma = hugetlb_cma[nid]; m = cma_reserve_early(cma, huge_page_size(h)); if (!m) { if (node_exact) @@ -71,13 +71,11 @@ hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, bool node_exact) for_each_node_mask(node, hugetlb_bootmem_nodes) { cma = hugetlb_cma[node]; - if (!cma || node == *nid) + if (!cma || node == nid) continue; m = cma_reserve_early(cma, huge_page_size(h)); - if (m) { - *nid = node; + if (m) break; - } } } diff --git a/mm/hugetlb_cma.h b/mm/hugetlb_cma.h index c619c394b1ae..057852c792bd 100644 --- a/mm/hugetlb_cma.h +++ b/mm/hugetlb_cma.h @@ -6,7 +6,7 @@ void hugetlb_cma_free_frozen_folio(struct folio *folio); struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, int nid, nodemask_t *nodemask); -struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, +struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact); bool hugetlb_cma_exclusive_alloc(void); unsigned long hugetlb_cma_total_size(void); @@ -24,7 +24,7 @@ static inline struct folio *hugetlb_cma_alloc_frozen_folio(int order, } static inline -struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int *nid, +struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact) { return NULL; From patchwork Wed May 13 13:04:42 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237828 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JUjQbwS/; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20863-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1g1sJ2z1y5L for ; Wed, 13 May 2026 23:09:27 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv111SFNz2yMy; Wed, 13 May 2026 23:08:53 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::632" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677733; cv=none; b=XbP9vpR/X6SGXB4nuQfYBSYQSHFiZwlW9kBWXu7fXCtF8ldVqDs5nZvlgbjCJVxAIM1zP/bc6c/XKsUH0bOqddzgl+F+V97mBUOSjAWqSIKqVAePO/jmHLv73YBFjNYH3EB4IChAXanKXxiqb8l8EPv5V8K90+56SKBjMdvGDlCWWytTMKEbrEC0PVgV+0vm1Hz5qjLz70x865trW85krNP0+PFvjhl2ZHAidXaFZ4GHWhtE2Ya7okD0DaDzCctwUqQ8XK+xPbD1ObrAcd0EXEM7S/LbxwTgvM23nYCqJpPhzSrJ0+pIrGycuj4kjHOTjBsDi47gq2qeAC0yISAUtw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677733; c=relaxed/relaxed; bh=hECduHIccySBzXm1iwDhAhigl+PnFtS+sgbTNemvQzE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KcwlknxS97UKCrisNeB5oV2vUvj5yTFVCI19HYmrmBn52iwbjkb8eraH1s77X82xHTbES+G8z+jN4lQ9wF73gtiLUdADR6CLkJS1sPxpI/ObCWnUfSnOmb+UkyVoPzZDk8X5Aa8ZO+M40VC1lOZSQAFHgJs+1WLIhlgq6Jy4aOt8YarE71EyzbFfS8GJNOml+87kAHuRRSyLswjrgXzWTeBc9zJC3/oL58wtxo0KHyencJak0ZHyR3p4YOsofFr8dycaeYYUXPniQn70EiFnXT23iiYXXzNfCQ5dYvraO0peoA78YNNtWy8Txmq9y3YwIPBdKSYlXSopDXZFXgKenQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JUjQbwS/; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JUjQbwS/; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv103X4tz2xSb for ; Wed, 13 May 2026 23:08:52 +1000 (AEST) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2ba21d32776so45313285ad.2 for ; Wed, 13 May 2026 06:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677730; x=1779282530; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hECduHIccySBzXm1iwDhAhigl+PnFtS+sgbTNemvQzE=; b=JUjQbwS/nnLzbMFjsE4y7xOo6G2Hj5xgvgVoH9bfmr5ntWFLJzz4NqzN7ev7Msykvm fnDR9KEGq7nWWLi7LxDQLIfijm40lfrY6jFXnheItOjUGRfkPZPQX8JMabG3rbCPgZw5 sWyGHCVQQNa3MlY5tLCTjG142/cZpwAgIzkq7k7GNJ8jWWW3d3LSqmsdITDqk9vvZgsh tsAkXorT3HUdCwPg0aRQnE7QAd3//e/i94wABTqQ5ni1FlwFP1K3vCL8Ooy2AuJMS+BL AHN1/OPC34NgQdyOgIeM0uEU/424XJvCDdIpGLcn3G0ucR0KyecTuhb4u6e8c3fySlNu P6xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677730; x=1779282530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=hECduHIccySBzXm1iwDhAhigl+PnFtS+sgbTNemvQzE=; b=OV+9Q1tae/whE3z6Ynnr4h1uhWFF22NHmBZ9MIexwOfIUMJMmSrq4y7XcdHMUL6dYE 9EvA7Pxh8dCh/bCKwZpuWDCWBAExaBvkd/SS9OomB8W4vcvEuGzoXkRLHEolcJVnzzis gWOMH+fWUW3LOTglDX7khOWO+Adtsn2mvUz6iz7mrpoNln2A8RIcCKiF8BHoHyT0CTnS +ouHLCxTtIG3rjm9xbft+n4XpglvdcM/QfvcnIiQMiQKeffYix+cCzPrzx07q1oXx7GJ kI1+VgNbkIbfDDCwf8WfpcAHQF+Dbm/LP4xaqPhXOD1LazOj8AEko/mwJVGKUKCGSbS2 nbBw== X-Forwarded-Encrypted: i=1; AFNElJ/ElTSH4KhEQbTjqmdT9uInYwarljGfAoGt0LcKH11d8C9X6EU1cXzFs5Di7oy8Ai/jcIEatwyKjaU1fF4=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyiPZ8+OZS8oWgIR9O6zGznW9JILGttmO0e5T+ZeDqYYHYnhdrN JUQm1KkhtuYLpaOBGVyI+fcFI7tywm76gz27I7eGy8ceEdGsacS1tIiqVFVka0bQnc0= X-Gm-Gg: Acq92OEqPDhSfYdRf4cG6KJMzU/Z/V+yFgOcwUGv05BWMmPK2PXfn5HAAtUCAcarT0l 6iqlehkre/Y+mmC8PZMIScq9ZzXieQYCOn5Gsz6Vz7palltmTTeQTfPBQQyrDgAqhAhLd9t3oqI Ww0l7wqMeaebpxo+TuY4zHz/LhMRPhq+0ri3B4Pl8u0wSKdNJ0mbFi5BC1H9wmnomAAGAydw/+z llprMs2SUeImXfecILUTd86ylRrw0AeT6RnecVMoIT7RwC+q3CVEj9iHvAOxJLtti0PGt4Z+qut n5y7LXz9peHi29tZ3vkKojl/PO9djY3W6csoQraccPZVFilnxSMOTgDRrOLtA3HoPAOm1j8YlyP 6dw9IkvmghzmVgXXnn0MZ7qwneknI9Gr4lahvvN1FPYYv9wKInNezEgG5Hp0xZjKI2GdtShD0tl 331KXmHzWfMrq31cCvReZtPzeID60TLOn8zHH0HaiwEexeiEGBiw89QD/Rqd38r+PaJMjx3g== X-Received: by 2002:a17:903:3803:b0:2b9:e82f:bfef with SMTP id d9443c01a7336-2bd275b1d3bmr34581465ad.21.1778677730063; Wed, 13 May 2026 06:08:50 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:49 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 14/69] mm/hugetlb: Free cross-zone bootmem gigantic pages after allocation Date: Wed, 13 May 2026 21:04:42 +0800 Message-ID: <20260513130542.35604-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Now that hugetlb reservation runs after zone initialization, bootmem gigantic page allocation can detect pages that span multiple zones. Keep those cross-zone pages separate during allocation and free them after allocation completes, so later hugetlb initialization only sees zone-valid gigantic pages. This chooses to free cross-zone gigantic pages directly instead of retrying allocation. In practice, such cross-zone cases are expected to be very rare, so adding retry logic does not seem justified at this point. Keeping the handling simple also preserves the previous behavior. If similar real-world reports show up later, retry support can be reconsidered then. Signed-off-by: Muchun Song --- mm/hugetlb.c | 75 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 64 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e9ba0be2eb17..d5d324f69d7a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3077,12 +3077,15 @@ void *__init __alloc_bootmem_huge_page(struct hstate *h, int nid) static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) { + unsigned long pfn; + unsigned int nid_request = nid; struct huge_bootmem_page *m = arch_alloc_bootmem_huge_page(h, nid); if (!m) return false; - nid = early_pfn_to_nid(PHYS_PFN(__pa(m))); + pfn = PHYS_PFN(__pa(m)); + nid = early_pfn_to_nid(pfn); /* * Use the beginning of the huge page to store the huge_bootmem_page * struct (until gather_bootmem puts them into the mem_map). @@ -3090,22 +3093,38 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) * Put them into a private list first because mem_map is not up yet. */ INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[nid]); m->hstate = h; if (!hugetlb_early_cma(h)) { m->cma = NULL; m->flags = 0; } - /* - * Only initialize the head struct page in memmap_init_reserved_pages, - * rest of the struct pages will be initialized by the HugeTLB - * subsystem itself. - * The head struct page is used to get folio information by the HugeTLB - * subsystem like zone id and node id. - */ - memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), - huge_page_size(h) - PAGE_SIZE); + /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ + if (!hugetlb_early_cma(h) && + pfn_range_intersects_zones(nid, pfn, pages_per_huge_page(h))) { + /* + * If the allocated page is on a different node than requested + * (e.g. on PowerPC LPARs), put it on the requested node's list. + * Otherwise, the cross-zone page will be stranded and never + * freed, as the cleanup code only operates on the requested node. + */ + if (WARN_ON_ONCE(nid_request != NUMA_NO_NODE && nid != nid_request)) + list_add(&m->list, &huge_boot_pages[nid_request]); + else + list_add(&m->list, &huge_boot_pages[nid]); + } else { + list_add_tail(&m->list, &huge_boot_pages[nid]); + m->flags |= HUGE_BOOTMEM_ZONES_VALID; + /* + * Only initialize the head struct page in memmap_init_reserved_pages, + * rest of the struct pages will be initialized by the HugeTLB + * subsystem itself. + * The head struct page is used to get folio information by the HugeTLB + * subsystem like zone id and node id. + */ + memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), + huge_page_size(h) - PAGE_SIZE); + } return true; } @@ -3384,6 +3403,34 @@ void __init hugetlb_struct_page_init(void) padata_do_multithreaded(&job); } +static unsigned long __init hugetlb_free_cross_zone_pages(struct hstate *h, int nid) +{ + unsigned long freed = 0; + struct huge_bootmem_page *m, *tmp; + + if (!hstate_is_gigantic(h)) + return freed; + + list_for_each_entry_safe(m, tmp, &huge_boot_pages[nid], list) { + if (m->flags & HUGE_BOOTMEM_ZONES_VALID) + break; + + list_del(&m->list); + memblock_free(m, huge_page_size(h)); + freed++; + } + + if (freed) { + char buf[32]; + + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, sizeof(buf)); + pr_warn("HugeTLB: freed %lu cross-zone hugepages of size %s on node %d.\n", + freed, buf, nid); + } + + return freed; +} + static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) { unsigned long i; @@ -3414,6 +3461,8 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) cond_resched(); } + i -= hugetlb_free_cross_zone_pages(h, nid); + if (!list_empty(&folio_list)) prep_and_add_allocated_folios(h, &folio_list); @@ -3487,6 +3536,7 @@ static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned l static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) { + int nid; unsigned long i; for (i = 0; i < h->max_huge_pages; ++i) { @@ -3495,6 +3545,9 @@ static unsigned long __init hugetlb_gigantic_pages_alloc_boot(struct hstate *h) cond_resched(); } + for_each_node(nid) + i -= hugetlb_free_cross_zone_pages(h, nid); + return i; } From patchwork Wed May 13 13:04:43 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237829 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lypCB+y3; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20864-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1p0KXVz1y5L for ; Wed, 13 May 2026 23:09:34 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv163KKPz2yQG; Wed, 13 May 2026 23:08:58 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1036" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677738; cv=none; b=VLzGb7m1pt0CVKyJ5RzdqNvrBKJ6uIJGvIZnCa3et+zMOtSqTssevEX3riFNrlbJL20DnXL3uLT2h71ik0egHyU+GXNx6rbRYl5Jy9S3raJMMGT+r+jtwffyEPnbD1ViC9LZdn8X9Mi13l9WF/nUa0+/m+CASyM0WBy5aSwXQZ79WQQEROuZvzdjdNToxmcYGiZYhNYuy5of14YoiT+eLCmpKrhBsWTNL9SvHfi2AEY0Pmzi9+EeFpVAqyOqoLqE9mKI8sLAQ8d2lQhSUjRJQki1qX3g+vwhksxDTgGjYRF9QRm2kOew/9JlLM/1fPteUIR0v9mUcj0XlB4z0vfmoA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677738; c=relaxed/relaxed; bh=NFXyqH1Cou6po67hW+Tbj9/tmuAtnCuDUU0al6ZhZaU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J6n27XHLhtbj/OGCQVOHWO/fsmEvqgv35DhWUw62CwDAY7eicgiVo2dlkezyePHrq6As4p1Gu4sO+6/hiuFbfgkj9piBfs8t9edUdEalB9eHw5UT6TYqUBLJ5UHoBHRdJvugMw8YgExXWlCXJLAGih0kbfEmnuigR67sw+ZaegHIYE2s5J9C5NHzXCokJaSDNskWdbp/I2O2rCFiEuiaAjDE81WomtEnZX1kc8HBwIz1ZlWOeO7bBzvPv59ePN1celU68DWv9cwMnDWNNy737RgZd47b+iMcy3Cm2gMU+9zKF4/VRw7m8koLviBNwrDeqSqjYk60aOQLSWKoKb0KqA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lypCB+y3; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lypCB+y3; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv154Z8Lz2xSb for ; Wed, 13 May 2026 23:08:57 +1000 (AEST) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-3653cb9c6f8so5416304a91.0 for ; Wed, 13 May 2026 06:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677736; x=1779282536; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NFXyqH1Cou6po67hW+Tbj9/tmuAtnCuDUU0al6ZhZaU=; b=lypCB+y3tE4naiwjbjdYlG+VKasfDGJkqnnGeB6QlSB9nRhtqi2OK1HLKHdrlIL/l8 DygHwwufWPbQLUwDEBLzJ0iuDOH27b+oxrgYcK/DX1vgrU7eFFGf7U619T5IEws6s501 gApbM9bziOXdNeCVkEeZnWd3evWlM1rNT3G8e2ZkqcQ2yPpiBDkAo8FLrJseA8HQrE+Z Zc1Eum4nhPoxpKxU3PQvZnjipg6NqpaY527LITsf/az6gKbet9cbOA4rblLd1wlTwj0O q/6gyAuvEQ87xiRc+rWFKFShiFcJcrr/ECAfhcRZITF4Wpl37sadRAx7pLp09OCo2Dkb GgmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677736; x=1779282536; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=NFXyqH1Cou6po67hW+Tbj9/tmuAtnCuDUU0al6ZhZaU=; b=hPF+1Ltl8CD0GOs6dt51nsWpvXDhLpGb9RVI+g+TWldf2opITm+kxGPQAfD+jk6CvF FwCEuUqXFOzUrW/otCI1tD79aP3bhyXKrdCAqgv5RHl5tjSr9uVnBL6Bte1fY5i2Zrm/ TuFX3X4avkcFYpyiLpSfXyM1zSunsFYVPUe3HIi+cuwUKPMSGeYop6oW4joGvK3mNoer CL+qRyH8tIVXqgOv5wKRYCjC6S+tjMbkp/R7ED+AriAiMIbhnvfKNoqv7+V3MX91ZJrZ +KD9Wz6xSvGxg92lyYupqu7de6AzqENbTEd38g05q45lxjoPxJ+6+qWyo/mWmypTjpOx nphw== X-Forwarded-Encrypted: i=1; AFNElJ8VZ1peo92Z3JC/v7458WnX3cWShe5r2Up29jSmP4KAIeeQFYAovOzEqurKtkBRlfIFRmi8KMURy43fk9k=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxSdTLA6vUTt0XVzUworErji2MkP1wqz1QV83G839lkvNsSHslI ivvi4ajLwgrVA6MlrAcSSOpfvZxM22IUoTLiDG+IhUi614lKnqk3URg3nLwEWxjCr5c= X-Gm-Gg: Acq92OFS7F4nUTYwPxV3ya6Vxi8SAhn3c3X8S4itC95ozzlJklQZRb3ioYAdo/q0RK7 SU47McPDZcNWTDEfKszgQu04gJIzefV3j4A3nM3h0R6PrcQpePKxEuozMNpxI59aLxHmS0e4iuZ SlU5c7BKJyV3HCX313WSpDcOXyNsWlWXAuhSR2QVkDYvgS4QNKS5IVnq55x42G1YBU1Go1y3jyZ R19GQLAZnTS1zYOLEUI1YHfPW7i7S+RMBnutRZ9jAegqicspbxU6Cda7OClTWJipttARZKec6D3 WJkjYkzo//ZNbDuZ7+O1aqacTYuhvHgRR+ag9k/Cjs73HStMiDNVsAqy+pSmrcfd/fbD2h00q73 CeRTVjk1bnNN1bpxKpjwxl/9GlYd8RO1MzUCKJCYXC8xpvrWTQKAZUp80Bj7RAPrmBAhr7+Vwlm a6Yyauh7Zz5XPStaAjD+JoCF3D40uSgZrIYxHvJGBdxCLBMQ4IZ/gAYsLiUEg= X-Received: by 2002:a17:90a:d00e:b0:368:b724:6d53 with SMTP id 98e67ed59e1d1-368f398cc9dmr3518409a91.4.1778677735539; Wed, 13 May 2026 06:08:55 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:08:55 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 15/69] mm/hugetlb_vmemmap: Move bootmem HVO setup to early init Date: Wed, 13 May 2026 21:04:43 +0800 Message-ID: <20260513130542.35604-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Bootmem HugeTLB pages currently defer HVO setup to hugetlb_vmemmap_init_late(), because the optimization needs zone information. Now that zone initialization is available earlier, the bootmem HVO setup can be done directly from hugetlb_vmemmap_init_early(). This lets gigantic HugeTLB pages apply HVO as soon as they are allocated. Bootmem gigantic pages that span multiple zones are now filtered out when they are allocated, so the remaining bootmem gigantic pages seen by later hugetlb initialization are already zone-valid. As a result, hugetlb_vmemmap_init_late() no longer needs to handle bootmem HVO setup. Signed-off-by: Muchun Song --- mm/hugetlb_vmemmap.c | 67 +++++++++----------------------------------- 1 file changed, 13 insertions(+), 54 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4f58cd940f61..e2251bc47444 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -745,6 +745,8 @@ static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m) return true; } +static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn); + /* * Initialize memmap section for a gigantic page, HVO-style. */ @@ -752,6 +754,7 @@ void __init hugetlb_vmemmap_init_early(int nid) { unsigned long psize, paddr, section_size; unsigned long ns, i, pnum, pfn, nr_pages; + unsigned long start, end; struct huge_bootmem_page *m = NULL; void *map; @@ -761,6 +764,8 @@ void __init hugetlb_vmemmap_init_early(int nid) section_size = (1UL << PA_SECTION_SHIFT); list_for_each_entry(m, &huge_boot_pages[nid], list) { + struct zone *zone; + if (!vmemmap_should_optimize_bootmem_page(m)) continue; @@ -769,6 +774,14 @@ void __init hugetlb_vmemmap_init_early(int nid) paddr = virt_to_phys(m); pfn = PHYS_PFN(paddr); map = pfn_to_page(pfn); + start = (unsigned long)map; + end = start + hugetlb_vmemmap_size(m->hstate); + zone = pfn_to_zone(nid, pfn); + + if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), + zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) + panic("Failed to allocate memmap for HugeTLB page\n"); + memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SIZE)); pnum = pfn_to_section_nr(pfn); ns = psize / section_size; @@ -800,60 +813,6 @@ static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) void __init hugetlb_vmemmap_init_late(int nid) { - struct huge_bootmem_page *m, *tm; - unsigned long phys, nr_pages, start, end; - unsigned long pfn, nr_mmap; - struct zone *zone = NULL; - struct hstate *h; - void *map; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return; - - list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { - if (!(m->flags & HUGE_BOOTMEM_HVO)) - continue; - - phys = virt_to_phys(m); - h = m->hstate; - pfn = PHYS_PFN(phys); - nr_pages = pages_per_huge_page(h); - map = pfn_to_page(pfn); - start = (unsigned long)map; - end = start + nr_pages * sizeof(struct page); - - if (!hugetlb_bootmem_page_zones_valid(nid, m)) { - /* - * Oops, the hugetlb page spans multiple zones. - * Remove it from the list, and populate it normally. - */ - list_del(&m->list); - - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - - memblock_phys_free(phys, huge_page_size(h)); - continue; - } - - if (!zone || !zone_spans_pfn(zone, pfn)) - zone = pfn_to_zone(nid, pfn); - if (WARN_ON_ONCE(!zone)) - continue; - - if (vmemmap_populate_hvo(start, end, huge_page_order(h), zone, - HUGETLB_VMEMMAP_RESERVE_SIZE) < 0) { - /* Fallback if HVO population fails */ - vmemmap_populate(start, end, nid, NULL); - nr_mmap = end - start; - } else { - m->flags |= HUGE_BOOTMEM_ZONES_VALID; - nr_mmap = HUGETLB_VMEMMAP_RESERVE_SIZE; - } - - memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); - } } #endif From patchwork Wed May 13 13:04:44 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237830 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Vm7S0MN6; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20865-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv1x4Rbjz1y5L for ; Wed, 13 May 2026 23:09:41 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1C5ZkJz2yQn; Wed, 13 May 2026 23:09:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677743; cv=none; b=YC1pIKgqF554J5qNiQjENIej/mrha1eCGvE2JpdT6mzDSrnwCSAFEUsGjFP0cRkb59Zu0IbuR7v4XECJmOwHnv1tKZn+hW2JG6QRCp0S1h9kKASQ8agVxYHKLKtsgjmo4bF5K8LWnhmFp0tkhrx/sNcbaC8D4v6jJ7JTiU4qGhNiCbV5Q2hqO6vQe8q7WPsKjPKPGw5HI/P7HMSa5b5K2cHRJ7NbRpnaPUyVWmCpHe5cmN0+Vht2B8nK+bbwJ8nPAsNmKnYkfJ4e90YsUxDtJh8ugyNAD2T+NHD0PU4S8+mSMNzXdUB13n5EEs6tn9DqB4hnTEzvD4IfkcQA0PjYcQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677743; c=relaxed/relaxed; bh=J6ElOBv7vrKvKF8FNit7K/7AGAfvZCZIjE1YfuNh7wQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YJT1ACVjKFzgQ8QBHGCpfF7hPAy+f56OUcsrjf82NpdthG1x0LwKoIdLAuinYD/+HVno1kYlW5ktFBPnpgcJOaZWCjJoHcP1UNpvI9UhueDQ1mYHFUNrPe+74Cofjt3EsK8sj8dJiyn5Y6OAjKBMGj1/ddaMFLQHW6wJ3dz6pLib/u9CF7Wn1llFwq3Y/80TQx+ea12Qv0+1bZr9Zz9SMPDv1V+LM3rsIVzhELBzGc0J44OFtwI0asYF1Fi1YhBIenyX617+Sb6Ukt72fUde6xPP05lOFFS+hN9vL3p7Xt3gbUkrLbktlBeW3Zciudk5FynPO8OiDQHbYMoZ9iHdWA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Vm7S0MN6; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Vm7S0MN6; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1B6wM6z2xSb for ; Wed, 13 May 2026 23:09:02 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2b9fcf7c91bso67451965ad.0 for ; Wed, 13 May 2026 06:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677741; x=1779282541; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J6ElOBv7vrKvKF8FNit7K/7AGAfvZCZIjE1YfuNh7wQ=; b=Vm7S0MN6Lvs9+Ra2LTqISkVQ849DsQUtHsc4j7DIPPOYcCB/tQIgFosXWzAx3cz003 CAhBMNk66g5NS2wD1nuP3DyCpfNMDvWo0KLwJ+HNSfwRx3yUXvbge8tFHNSYM+kkCy2S oqbByB63TqalBq0gOnv+K7dAcivNkBOw0U92KGLjeHmk3kwkgw+NN83zY9B4ImebX+NJ YiG8hhDTgjOD/18JWawpVsOheiM8wq3q43XQod8Zp0ts+mqxvze6djzDQ9jUY4LQglO6 izGyUSycXQHfjrp++oqYSxo84JOKrJcOucd5NP5kp3BJYYuF+5gI83f7kRCBJyYyhyLc d2Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677741; x=1779282541; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=J6ElOBv7vrKvKF8FNit7K/7AGAfvZCZIjE1YfuNh7wQ=; b=jzp3t1Asl18ghEXeAWr5WQjdfqMXHkFMTjTaJENANGmtXfL6DSirk3fsxv3SyPAVfo lCroAesHPu7jK4BMeGD5cjrii7gPszEn9qhWe3piHnJO++2xNkRPOPvD6oX71STudd0+ IZ2S9lzsxPEp8j9fVYwB++93TFOFV8dtKE5BdRzolyxXCHnCkz8szIW6O+B2tTVzY+PC IcAGViwJkeelokqgXD3CTnl5TMfsQISY0K/2/v0KT0LinMKI3VSHeWg3A5IOtTqPOM/3 vE5ERuODCmbScYwSMiYcwH9cM68i+DLBGxA1AdZMoG7okdDdMPdazsPmMdTMa0eWyYxm PK7g== X-Forwarded-Encrypted: i=1; AFNElJ+4uDwaGHJM7gJjV5VfTjCGls9ComEiK4yQM78AVKpdA3x6CO9B6M1hTZHipLyElkRc707SiCAWqwuFtlA=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx7tVuPs2GnYfdiGUOsgJhyrh9ZATaFfvcVGgFKXkECkv+IYsAD P7J85ygOzDeafiLmdMCjHZ2asesRpL7vEjYqhjPJj6eovxKuw6BpTE3yP3BMWQjlAkM= X-Gm-Gg: Acq92OFywsd6VLHnWDyueHe42pwRrACs4uOnaHSIH+drylpc8A0gzYcgoIoBug5Z5// h+L+Fv8GJ+HkxvZcxL73qhhz6KR33xIuDMSKZ2zG3krDychZE6w7k4Y+I7sHoQFGWQuWYLoo/zc Q3tgR+knSXohdF3FfTKZopXhhYXFvdzM93y6EELGZCJRH3doP2reqyh7KHzIeCGQ2ofd6gudbvU g6/jyGszFsDH4rRCH34bv5aHHand2W3c++6B9I9xZsqChOvi11YLnEBnc4hRKFTpjK4dEcU7uSX WwTHoRMBVs23FUG2fGWzxRFSDfHZpEzhFPIPEQWumt4l5hn3p//SoRozehmIHm/pHvZvuiho6zV MygqsME2+ev0vbEDY0kfLiSiOxhHsnulsbiBmE57lhc/myNMRTDwu48E+17s4kcuFKZ8v5Kg8Qq AX7AXflG+aXtL2nPIJbWtUO5sgsah/d7dQKMRpoWNfyDc3Cnfo0CG43rQnV7HQknGBZiYovQ== X-Received: by 2002:a17:903:46c4:b0:2bd:2de3:51ad with SMTP id d9443c01a7336-2bd3043ca95mr32688825ad.34.1778677740889; Wed, 13 May 2026 06:09:00 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.08.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:00 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 16/69] mm/hugetlb: Remove obsolete bootmem cross-zone checks Date: Wed, 13 May 2026 21:04:44 +0800 Message-ID: <20260513130542.35604-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Bootmem gigantic HugeTLB pages used to be validated again during gather_bootmem_prealloc_node() and any cross-zone pages were discarded there. That validation is no longer needed. Cross-zone bootmem gigantic pages are now detected during allocation and freed before they reach the later bootmem gathering path, so the remaining pages are already zone-valid. Remove the obsolete cross-zone validation, invalid-page freeing, and the associated discarded-page accounting. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 2 -- mm/hugetlb.c | 70 ----------------------------------------- 2 files changed, 72 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9a65271d167c..ece4e6a4a4c6 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -701,8 +701,6 @@ struct huge_bootmem_page { #define HUGE_BOOTMEM_ZONES_VALID 0x0002 #define HUGE_BOOTMEM_CMA 0x0004 -bool hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m); - int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); void wait_for_freed_hugetlb_folios(void); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d5d324f69d7a..dcf8e09ec6be 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -58,7 +58,6 @@ struct hstate hstates[HUGE_MAX_HSTATE]; __initdata nodemask_t hugetlb_bootmem_nodes; __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; -static unsigned long hstate_boot_nrinvalid[HUGE_MAX_HSTATE] __initdata; /* * Due to ordering constraints across the init code for various @@ -3238,57 +3237,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, } } -bool __init hugetlb_bootmem_page_zones_valid(int nid, - struct huge_bootmem_page *m) -{ - unsigned long start_pfn; - bool valid; - - if (m->flags & HUGE_BOOTMEM_ZONES_VALID) { - /* - * Already validated, skip check. - */ - return true; - } - - if (hugetlb_bootmem_page_earlycma(m)) { - valid = cma_validate_zones(m->cma); - goto out; - } - - start_pfn = virt_to_phys(m) >> PAGE_SHIFT; - - valid = !pfn_range_intersects_zones(nid, start_pfn, - pages_per_huge_page(m->hstate)); -out: - if (!valid) - hstate_boot_nrinvalid[hstate_index(m->hstate)]++; - - return valid; -} - -/* - * Free a bootmem page that was found to be invalid (intersecting with - * multiple zones). - * - * Since it intersects with multiple zones, we can't just do a free - * operation on all pages at once, but instead have to walk all - * pages, freeing them one by one. - */ -static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, - struct hstate *h) -{ - unsigned long npages = pages_per_huge_page(h); - unsigned long pfn; - - while (npages--) { - pfn = page_to_pfn(page); - __init_page_from_nid(pfn, nid); - free_reserved_page(page); - page++; - } -} - /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3304,17 +3252,6 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) struct folio *folio = (void *)page; h = m->hstate; - if (!hugetlb_bootmem_page_zones_valid(nid, m)) { - /* - * Can't use this page. Initialize the - * page structures if that hasn't already - * been done, and give them to the page - * allocator. - */ - hugetlb_bootmem_free_invalid_page(nid, page, h); - continue; - } - /* * It is possible to have multiple huge page sizes (hstates) * in this list. If so, process each size separately. @@ -3703,20 +3640,13 @@ static void __init hugetlb_init_hstates(void) static void __init report_hugepages(void) { struct hstate *h; - unsigned long nrinvalid; for_each_hstate(h) { char buf[32]; - nrinvalid = hstate_boot_nrinvalid[hstate_index(h)]; - h->max_huge_pages -= nrinvalid; - string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", buf, h->nr_huge_pages); - if (nrinvalid) - pr_info("HugeTLB: %s page size: %lu invalid page%s discarded\n", - buf, nrinvalid, str_plural(nrinvalid)); pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", hugetlb_vmemmap_optimizable_size(h) / SZ_1K, buf); } From patchwork Wed May 13 13:04:45 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237831 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CjPulOsI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20866-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv246m4fz1y5L for ; Wed, 13 May 2026 23:09:48 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1K1Nygz2yS6; Wed, 13 May 2026 23:09:09 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677749; cv=none; b=Jgf8M7UTukcihSfVkf6vH+d/RIrZ3iPYCosY5VdFDvQNI6ptO23N43W4SMKYqyCCwTAa5JsH4rxqwxGiZi7rIZmt7r+lQqgf10DP6RTGNhbXHQTxbxWd1xnYWEdVHH4jTM2DtUYI6X9rx+xgHwtV6VV9huAL+DXLPvL3CT280IW62ywtl1ypB8+PwB1SaeZNc7eIpHNjAPT5K+qXylzctVZFWY2lskUZCA24P+ZacFc134aa11ZiWCfIBtgihdZzAmPn8cme2rwgIan8vhMNV9qUVu780t5WbVH0CyxdZas2fuwpgKxQF2gfvIeY6rgUYKxdQEedY++vRck0sw0DLg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677749; c=relaxed/relaxed; bh=dkeVw3HOR7cCG2HPcJochpzFU9bdzNUhNCW0ub/Qsyw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PP+oJI02L7T3Tm1jeMsU9KHyYRh+DGwEVCRVUbduqyW4SoAKkd2lMoILNurRI65+ELfS8RO/wXpmJkN0iMj/NrVWCNOln5LHUMFuTcgMoufw5vN6RRi54lsQRG1cq/ugHysgnLhXhH1NxdeXBUM3ZFpPC7nujh4vBAadvNEyCSq4vvvtkaG1TDQGUYcOBfwgCS3LACeI6QHbZtNtbG0tX5FdjInEjSTIXskSeyU7LDuKGhnBjyCvYLnXzDNumQjuxLoGTWPaA/5/YKMTh9TjE8CYqigLVh6KGFgzdV/GykKPqyluvXLLLzx0MgFHQHyJMFSzafj+NV5RquXtcgzkLA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CjPulOsI; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CjPulOsI; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1J3kTZz2xSb for ; Wed, 13 May 2026 23:09:08 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2bab2548e8bso31486845ad.0 for ; Wed, 13 May 2026 06:09:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677747; x=1779282547; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dkeVw3HOR7cCG2HPcJochpzFU9bdzNUhNCW0ub/Qsyw=; b=CjPulOsIukYM3DsRRRJoWRhP1pDFO/2ylhE2WspylPsh+b6tfIrsifL262KKMQBBrp XKtrVujhD/E9ynF3O9nt5lq7dpMQ23cxYQF11tConr6XyPaHM8EdKp2LKxO0a2SEwvOV PL4KkhS2op7ustRHXrHSJ3bnEJuMawM7u/SN7trXD+QQJyzMx/PmO3T/pVLfsa+Tvvsa JDStjEjymDAtBPKvS/1xOoJPu9piqnKDiBSIYPkRiveRiUIMY7+1qOcqr7t7kWIOYrSR p7TssmJO0vS8XOkf+yunukzmSsffmgivalzUbcvF1MBWfqwQ9CRY/xU02dRdlJgNDdLv cYRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677747; x=1779282547; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dkeVw3HOR7cCG2HPcJochpzFU9bdzNUhNCW0ub/Qsyw=; b=s6pWm/80FoQ0Tdhq7WG97XJAUXTCPnkeyiA8Vz0txTJQR132P944Tg1q/0ufkLQrzr kAlhX9F1RQTbC6n0K8PyhwkB9TrXY5/ov/NtbMSSkUZMTRBKUeLHcHgOH+YJEklVGBmX D++s8H19QTDRjwNkt3T0e5Yk79Fy17cQICJeix9I1G+5p9nDw+RGHZag7vyitj28dJCV rSJU+bZEyWKBxOV0l44VCV23oXHE2IJGKT+StBnOAjt1KAlPiZdE2Oj6plCmySUrOW4l uPoOihjyAvxMcEjnLQ57u/Zv0QZFa5smE1x6vttisLtFBA30Q/TWdzSxAKa4V2/yOdcf 1uLQ== X-Forwarded-Encrypted: i=1; AFNElJ/Dq1+waopJZyM6fzfhKZEZUGic8snGnGHVB0wzHrB5S/efCp2yo97pLrrH7yt7SSMP7ZsdeBHvwFuDjZY=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyLUpJ40AEqMwJVtJHHcr6xBQx9i8HV8gO4Hu1VJNEh5MATX2vS HjUhCx79hdsDLzb5/E+eHUZyPGxPXfrmHG8YxqqdexNWB3KjBDphcfQTAaYQI8CD67E= X-Gm-Gg: Acq92OHhKUkTfCsGx5+vCEKuYeAG4eApQyL/eQoZbzuYq71LC/XlcsrULDX9euU/deE awMYnhSp2JqQrJHPFQXqUeYmIKkd/UcPtAagnmu9XZzitDyffmdc85dfYl2DZRCJbShA1LFKjW/ 5G3rkjDDh6fo5jGnr2oRM77syozCMzTnqtoQ4kaMnKyJu4mDLmDUQwT86Txhiugpn+ywwyz3K8i RMW/NSzAGNXGsK/lHUT20CFsj4/uwOVRmc8qm+8kp3VYPzA4jcm/Ur24TLHCGZu6n80V/K5nO5g xhNHuOkQ2a2tpeRmt5XdgQecELPy25szV7jMgevBaoGfOwWeVCa59r7oeaK6QfAUkSHdVnxOA+V WaP7qjcN/Sbja/snJ2wM1XqlnW8vPC2m2p1yX2UJwDVfFzcgJLdu44kWXVw6QSVJ6QIObqT1ztb TOSVUHU+sVxpscc9TyMtrUR6bRieI13vkXMTAuFGGLmZCLZA35zeR3fliQPdQ= X-Received: by 2002:a17:903:1b4f:b0:2b0:9c2b:641d with SMTP id d9443c01a7336-2bd2f4dbc06mr31387055ad.2.1778677746427; Wed, 13 May 2026 06:09:06 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 17/69] mm/sparse-vmemmap: Remove sparse_vmemmap_init_nid_late() Date: Wed, 13 May 2026 21:04:45 +0800 Message-ID: <20260513130542.35604-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org hugetlb_vmemmap_init_late() no longer has any users, so the remaining late-init path in sparse_vmemmap_init_nid_late() is dead code. Remove sparse_vmemmap_init_nid_late() and its declarations. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 7 ------- mm/hugetlb_vmemmap.c | 4 ---- mm/hugetlb_vmemmap.h | 5 ----- mm/sparse-vmemmap.c | 11 ----------- mm/sparse.c | 1 - 5 files changed, 28 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9adb2ad21da5..362e16497533 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2167,8 +2167,6 @@ static inline int preinited_vmemmap_section(const struct mem_section *section) } void sparse_vmemmap_init_nid_early(int nid); -void sparse_vmemmap_init_nid_late(int nid); - #else static inline int preinited_vmemmap_section(const struct mem_section *section) { @@ -2177,10 +2175,6 @@ static inline int preinited_vmemmap_section(const struct mem_section *section) static inline void sparse_vmemmap_init_nid_early(int nid) { } - -static inline void sparse_vmemmap_init_nid_late(int nid) -{ -} #endif static inline int online_section_nr(unsigned long nr) @@ -2385,7 +2379,6 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #else #define sparse_vmemmap_init_nid_early(_nid) do {} while (0) -#define sparse_vmemmap_init_nid_late(_nid) do {} while (0) #define pfn_in_present_section pfn_valid #endif /* CONFIG_SPARSEMEM */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e2251bc47444..952216a49bcb 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -810,10 +810,6 @@ static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) return NULL; } - -void __init hugetlb_vmemmap_init_late(int nid) -{ -} #endif static const struct ctl_table hugetlb_vmemmap_sysctls[] = { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 18b490825215..7ac49c52457d 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -29,7 +29,6 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list); #ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT void hugetlb_vmemmap_init_early(int nid); -void hugetlb_vmemmap_init_late(int nid); #endif @@ -81,10 +80,6 @@ static inline void hugetlb_vmemmap_init_early(int nid) { } -static inline void hugetlb_vmemmap_init_late(int nid) -{ -} - static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index fcf0ce5212f1..17d45dac4324 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -574,17 +574,6 @@ void __init sparse_vmemmap_init_nid_early(int nid) { hugetlb_vmemmap_init_early(nid); } - -/* - * This is called just before the initialization of page structures - * through memmap_init. Zones are now initialized, so any work that - * needs to be done that needs zone information can be done from - * here. - */ -void __init sparse_vmemmap_init_nid_late(int nid) -{ - hugetlb_vmemmap_init_late(nid); -} #endif static void subsection_mask_set(unsigned long *map, unsigned long pfn, diff --git a/mm/sparse.c b/mm/sparse.c index 3917a47153d8..324213d8bdcb 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -320,7 +320,6 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, } } sparse_usage_fini(); - sparse_vmemmap_init_nid_late(nid); } /* From patchwork Wed May 13 13:04:46 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237832 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QTkd1pru; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20867-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv2D2mksz1y5L for ; Wed, 13 May 2026 23:09:56 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1R2K5Tz2ySN; Wed, 13 May 2026 23:09:15 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::629" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677755; cv=none; b=DNbh+5xhzP+xlU39BC2/S1TK91fubs1/1LJE+nSMiX5E0to/0CSXAF5HxOCrKtUgJT5W2UlqBbT7mlqvu0uZdd7tQguqqBydDCyk3V2o07PFyCmqKgmEZupWH3bz+Fu7+0oIXEJ7l1N2flUjKPEd/Hy06v8mbGCukKsbF8FR5sHUd4BOzlKgeyIhpV2JNZimdSEdhS3fy2IsiXrSjnmUbmGkq5PqRzA7QAC4gUDlq5bzUBwGRHjTNQVshQ5g+ByLWKaZ2LtXTXP7NTrphfke4yLpMdl83OyMelBh06IXecLUPnVmtSsJf06NS8k3lQk3DIXH70zNvGOLBaH1UzLUJA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677755; c=relaxed/relaxed; bh=QXUIDcJ+Gk/7y1fx1IgYV+5y2e2wzxCiGEVbnTW1zss=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mQb5QSbU6tZw6C+yqIkSkI9abcj6XUDIXsaYqJsfPd0rPfezLQYjjBPwDhRyVtWcuK2PxxyOz6dsyOWhMZoDbKgtiq8Y7YqnjmrwsboQoumGFoQI9ej2lJu2gbvL55Etl9vMqL7Ku7YIrBzaNN0U5l5EVbgK42dTvPrSyEMqN9QbPU/h4jy7sU+m0r8K+tv6Bk7nHdK0D7I9IPHeh4NuFTcfmrfDp1lUn5Y15Cud1//Hjcb4uxsr723bcFEZhedaqFogxsj6rDno1nTp4Y69+bvvSbqzSZxiOiu54irk3zPV0W91lruIfyqRX4aJwnYvPNiJtBB5XlhEh2Tl7XCU6w== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QTkd1pru; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=QTkd1pru; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1Q2HcPz2xSb for ; Wed, 13 May 2026 23:09:14 +1000 (AEST) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-2b4583f0a1aso44250175ad.3 for ; Wed, 13 May 2026 06:09:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677752; x=1779282552; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QXUIDcJ+Gk/7y1fx1IgYV+5y2e2wzxCiGEVbnTW1zss=; b=QTkd1pruIKx0yLD+yh4hfT/IwCIkeAebuI/gD0GVpJ0362iXQ9o9ovdlo2i4cnXQSn Nhia+kHVosTbrimHWCfKaswBYwCRkt90iDY3aTY2wxVdZLI2StNxs8WB7VANhf5MZ1+T K978Vmj2z9u4guuAd0yvrOHwkqFhvteVtj3uACCThtzJ/dXTlaUCm0QulIBia0BUnCVX 1EcRjJeWiMUOa8GShppFgLdzjwfOo5CKLQXQgMCSl9csja5yfFwfObt2cjP03ynHZml/ BVu8OmbOKMDwl0SXEg9H7eGLkTTeY+CQTGVOUVKp7RRAvOjXBBbfYVVDHw7KX+s8pmh0 EuhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677752; x=1779282552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=QXUIDcJ+Gk/7y1fx1IgYV+5y2e2wzxCiGEVbnTW1zss=; b=GF9ShFzUe3+ar/ceuurIPO//tXd4rzXUAM6Smzj1L9NbI5/XEmUI85ijVJIq62JoPW cvGjAkJX3RnyRoEnUEK2c6OmkH72zkVwvnQO/vU8zRqfjcr7v8uCP5CnNwV4oLOQ9UbM DERDJXtgzN1Rl7pXpIcjbDK4hMrqHbxAbr2VEZMsyOUcMkyz//otojlXlgQwUEA94RJc /zOVqTi2Pw2hH6nEkhnFq/s3tQLdKOO4FsEEnLs7bzHYUckqen721NbbsDlcg/bKvb/R /odpncHWSc2GDrsWTRY46HmF588stHPM/qZkaT+7511xc0odIbs5T3OI6d/XjRqD6DGu dTxw== X-Forwarded-Encrypted: i=1; AFNElJ+Ux3NVK0lqhXtgh4lPDOejMsrNARNNcY7C/FNw6/UCxuHWRvCCjya3FeV2gfse/AuG4TWkxfp+A9vEd6U=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwOlQ1uNytVJxDQ+PcdKO8yRlOib18PvV+zwJnxl/5Bq1Ma2BnO bZ/BTrJNIzZByGFptGcc0B7TQtuILw2rkjmeOUejJUA/i+XQt9u6YwB8bqsXJG15pZY= X-Gm-Gg: Acq92OERN/zH1P7BhNfNF+GTHUheND4fUzO+wQQrXhPVloaTi7He0STCrbrl2D78/Uy 0bOIbuH2SEI6Dv3GjMyO7jncF7OMJHGyp4ym1iA7mOlHwvvmnYbNo6Yk0Yfmyap2Q4fQ59v3UcU hsh3L7+h0YNy4/K7b7n7GCW9OKQU5Uox8ipoZbW5axq9qRb68y0yTiznblBAq21tAKowXrBXFfY yn82FrujZDgaa8SZrYkRIYhZ5iEqZdDR+7cGvJGF6Evmj/1VPC1bTe29ex8SETxd3WooEQ2tXcx 5DkS7Zqw0CJP/5TZ+g9W/W5PL6D3Tqjp1ur/0sbr0otFsVL8+X7fJiZ0Sn1bIpnPVlRVU31nFZ5 0CRUeU3ejg0QB6XDkvkQKCLT53GIqBcFDzgT1I5byEhAXXKcPkcpHqf7RzK1/9tPw85KSc6/m9s PH/zHpfAcUFfz4vV7dKmhm/ltkxb1n32tR96gVB1bZ2SfgyP+97WtMXY0dvzk= X-Received: by 2002:a17:902:7d88:b0:2ae:54b2:27c7 with SMTP id d9443c01a7336-2bd2772c78bmr26046865ad.39.1778677752045; Wed, 13 May 2026 06:09:12 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:11 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 18/69] mm/hugetlb: Remove unused bootmem cma field Date: Wed, 13 May 2026 21:04:46 +0800 Message-ID: <20260513130542.35604-19-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org struct huge_bootmem_page no longer needs to keep the CMA pointer. The bootmem path only needs to remember whether a huge page came from CMA, which is already encoded in the flags field. Set HUGE_BOOTMEM_CMA when the page is allocated and drop the unused cma field together with the redundant assignments. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 5 +---- mm/hugetlb_cma.c | 27 ++++++++++----------------- 3 files changed, 11 insertions(+), 22 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ece4e6a4a4c6..fd901bb3630c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -694,7 +694,6 @@ struct huge_bootmem_page { struct list_head list; struct hstate *hstate; unsigned long flags; - struct cma *cma; }; #define HUGE_BOOTMEM_HVO 0x0001 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index dcf8e09ec6be..1f0a0e31d624 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3093,10 +3093,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) */ INIT_LIST_HEAD(&m->list); m->hstate = h; - if (!hugetlb_early_cma(h)) { - m->cma = NULL; - m->flags = 0; - } + m->flags = hugetlb_early_cma(h) ? HUGE_BOOTMEM_CMA : 0; /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ if (!hugetlb_early_cma(h) && diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index 6b5c2aec4449..fbe5ed7ffaa7 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -65,26 +65,19 @@ hugetlb_cma_alloc_bootmem(struct hstate *h, int nid, bool node_exact) cma = hugetlb_cma[nid]; m = cma_reserve_early(cma, huge_page_size(h)); - if (!m) { - if (node_exact) - return NULL; + if (m || node_exact) + return m; - for_each_node_mask(node, hugetlb_bootmem_nodes) { - cma = hugetlb_cma[node]; - if (!cma || node == nid) - continue; - m = cma_reserve_early(cma, huge_page_size(h)); - if (m) - break; - } - } - - if (m) { - m->flags = HUGE_BOOTMEM_CMA; - m->cma = cma; + for_each_node_mask(node, hugetlb_bootmem_nodes) { + cma = hugetlb_cma[node]; + if (!cma || node == nid) + continue; + m = cma_reserve_early(cma, huge_page_size(h)); + if (m) + return m; } - return m; + return NULL; } static int __init cmdline_parse_hugetlb_cma(char *p) From patchwork Wed May 13 13:04:47 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237833 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=aDnJIndo; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20868-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv2N4y7hz1y5L for ; Wed, 13 May 2026 23:10:04 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1X2yDpz30B4; Wed, 13 May 2026 23:09:20 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677760; cv=none; b=B8/EEu430jUUnDuR8kN3aej+qnerGnI4ihhyM8E/okYX/6KkQmd6Hsm4pCXDCimKbvfvlekhCSV8KuqeG27wHqKfHcUvnO0OsKl13C2uahsCG0vZWM2oUXESpaaxbbRsCkP5BxUk+SHFQU8Jl0HktVDqe8BCEyCteyKgLQh8gE7rj9wH9Tq1FRcZ2Dbte5/InLTG15GsQd4MwWveW624Lu82N+/wsf5lifvHIFzXevFwdGoYrCPJPv/FvVHTrC+50Tp0LhKGmQTWEdvE5jYfTBm8B3AZ0UNjfPLPT4j9rW3Y5fp4hvVwEr/Qnd0iaskTBT9i91BL8DJ4fnMV/lv7nA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677760; c=relaxed/relaxed; bh=4Fut968wGN1n7g7zLkJRGy6a00PEkBTyJzeVX+ErWGA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aRp8+hocrbZYPQkuwNb6Lc5VCqYiC8XulvPb035j4o6PwUmGPXazFxuznGz3klFYT4GXL7i1X/zdw2u2PSqw6rbmqg9mZ6sVpTztWP/iNB1PEtgCF+6smZaAvfyGF0O9YfU3cT1q/HXuWo/AxetLgvWojNnB1nesljT7t5AaetqngHabXqNQsE0muRv2DYv+s9qCodhx9OB0nA3iQA16r2hePDK3jWfwCG58fa0naJRe8ZAlz7Hg+bciGUHBkaXVDu6X/z8uQk3ZB/zrTc6ywDBWkFtpYKEAkHfexZfzvwrt3CbsHgnPEqUl2imkUuVWsAkIPA0hA1mCPE5LpZnm6g== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=aDnJIndo; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=aDnJIndo; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1W46ykz2xSb for ; Wed, 13 May 2026 23:09:19 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2b9fcf7c91bso67456615ad.0 for ; Wed, 13 May 2026 06:09:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677758; x=1779282558; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4Fut968wGN1n7g7zLkJRGy6a00PEkBTyJzeVX+ErWGA=; b=aDnJIndosSLLLpyzoQKTwYM/nhC03iVs0vLyMp/+MpiisivsKWAop0Hj/m922Mg1zH SSPhiapD32Iqsy+S/mzf5M0TEHDabDlInIMygj8pUqh+emm8QJVVpkxe476W+B9nrGXI h3fB2fzEixumo3R656YxijDOePdAZVCcStRofBCX81Qp6zj/lLZt4eaB/1GU3Qmhtk1l Qj3+nAYXZXIQ4lj002/hEjhcV0KeLfn6TpD0e8/gQ3jyAtWUcp/t03CRdju3FJ6i3PQK Pofk1fexhWsgv5W/Cq3z9hB15yJtOdQIHhYDpVYRR2wKUR+lNibIXHObAexO3Ln0IJap 25Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677758; x=1779282558; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4Fut968wGN1n7g7zLkJRGy6a00PEkBTyJzeVX+ErWGA=; b=dsp1TezZx+5UcHq1ftO2YYX8sI5PMshoLExpbRpwBTTiags5VhfHj5Vai4rXi8nsf8 X8CSPMGarb95p5YWAdkQmhRar7bb0Nzj5HIn2Sek2TSapK7mbXeSEzB3wr30fXOU9pIL rxg/j/DurbB2cId8tHWJ53z2IxKlRRSu3xaxHtOhw2HKsC71OIwVQe/0pJQ/OUi2jhqz csIEZ17LDrqvvwo0kWat9EdKM7ezOwWd+hVR0ASA+6jOa8NTibdxLfs9zufzhbf+cTC4 MkzdH7GeTLleGvW589G1I/ktZUDWhamnqSRdd4t9vI9khYEKAZyrtAIpqoF6lSl8pScj hfog== X-Forwarded-Encrypted: i=1; AFNElJ8v9Runv3EPmQ83uH6KSuMo5cccHGiZmP6uqy7IrAGjymJbuxALaRtkxlBHdrrfbMNMrYGHpWRP/2CbFlI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwzOAl2LDIPgEyXTFtIUJaNfIdBCZ91IY7PipS6x0bakf5DoiXp QdS1ahtS8A/k0NEsRAxYh0aSWkAfX/Qvw0Ih2IP6fR725ix2LByea+uJfyt2lgEgkpQ= X-Gm-Gg: Acq92OEJ86kkAmPWz6FuLPI6hjoHuZLYouaI2dJF0ujLU1A3PIsPGkwIy0kHLc7R3H3 cexLDuLToH200EsPMk6DU+PHHAMkfIZrO5RAx5UT2DhS/GybX9TsAY5lNX723I9l3ty0YV9dxk1 lROCOyqZTofQb5m5up4ZxpzacHT4aG+5kShTRKZYo9HtXQm/xZhL3Zz6wRH7zAzSox288JixkrM sB9juiRCK2ZDCfJ4EPXMH3IdK9NC7fZNpdmbUpXv/jnpBN9MI7DCm38U8iqWpB6vnDMPi+26W+f y/QlibBG/hTERDfU/L+rzGLCsCGiuzfRte+BnG6NIQSa4Pixv4VIDoyx8ZqU5d1xsTQJyCL+PH1 eid8hrLbxvIQEWB5D4EkjEdW5B++rYzktmoGHdgALEsl232fHqnSIZ6/2xojx7+VUFUoeFBEJ2e gzI0hsk2UorVe7e7Pr4gP9yKEEaC6cqtK+lLH7x57TMwpV1A1H5OtMbVBcWuA= X-Received: by 2002:a17:903:bcb:b0:2b9:cd2d:6f14 with SMTP id d9443c01a7336-2bd2f4f740fmr22553475ad.2.1778677757605; Wed, 13 May 2026 06:09:17 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:17 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 19/69] mm/mm_init: Make __init_page_from_nid() static Date: Wed, 13 May 2026 21:04:47 +0800 Message-ID: <20260513130542.35604-20-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org __init_page_from_nid() no longer has external users and is only used locally in mm/mm_init.c under CONFIG_DEFERRED_STRUCT_PAGE_INIT. Make it static and keep it inside that block. Signed-off-by: Muchun Song --- mm/internal.h | 1 - mm/mm_init.c | 4 ++-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 6bd9aa37b952..4a5053368078 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1754,7 +1754,6 @@ static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid); -void __meminit __init_page_from_nid(unsigned long pfn, int nid); /* shrinker related functions */ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, diff --git a/mm/mm_init.c b/mm/mm_init.c index 165b83c9a9c3..c64e5d63c4ae 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -688,10 +688,11 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn, } #endif +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * Initialize a reserved page unconditionally, finding its zone first. */ -void __meminit __init_page_from_nid(unsigned long pfn, int nid) +static void __meminit __init_page_from_nid(unsigned long pfn, int nid) { pg_data_t *pgdat; int zid; @@ -713,7 +714,6 @@ void __meminit __init_page_from_nid(unsigned long pfn, int nid) } } -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) { pgdat->first_deferred_pfn = ULONG_MAX; From patchwork Wed May 13 13:04:48 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237834 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OEC4XgxJ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20869-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv2X4t3Jz1yKH for ; Wed, 13 May 2026 23:10:12 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1f43rGz2ySW; Wed, 13 May 2026 23:09:26 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::635" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677766; cv=none; b=arPxliLgg0JvMXOCUKsCMwor3GxUWykPu2Hj9yPctwAPdl2eVVqyyhfYhHK+/DF27tWCQ/+IU0g0UY1jU2AycEgTJpFNd+mw2dsVmOamhhbspMYKY5CHKG84yWlQ32IWPfaKN/eowT9YyvQHDcyhf6m27Gd/0ApHTvCjQ/kMhIeftUa5+zaRc+odOkgWGRsIrWdOlQZT8C7IE2oDd5uKOQobuJ6vqzIL/wY8sSU6TldNwCW6KWg9+Xik4F/wpIMjCp536KviahVgyqniLXebJF+ihQ7mvvsfDXCqc/mdhnmGrJ9/VVQNcsnskfFPFCjfK1BXLwYtKMRKlSc640gkFw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677766; c=relaxed/relaxed; bh=jxlTDmfL7zgwYuPQPoNgGF9X7KRsj/YO5X5OnwZ25t4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RQXdiGtcrdRGbY8OT/gUsm6QLJt27s9o3yc0xZyyemaOJYjgkkcxlsieq8aBueaMIWHcpMkbN8ZyREfTSXkB3k5VEZVVCeC0jT5/klfzfmKdYkak9jKXJAZH3hFaUyx84ieZEFUqgNoaBnPk2Odw9Ecn9X2vd+g1U6eoZncc0+AYEUcg+qU5oPf4vexIl9Q6P8puWGKCYdmlAkEHUtS+Xoo1O6dlB4vFVTD/vp4bhX23Wh7EnJtK9GoLWYwkpA1jAbWVgdKJmC7TZigfFcFO7odZLF0wpOeJcW2cKh3hbHj3ZF63aziq0S/D+K8YRaXIZg4B9TPIdybSQobTSW5mnQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OEC4XgxJ; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::635; helo=mail-pl1-x635.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OEC4XgxJ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::635; helo=mail-pl1-x635.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1d5FRLz2xSb for ; Wed, 13 May 2026 23:09:25 +1000 (AEST) Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-2bcf48850c4so18641855ad.1 for ; Wed, 13 May 2026 06:09:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677764; x=1779282564; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jxlTDmfL7zgwYuPQPoNgGF9X7KRsj/YO5X5OnwZ25t4=; b=OEC4XgxJAx5OCwlXfVF62twjsBa+5HGcKjhsCRq+6XV0U0pgg9lvFKiFlyIY+Q+hKn uLIX9ibi8kfCu4CxL/0luwZv7+Xs2vY32FTQzqd9LmUeFdOMxjYfcFoWNDyMri0oHzNX WW0+ZpgvxyklnPJBPkdFtwlgAv+7H9kPuUINvDjaiyYDph8DqExN0TGP8V81Jr3myosw uH+POtJK93PprjPTaN2wx73eUyGjRf6tfy+1FXyPqWsIZWyMZrtAHAd5QssGf3ycnOfw mprfDy43jeG5kY4FgXjaPMgCFecB7f4n4ozG9PhLYJBJkMepNwwzvYSodUPk0mt3Itka wGMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677764; x=1779282564; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=jxlTDmfL7zgwYuPQPoNgGF9X7KRsj/YO5X5OnwZ25t4=; b=OWhwjJMoN6dJmVplevWEOHAk76JLXR8jzQ31JI3dc5ZBbmLQs4hoRJ/q9lKdbhmLUE akVYNyIAfwVEiMOXiSrvHzskeJF7YXwiuvs3CGwfUPOY1iS3sRqZ1w8j2eWAzknulTxx n0TtZR2Z7erIcbdeBrQMfoHRlWczt/+AHiD8U79Mm1BIH+qSHc/mT0c3hAI/C+XXB1ii 3uBQbwRBUmJJVdnS+XBkl1rp9uBKAxxlgKGpUMBcclk2oQrwMP9PTN4hwZgTQAnxViKo GmLfvNbg/mFiMro7mjRkBFHmRt43BPLlHbuZweW8bOI1/2W9S6PbqXzfCVjfbsNJHcoH 7mgA== X-Forwarded-Encrypted: i=1; AFNElJ8jdoF3Jbd30rEiacsM9KqGrSvXGDjPSm1bldy/mupraMfh42ngoRaHh5Bi7T1bOCKowENnmS0jwyWEJLI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyFNdWKfi7i7AKstjCWrKNtbXrQhRlSdyKPOPk9+vgyxU+48r6V tKYoygAZ5IoBLXdOVFpCR5SpDnzCZ452goakN2xPWKFJTstg/jAPv3Ofb9uuYTtxr6I= X-Gm-Gg: Acq92OGoj9GP4fq5gT8AWezqGME/OKEHrFm+BMKoe2nCVcFKAAk8C1dae6VeBqTRsYJ 57We+uTvV9MO1OM6SkdfSmNEm8P5oErLIZUdeE8tlMOI5bp1P+1m8TcZA+hQTHpM8UNVUz65/JW s8n3prb6sjD75OLxBc9T4bJr6M/G3xGa64mbUU5+akvP5Oe9wp0iUi8NOkKZJHUbYEn/arCucph h3NDXC/676RX3lc1rBArAa3gfe8LbCobYs9z+ZI4dTrd9XkUTsl3BmofyxVlwdmwMJOwvG5ZdFL RsZe9J5pH64pKR8SWoc19ThRx8I5pNuQ3Z8uyEri9h5XV1UsSxtnNyso0GXpbYUda2bxTefUYeB 1cSgcJMwR8xIWnb0J9F2etQsYiOB1Cpw+2oIHve5Ai6Udeqz4GuYWedPggfGlX7h+W33EFMht2Z q2Z3rRETxWgNEW5hTKAZhxXJTnUNHxlEA8IBL7YuwNQwJc8bibsVEuYYOICRQ= X-Received: by 2002:a17:903:120a:b0:2b9:fb0b:b727 with SMTP id d9443c01a7336-2bd27714d46mr39301505ad.39.1778677763716; Wed, 13 May 2026 06:09:23 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:22 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 20/69] mm/sparse-vmemmap: Drop VMEMMAP_POPULATE_PAGEREF Date: Wed, 13 May 2026 21:04:48 +0800 Message-ID: <20260513130542.35604-21-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org VMEMMAP_POPULATE_PAGEREF is only needed once the slab allocator is available, so it does not need to be passed through the vmemmap population call chain. Drop the flag and test slab_is_available() directly in vmemmap_pte_populate() instead. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 40 ++++++++++++++-------------------------- 1 file changed, 14 insertions(+), 26 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 17d45dac4324..d7e9fb47f7ee 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -33,13 +33,6 @@ #include #include "hugetlb_vmemmap.h" - -/* - * Flags for vmemmap_populate_range and friends. - */ -/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages */ -#define VMEMMAP_POPULATE_PAGEREF 0x0001 - #include "internal.h" /* @@ -147,8 +140,8 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn, unsigned long flags) + struct vmem_altmap *altmap, + unsigned long ptpfn) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(ptep_get(pte))) { @@ -170,7 +163,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in * and through vmemmap_populate_compound_pages() when * slab is available. */ - if (flags & VMEMMAP_POPULATE_PAGEREF) + if (slab_is_available()) get_page(pfn_to_page(ptpfn)); } entry = pfn_pte(ptpfn, PAGE_KERNEL); @@ -243,8 +236,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) + unsigned long ptpfn) { pgd_t *pgd; p4d_t *p4d; @@ -264,7 +256,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn, flags); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -275,15 +267,14 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, static int __meminit vmemmap_populate_range(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap, - unsigned long ptpfn, - unsigned long flags) + unsigned long ptpfn) { unsigned long addr = start; pte_t *pte; for (; addr < end; addr += PAGE_SIZE) { pte = vmemmap_populate_address(addr, node, altmap, - ptpfn, flags); + ptpfn); if (!pte) return -ENOMEM; } @@ -294,7 +285,7 @@ static int __meminit vmemmap_populate_range(unsigned long start, int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_range(start, end, node, altmap, -1, 0); + return vmemmap_populate_range(start, end, node, altmap, -1); } /* @@ -370,7 +361,7 @@ int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, return -ENOMEM; for (maddr = addr; maddr < addr + headsize; maddr += PAGE_SIZE) { - pte = vmemmap_populate_address(maddr, node, NULL, -1, 0); + pte = vmemmap_populate_address(maddr, node, NULL, -1); if (!pte) return -ENOMEM; } @@ -378,8 +369,7 @@ int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, /* * Reuse the last page struct page mapped above for the rest. */ - return vmemmap_populate_range(maddr, end, node, NULL, - page_to_pfn(tail), 0); + return vmemmap_populate_range(maddr, end, node, NULL, page_to_pfn(tail)); } #endif @@ -503,8 +493,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte))); } size = min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -512,13 +501,13 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long next, last = addr + size; /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, -1, 0); + pte = vmemmap_populate_address(addr, node, NULL, -1); if (!pte) return -ENOMEM; /* Populate the tail pages vmemmap page */ next = addr + PAGE_SIZE; - pte = vmemmap_populate_address(next, node, NULL, -1, 0); + pte = vmemmap_populate_address(next, node, NULL, -1); if (!pte) return -ENOMEM; @@ -528,8 +517,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, */ next += PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte)), - VMEMMAP_POPULATE_PAGEREF); + pte_pfn(ptep_get(pte))); if (rc) return -ENOMEM; } From patchwork Wed May 13 13:04:49 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237835 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=djn+rtZw; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20870-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv2h5x1vz1yKH for ; Wed, 13 May 2026 23:10:20 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1m39dwz2ySg; Wed, 13 May 2026 23:09:32 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::630" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677772; cv=none; b=kok7WT1mFLzhWzqrbTdI1Yx87NRtITlSyk8SMyMUxTV39qppQDDYDVQzBDImywAMrpeaTjAwzeFtm0TK/urrJBOe+BkDUE35w6oO5fhiTIqh7kgx3eK2Vzl2pE4o8CKj83T8AG1EDf3PWZ46M/wsP1dJIqczYi2gpUI3sOkawUdtQXn/h5ZnraVLavA4Yp/QQ5OaVMotOTipsZbOCkqdiUQNOAxCfcgh64bqIRvWci1CIbxOHlGkJyu7S9HqKgXeKRXrCW87RddM3mLlOV6NHeupBXjPd9ei5oPwu9eH6KVgrXH7/gOgB4Q16fSRk4jO4dPmH+s4aypG32pJao32uA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677772; c=relaxed/relaxed; bh=G/meEdlZTZDHs3WMdtJ4iXM2mc4vrDnOJRWI7aUMJSw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R/4lMk3u/ol4nZ9UUDi+Xz2nIq3Ml6qRxpdvnHtTeiYkv+5TfWvys0v0cJ1Jjio+HQXotB/gZMw0awbtJl0Vber2gavj+QLfI2k+m0dtcU2vrR2Q7XGpnR7sKGafqLqOOgMfqqNEzau59z7V9kj+9Wxy9eWswWhJz3WKpeDD847W63o1t+Ahu1qRsJvSNkqlYgbtI5POXJs6rc6C/NK1iMYG46EV+1beuOFjZ26HTMt2cYs3f1CAB8KztNMmMxGvPVuBp0WsoIWtqJdU16BArgxg0iq+NRYexCDUgQWyN1J80ET7+XKh4AKij43vT/hZ5WAhxBZcwXo/kK7gfO+dRQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=djn+rtZw; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=djn+rtZw; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1l5NgDz2xSb for ; Wed, 13 May 2026 23:09:31 +1000 (AEST) Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-2bcd730e090so24803715ad.2 for ; Wed, 13 May 2026 06:09:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677770; x=1779282570; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G/meEdlZTZDHs3WMdtJ4iXM2mc4vrDnOJRWI7aUMJSw=; b=djn+rtZwDh3yKk1ZD5naOuebg9z/N6T0xVSEiONNWtJ2PZS+2eUQie4elmjQUoJ71z +yVpNQ7AfV5I/Hab0bQuiFuAwaPK2CKzPZ8w32YTSGykal1KkwjBr49xmx63qtoA6RNB 1T2R5Y66Yo6dD+EhUKMZACq9Ct0pU1DiTei/2TABNvfhKqjoX6ydUjIYLzLv93NkrNAS jvAc5Gg9EsCxQhGKvAfaPOibtXKDS4sGUB2tOBHauD8/clzAOc9Bd0gURQ8bG6H+MWxp guotdUPGESJjdZfOAmmlwH5KzelK0wIoIEGqVoUC6Y4uQbkdit6EF3hj3kegV1edrHhJ 8rdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677770; x=1779282570; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=G/meEdlZTZDHs3WMdtJ4iXM2mc4vrDnOJRWI7aUMJSw=; b=l1ED7UILXva5V/GuMHBuC8HHkWinqObVRA/cDgMD/mJMkZiBRG1CvheoCX3vfRI33b KyEEROCip5uOFyB2BL88rrtV9eOx1kHLFBYl/lpMugHMo2SE1Iq2OUxLvt+DnDxoSZcK 3pFTVsbKLla1utwhFH1r82zNTZjYIRLorJsaXcal0D8iJ0fwg+Xupw8oweqf08ymztXv Yhl3E04TWX2jXTZbBc/969IKXeOq1nQ/ynWNPqxs3segyGe/K5bx6Euhy+30bC09we0l a5fxGKjOkM13r/VRcsA20mUyi9WhEAL41NDca1JVpmRAe26WjZr3LcPyFd/zHChBBaXP uWpw== X-Forwarded-Encrypted: i=1; AFNElJ9jsHRe3972A2pLiXtmtlE2yBB3eTY5E1JYR7eslt2844lhm3gqAonyv7SfNzBTwnTpSkMZFwBZYg39XSw=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxCGFki6pvt7xou7jvQbvBnu6EUDsWhNXDjMxBRw1tFSO+dx0uI wdUcGzcM7hBobIOBe9hexv/CTa3/NQjrOWOks3Ni0ZKEx+XKwvHJoR6CbMO0eK0x7hY= X-Gm-Gg: Acq92OEh7OEyQOVaSUNfEqcs3Prv38f3gu0jmTrDNrHAAaxEvPH/Za8hkZ6tKMPQ7Yu x896/N4JM6zhaq7Kq+8r+rMWbLMVPJ1rvs553UaE8ONpWxhdIP61QIZtpqJ6S7QhGpXKcQKuCy6 9RwL6mA/6dQHDrdWVQ9b6VOGCYfdbuNsyo70mOqiko79a2Q0LAqPM9zbRNn9EZMQcRv/5XwCEn8 QhVBLEhXayG8XZF+DkrlnnkjoIyI0xelovFVF9A6ONgOGW08CF+nENbi3td9kHTFEs2wgN3gBQ2 TLKdxrRshSd3W2RD6gSxWra3GE7CH1X4FphdCwfQ+AJxl9RSa42qgF0h4ZvdUBREwjBkYHqaACZ QMhpJkXc4aDSLOhWTVXVjPC91KqkVBARh2ncSVgrC7a5XebVCAtlX0sz19pIfYbr9N/qSCy04On 4VHvvAW98nSQGhnsBNEjqRRqugzj39XDoQsRxOTI59aS0UDctFI7I7MOtd3vc= X-Received: by 2002:a17:903:38c8:b0:2b0:6f21:8289 with SMTP id d9443c01a7336-2bd30210f0dmr30869055ad.25.1778677769571; Wed, 13 May 2026 06:09:29 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 21/69] mm: Rename vmemmap optimization macros around folio semantics Date: Wed, 13 May 2026 21:04:49 +0800 Message-ID: <20260513130542.35604-22-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The existing vmemmap optimization macros are named in terms of tail pages, but they actually describe which folio sizes can use the optimization and how much vmemmap backing an optimized folio keeps. Rename them to reflect that meaning directly. This makes the names work for both HugeTLB and other folio-based users such as DAX. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 18 ++++++++++-------- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.c | 2 +- mm/sparse-vmemmap.c | 4 ++-- 4 files changed, 15 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 362e16497533..40b1cea98b82 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -107,13 +107,15 @@ is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) -/* - * vmemmap optimization (like HVO) is only possible for page orders that fill - * two or more pages with struct pages. - */ -#define VMEMMAP_TAIL_MIN_ORDER (ilog2(2 * PAGE_SIZE / sizeof(struct page))) -#define __NR_VMEMMAP_TAILS (MAX_FOLIO_ORDER - VMEMMAP_TAIL_MIN_ORDER + 1) -#define NR_VMEMMAP_TAILS (__NR_VMEMMAP_TAILS > 0 ? __NR_VMEMMAP_TAILS : 0) +/* The number of vmemmap pages required by a vmemmap-optimized folio. */ +#define OPTIMIZED_FOLIO_VMEMMAP_PAGES 1 +#define OPTIMIZED_FOLIO_VMEMMAP_SIZE (OPTIMIZED_FOLIO_VMEMMAP_PAGES * PAGE_SIZE) +#define OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES (OPTIMIZED_FOLIO_VMEMMAP_SIZE / sizeof(struct page)) +#define OPTIMIZABLE_FOLIO_MIN_ORDER (ilog2(OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES) + 1) + +#define __NR_OPTIMIZABLE_FOLIO_ORDERS (MAX_FOLIO_ORDER - OPTIMIZABLE_FOLIO_MIN_ORDER + 1) +#define NR_OPTIMIZABLE_FOLIO_ORDERS \ + (__NR_OPTIMIZABLE_FOLIO_ORDERS > 0 ? __NR_OPTIMIZABLE_FOLIO_ORDERS : 0) enum migratetype { MIGRATE_UNMOVABLE, @@ -1146,7 +1148,7 @@ struct zone { atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct page *vmemmap_tails[NR_VMEMMAP_TAILS]; + struct page *vmemmap_tails[NR_OPTIMIZABLE_FOLIO_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1f0a0e31d624..53448b05ca11 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3318,7 +3318,7 @@ void __init hugetlb_struct_page_init(void) struct zone *zone; for_each_zone(zone) { - for (int i = 0; i < NR_VMEMMAP_TAILS; i++) { + for (int i = 0; i < NR_OPTIMIZABLE_FOLIO_ORDERS; i++) { struct page *tail, *p; unsigned int order; @@ -3326,7 +3326,7 @@ void __init hugetlb_struct_page_init(void) if (!tail) continue; - order = i + VMEMMAP_TAIL_MIN_ORDER; + order = i + OPTIMIZABLE_FOLIO_MIN_ORDER; p = page_to_virt(tail); for (int j = 0; j < PAGE_SIZE / sizeof(struct page); j++) init_compound_tail(p + j, NULL, order, zone); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 952216a49bcb..e9906d32a64c 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -495,7 +495,7 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio * static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { - const unsigned int idx = order - VMEMMAP_TAIL_MIN_ORDER; + const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; struct page *tail, *p; int node = zone_to_nid(zone); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index d7e9fb47f7ee..39529245d790 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -318,12 +318,12 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * unsigned int idx; int node = zone_to_nid(zone); - if (WARN_ON_ONCE(order < VMEMMAP_TAIL_MIN_ORDER)) + if (WARN_ON_ONCE(order < OPTIMIZABLE_FOLIO_MIN_ORDER)) return NULL; if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) return NULL; - idx = order - VMEMMAP_TAIL_MIN_ORDER; + idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; tail = zone->vmemmap_tails[idx]; if (tail) return tail; From patchwork Wed May 13 13:04:50 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237836 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=A+ghXsRI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20871-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv2r2Lc1z1yKH for ; Wed, 13 May 2026 23:10:28 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1t2fwtz2ySj; Wed, 13 May 2026 23:09:38 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677778; cv=none; b=MltNpds1ecuqZNk8uvvyw1UAh0Zo6owTjdjadH6KIGNSxzu7NZiIatTMAFNTb4CragPvMXTn/tLG078YVw9pAeXi37nB27r47Z8dYezhZodE72JwdIJC7e2EYzWAFZu6uOV0mrdaJr8YPTwkQ5aFJRHoIUzrUof8RXdCUTNwyOpPSfHz/0Y13Nq5/9OyEGiVCZye/QE+6nIGoj5JbuX2HBSNOMcmqWiQhBWY3XoNtWX5k5zkaGe768QsDidr0hLnTeRxMdTmIAFW0XoAMNGvbWXUAb1N5I+o0SJW/YoreqW1yEbhcsN5H/g7Nd/K2bXkxUT4KF/2Iw4eJ9zoP7H5Mg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677778; c=relaxed/relaxed; bh=T0ssBedciqiVv0raptLCn49uzFGNVgYKzMgS4u/pGVs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=H+J1BZ40lMSO9Ar0R7HyJjGOxydnCCsgtVEdkExVgS8bPSb5x3NoxqZS2NNQuhdzg87rw7r8O5SvpB5LPCOSmwt9+m1BUI5JoVLXQo3CXI9mR9EIk6UT8UjtAnO9g+7Z+RzDSERskIhFVGMy0Kknnlq6DlAhGvLh5ISiaXvsi7t0r0UQu37SJTAuVyD07Owewds60ekGdYJCGIho8GlNSnB5czycqj1jV7DxWVcBI5uHE8dsp16wlLpVN2xNnKfIqxcGWpbUW9mLZEy/BC02aZSv7OI0iWZ9Vt5EfpDMRKzQyv3BiYRFctFJ2KU4ZvcLWDujMSmoT847my3gyQ0xFw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=A+ghXsRI; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::52f; helo=mail-pg1-x52f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=A+ghXsRI; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52f; helo=mail-pg1-x52f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1s2jCPz2xpp for ; Wed, 13 May 2026 23:09:37 +1000 (AEST) Received: by mail-pg1-x52f.google.com with SMTP id 41be03b00d2f7-c8016d642b2so3639543a12.0 for ; Wed, 13 May 2026 06:09:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677775; x=1779282575; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T0ssBedciqiVv0raptLCn49uzFGNVgYKzMgS4u/pGVs=; b=A+ghXsRI+5L7082/nQIgnhzz/28NCvMgTNIJ1lAYdGgtsI/TxQkd++JOCQhC32uAnn ZckmdH1K01BAYvWZdO09hYhDkROVlZDZJd71baRi4+7kCKoQCXbgUFwNHf7MQ+Yk/hdw oYaZuoBgXCppwTFiY34O92j7fhRiF5FV1nH690pyawnx3A67+SiyrYX6ZSaRgI9Oiw6N fboQJ9mq1x5uURrTQPXH+gaPPkvumJMdGGuyQeplYua9sST2Nf9S+upptTzDp8Lkwe/Z XxrliHR2/9fV/uPHSc1izhYLkYQN+g/oDy7IQrNGWT2O+f9oKTQBtCAf0ZwbOhXNEk+V QBWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677775; x=1779282575; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=T0ssBedciqiVv0raptLCn49uzFGNVgYKzMgS4u/pGVs=; b=Rna1iB9817ydx8KzD3E/t+GQV3FneJXqqHa56r/O8zDSiXtUyOhLq5FxaR4P8G/Pgb 4cxyaQCkh1jYEAx54yG2Mrb0TSh64lOiool+pGjq7JYilOOHtYy05sefBOFtuwZEPzJM Sq+Wgou6YJPLt/4Mrim5coZMcUgdH86vy7hANVHjYrRGU6kGC8g3z0JpA0oPAcVrRlOy +4ly/a6H+pBUE+00EX5oBHBh0IKUJWeewNc8E/MDQHd0uCA+b9jzU/gaEQFvnf/EpwDw cTp7lreYZwjoOTcR/lI+pWqyLq7uqdg7Djn1iHpQQaG6UngbpjQCvn8lAnquHXuWFvLX takQ== X-Forwarded-Encrypted: i=1; AFNElJ/u2QKLa3gREeNQg7AkF5i4DBBLJ03HloO4zvU3XJQXrtq3M3yZVWWR0akwL7ubnnKIwnk097duTXSFUwg=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwQhwkJXzjdqSAnR+Y2V/a8+8PS4wWoC0x3eAOZvy3CROeNxsOA crGz9MyoUqxugeT1f4D2/EFiSdtznOOU/58usIdcZfcltooTuw2lKQSYjyCnObs8KeX1X5Sdrur bFmfDGeGQmg== X-Gm-Gg: Acq92OGTTWN5vpNZs3lke49Ki8ydP4+SCkmQm6DAaf4dCbltKiKqsARFNiUTJjDFKLI rFl5oJ2BO3n561329zoPlZEAD37rpxbOt+2YUC4HKvajwwuRwvfaI/QLQWWjmzD4Rsakqx+41yT SzOs+PzjGIrA6RrRF+blv9QrwoiO3zaiELA3SE2huYrQ+XpXi2UZG7w04TG71by35gPPp4xHjwQ zm5S94J06UjKWgbVnepYX/zrrUDogZvrl5hO7IToUV4QKWGpXH3bQGcI4fqMXS1tb3fPHWhDRaB AqWmwRLEqITxOtLxTP6zHqTvSFH3cMvjfKpi0ZCA23AQFBRjryKigX/LpjdgLjLqGbFUQst++Kx +UXhcI03BuLuzXUC0eOSO1S7Knebw7FjFsyskyrwRdimlfRgUZsSo62/uevqvmFip/y1Qh8MiIe te4MPHrxpCS4GzyXMHmw/TbfGQq0qBY/L3xXfjOvaUb/FO/KpwLSZELyGGIDg= X-Received: by 2002:a17:903:320d:b0:2b2:5070:8b with SMTP id d9443c01a7336-2bd270fa021mr26473735ad.1.1778677774883; Wed, 13 May 2026 06:09:34 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 22/69] mm/sparse: Drop power-of-2 size requirement for struct mem_section Date: Wed, 13 May 2026 21:04:50 +0800 Message-ID: <20260513130542.35604-23-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org struct mem_section is currently forced to a power-of-2 size so the section-to-root lookup can use a mask instead of a modulo. That requirement adds configuration-dependent padding, especially with CONFIG_PAGE_EXTENSION, just to preserve the lookup scheme. Drop the constraint and use a plain modulo for the lookup instead. The divisor is constant, so the generated code remains cheap while avoiding the extra padding. It also removes an unnecessary layout constraint from the type. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 8 +------- mm/sparse.c | 2 -- scripts/gdb/linux/mm.py | 6 ++---- 3 files changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 40b1cea98b82..ae0271eaec05 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2027,12 +2027,7 @@ struct mem_section { * section. (see page_ext.h about this.) */ struct page_ext *page_ext; - unsigned long pad; #endif - /* - * WARNING: mem_section must be a power-of-2 in size for the - * calculation and use of SECTION_ROOT_MASK to make sense. - */ }; #ifdef CONFIG_SPARSEMEM_EXTREME @@ -2043,7 +2038,6 @@ struct mem_section { #define SECTION_NR_TO_ROOT(sec) ((sec) / SECTIONS_PER_ROOT) #define NR_SECTION_ROOTS DIV_ROUND_UP(NR_MEM_SECTIONS, SECTIONS_PER_ROOT) -#define SECTION_ROOT_MASK (SECTIONS_PER_ROOT - 1) #ifdef CONFIG_SPARSEMEM_EXTREME extern struct mem_section **mem_section; @@ -2067,7 +2061,7 @@ static inline struct mem_section *__nr_to_section(unsigned long nr) if (!mem_section || !mem_section[root]) return NULL; #endif - return &mem_section[root][nr & SECTION_ROOT_MASK]; + return &mem_section[root][nr % SECTIONS_PER_ROOT]; } extern size_t mem_section_usage_size(void); diff --git a/mm/sparse.c b/mm/sparse.c index 324213d8bdcb..9457a4d6a6fc 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -331,8 +331,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count = 1; int nid_begin; - /* see include/linux/mmzone.h 'struct mem_section' definition */ - BUILD_BUG_ON(!is_power_of_2(sizeof(struct mem_section))); memblocks_present(); if (compound_info_has_mask()) { diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index dffadccbb01d..da4e8e9655a6 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -70,7 +70,6 @@ class x86_page_ops(): self.SECTIONS_PER_ROOT = 1 self.NR_SECTION_ROOTS = DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK = self.SECTIONS_PER_ROOT - 1 try: self.SECTION_HAS_MEM_MAP = 1 << int(gdb.parse_and_eval('SECTION_HAS_MEM_MAP_BIT')) @@ -100,7 +99,7 @@ class x86_page_ops(): def __nr_to_section(self, nr): root = self.SECTION_NR_TO_ROOT(nr) mem_section = gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT @@ -249,7 +248,6 @@ class aarch64_page_ops(): self.SECTIONS_PER_ROOT = 1 self.NR_SECTION_ROOTS = DIV_ROUND_UP(self.NR_MEM_SECTIONS, self.SECTIONS_PER_ROOT) - self.SECTION_ROOT_MASK = self.SECTIONS_PER_ROOT - 1 self.SUBSECTION_SHIFT = 21 self.SEBSECTION_SIZE = 1 << self.SUBSECTION_SHIFT self.PFN_SUBSECTION_SHIFT = self.SUBSECTION_SHIFT - self.PAGE_SHIFT @@ -304,7 +302,7 @@ class aarch64_page_ops(): def __nr_to_section(self, nr): root = self.SECTION_NR_TO_ROOT(nr) mem_section = gdb.parse_and_eval("mem_section") - return mem_section[root][nr & self.SECTION_ROOT_MASK] + return mem_section[root][nr % self.SECTIONS_PER_ROOT] def pfn_to_section_nr(self, pfn): return pfn >> self.PFN_SECTION_SHIFT From patchwork Wed May 13 13:04:51 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237837 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hN4p/4k2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20872-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv300JXKz1yKH for ; Wed, 13 May 2026 23:10:36 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv1z267dz2yTH; Wed, 13 May 2026 23:09:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::632" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677783; cv=none; b=elNDIJFggm8ei7CclFgd/Fuz/L9bILuLfkws2gXD/zg981LpC/g1t7YPJ/kx3RjbLpWhq5efNdF35gVoJK8p5kplbm0I4l1SQYIg6XvsxDsK20G752VokqoJLliUIIhLRDceL6j3rVfyLGkPJZ2dRS5j9Xh+1Jgz+wcAvdLazDQ3bEsT2VoRPr30/D2aXYn7DEJL2CbxpyuMQnr+zpNmVW4WMCc4E8ehHpwdIkmNePzo8MIomb7EmMMXA5B3CO6t1LH/7rw54KRSBIwjs6FMwgKRib7TP2GSvpXU/7Pr0jI7Ko1gA095Eg1XXPU8WS051V4GTXamrTnctztCDSfJZg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677783; c=relaxed/relaxed; bh=thKeB2NiafjiEdbVk8XkvNjvthrX3e0E5pBtn2l+Jas=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UrPW7WZT0rctBE1u5r8fXemXMkqX1gY/ejVb9NCVXMH+8knx9R055y5WXTGDX8cEXfYz5RevcUH1EzHE88BZP7EBZ8uYICCGHGvaBWtA5H6MXkGrqf+yhYcrYkq6vUMvvklxT8xjhu3lEJPPX0Z5CEkLaUr4n1uF1iVEyDJTMNcr5gzIgV5GKyM3WzVjPjmXLGyp2kg0aff3NWspmkGhGINQMKXK3Wv9XJTtnFBc+UyAualfpjZ2D4oi5fXU/oimJ/KTPnHnt+LE5c6Wfn5s+AxdtBI6FnGP0/OQuvQ+48dXePIOfU3j1lAm+LVo6R1yYo/kEnZc/3cH+oYZZfe/Kw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hN4p/4k2; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hN4p/4k2; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv1y470Wz2xpp for ; Wed, 13 May 2026 23:09:42 +1000 (AEST) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2b45cb89f7eso43095335ad.0 for ; Wed, 13 May 2026 06:09:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677781; x=1779282581; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=thKeB2NiafjiEdbVk8XkvNjvthrX3e0E5pBtn2l+Jas=; b=hN4p/4k2uNEBWjbpslWSICE7uch3NjcNWiZVWw8PI9HWbJ8iUw7kRgkAUBG7jTEgBm 1xNM6GdcJAC8tNVtP8UbBLOl5RB/MwQQLEz7FnMry7Ie/Cdjp98UkLlgeYKpBy65EhPt Z30pZpT1FjFVgsuYcsWY2RKLmTQ2FjgRPXHz7m1nXqXlKD6LtOhZUmvaqgALVzRTjLjs N2JPBjMWvfsUTAliEYGHqhwrzRuOiMs3skXcxsZmnb5KabDwRpAU+QkBxvT8VzYqO1I+ 8/7rJK45A8FwUt+HoAuxSEAWKnT8uLxhkydZEfMadTJqbJnEdCAcY4RkmgjSEP2o8NDM JM3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677781; x=1779282581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=thKeB2NiafjiEdbVk8XkvNjvthrX3e0E5pBtn2l+Jas=; b=a2DVVULsUMAVpE/ykaIPKF4QT/mz8eETr52eOVvf53wupQ+JrgOVWKXi3d/7zW6GWY 15t5y9HGz+2JqP3dlDZaJzgQd1cFAXtQkbEMp6U7sJoWf/0s1txMvr3CYZ9LHB9B5Avg wVYCpGN6IIbsg1wS+gAMYFLyzUSW5b0DpxCydfEaKkJXL0B0UqU9IfnzlchxAPTX3tma VuqSS9P93E/5TwxgH+8GD6VnxVDed/4MQaB0OTU8BjyiWXU722Nip6hHtaaNKU7v+Cr5 T3J3y3GTT7HTGk2EExS++zW0b3PUgaxV+zY6Y/MIyvbhm3/qocMRn9EWtNrqDVqVhu0O fmHA== X-Forwarded-Encrypted: i=1; AFNElJ8OZdtOy42QmHUd245z9efpO/O9USq7NYpgUS/c+FFIV2rBC8stgnwhbOh8hlgM2QsNawNiFkGfj+0WUyE=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwFMbQU0tlYr2BE8Ntx6YuOChSfEJ/OL7f0FqewEGXJM7PldJGN aIMXMoLl8NUcxJLF6Rxyj9+AExI0jv+2DlrkoPKwKnDBwDnkCGaGT9gBvn5VTsC+SRQ= X-Gm-Gg: Acq92OEhmO0ImZEy7voA8agfwOFAkUR6zn4fQs6a7lRRmmc0OSeR11QKMD980yZbhz9 Gx7FO7Fi/KfE8hKhScVy7JYpCRB55LHvtYYxX8yUzppMhSzjlrDZBYxeR1xLM7+gUXw9Ojuz8uc 9iYghHsEKeXTHdfEA1damEYO3fylFE2cTXbyYTqJdCidve7Hx6JrlYA6aZbWgS7uiiTjuqbRP4N ukTc+6UYLb/TZfvhk5x2+wQrV78aVN1/Kvv0hB0gHYMuptwLrUUndc2DykR5NkBpbqUWaPwF/Y3 qf/9mbsxKnc9YMcZCWqNQKNyQMatSGBmkNciApTkcQfkBWP8BpRVJpxfY7YkCgoeagPWwpFSO/M ipzppNJhgBC09KD/ro7fesubuTYX4QkLPfIRWmVIXNrbwvEJOsjtgk3uaahGl513M/No5Aa22aW tgJKIy3tSMH0mDRCixSoa4uChu12JXkHeU0XvkWeBxo3j3eBcCAQAZJo87MJw= X-Received: by 2002:a17:903:2ace:b0:2bc:7486:21cf with SMTP id d9443c01a7336-2bd276e6d61mr36673085ad.36.1778677780397; Wed, 13 May 2026 06:09:40 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:40 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 23/69] mm/sparse-vmemmap: track compound page order in struct mem_section Date: Wed, 13 May 2026 21:04:51 +0800 Message-ID: <20260513130542.35604-24-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HugeTLB and DAX both rely on vmemmap optimization, but sparsemem does not record what compound page order a section is populated with. As a result, code that needs this information has to open-code separate handling across users of vmemmap optimization. It also prevents other memory management code, such as struct page initialization, from skipping initialization of shared vmemmap pages when needed. Track the compound page order in struct mem_section and provide small helpers to access it. A compound page larger than a section naturally carries the same order across all covered sections. This is a preparatory change for consolidating vmemmap optimization handling and for letting later code make initialization decisions based on the section's compound page order. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index ae0271eaec05..6f112e6f42bb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2028,6 +2028,14 @@ struct mem_section { */ struct page_ext *page_ext; #endif +#ifdef CONFIG_SPARSEMEM_VMEMMAP + /* + * The order of compound pages in this section. Typically, the section + * holds compound pages of this order; a larger compound page will span + * multiple sections. + */ + unsigned int order; +#endif }; #ifdef CONFIG_SPARSEMEM_EXTREME @@ -2224,6 +2232,17 @@ static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn = (*pfn & PAGE_SECTION_MASK) + (bit * PAGES_PER_SUBSECTION); return true; } + +static inline void section_set_order(struct mem_section *section, unsigned int order) +{ + VM_WARN_ON(section->order && order && section->order != order); + section->order = order; +} + +static inline unsigned int section_order(const struct mem_section *section) +{ + return section->order; +} #else static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) { @@ -2234,6 +2253,15 @@ static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long { return true; } + +static inline void section_set_order(struct mem_section *section, unsigned int order) +{ +} + +static inline unsigned int section_order(const struct mem_section *section) +{ + return 0; +} #endif void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, From patchwork Wed May 13 13:04:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237838 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=cDMhsTwb; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20873-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv375npXz1yKH for ; Wed, 13 May 2026 23:10:43 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv245m19z2yV6; Wed, 13 May 2026 23:09:48 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1036" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677788; cv=none; b=DUZRbtW/ScScCI0/X/bBjosQPsqHwDYiZIoI3fYBOUglckU0r7VGvF7ATKkxesb4I91bHv6PsbW4GexWfCyoyu0xRt36tLltqZyUDLZcwUOEmB4Nq/P0gCG9X22o6KFc0GGEzAPq6L5LIs1IlZmTmfxu0qqorc2f2iIQ2Gr30bXz1ikredNONcys61BzOZCPdZ58ogDv/2GjWuZsjLRhMPn+kbZ71PUaRC9OkNbqrlWOMsMHq//dL87wsw1zxK4RctqI8NuM1ppInw7WQeohkNg4WgWs7/zFiAHUpKNlxgOr2TsX8trr2LLqLg7Kd0rcQBScy0kDrHnOoO4WUBFnRw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677788; c=relaxed/relaxed; bh=3cDXE+87tm7ASxvvK0pHPqOmmyuztiqWVG9E1ozeyTc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iwhEXZdzeiwax99zotGU6m3w/X053wLb6md05l3d6t2BegXKpL+7DMsKlicH3vvbQCo9iHi2vju/xBWn5Jz4ISexUnJhLERIBAFNtPw+3SSa+w8FNGvk1kTnIPY17fMa5j9f5BuzVcPcTx6dcy7TY31eLBH9n4miI+7/dS1v2yQGQSdGwoWG8KFfm7vfxh2rkLuqqrPBKALeBwO/FA6/4f9fPUoOXj6jTtJ5t81QbGHTXoAq0z3d+/1DnbaRz2NzFO5lbekpweiTGxYmHpsvx2JZUNj6BLPIgoDdm7sniHtVUxr8sJSGS/dPRfBARhOW6e9affasPBV2/QYungCbbQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=cDMhsTwb; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=cDMhsTwb; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv240Rxmz2yTQ for ; Wed, 13 May 2026 23:09:47 +1000 (AEST) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-3664df30f53so3312794a91.1 for ; Wed, 13 May 2026 06:09:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677786; x=1779282586; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3cDXE+87tm7ASxvvK0pHPqOmmyuztiqWVG9E1ozeyTc=; b=cDMhsTwbtB2tLOGZSJHJqZg62wycuIzvYEvCKm6+nyWr7gzd2r/oWj1tAa8jMZxddj Eqk7ryhOtsYrpuEk9QOsoH6mHwtiPDdktKg2DqlW0GxOi5jApIauIK92m5vDHsFmnt/A /mZX2tlhHt745yWsVBHxHpOM9+dIbPq/0CcW94d/Q0w3/VfiHhIiOBWUcmgptLM+TBCf UlYj8AAfd/PfmwZ/EzO2O32lWuN0Wa4S28anjC4ZFxkqmGRWKXDcsJ/gG3/hlsdgOr5e AsrXY4eqaSWuwgA+FYCEEmtgOBgSWd+YlZMuiFieaIuijkuMC9Z696EEVK0H3zVLxIoX L1FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677786; x=1779282586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3cDXE+87tm7ASxvvK0pHPqOmmyuztiqWVG9E1ozeyTc=; b=Nc4rzJIQbQRlq5PVlTB2SjD5L2E2Nv3Mm+eVVTTMsdPQc159IqIZLk8Ba6F9/le8u4 EQuoZ+D9dX7ZBN23HW+TOD8ipoFgIoVVoXpuvImN5Fc0Z/eDZRSD+lvzt+GT8ZzmZToM JLjnZqRs9tX3KlmyaM/5DmE5eGyGmaI7kHz4eL1FVT1scjuynRYrzk6qIb6VwkhA5pQu H58VhlAE5AhSdPEFBJzr6VI/6YqEnXgA0fc2c4j8UTFOJaiPn3EfHQORbWoLP3YLbyDR 0slkR2TXdKVd6L4x8pNqh3+1Pm3b1WL0TCxDKQRwx54Uhj4apHjlzuGzOgKUw9YIWptq r/Gw== X-Forwarded-Encrypted: i=1; AFNElJ+cE3fvq9MNQy4xhzgI/MqubAvM6XDVP931u3vkeXiGyJc+cJoDH9oN54mKGP/U9CEAevbXRiHURShe7iw=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzJv/evGh3Bp+rl7SgftGExjijmNMwHRPoPpY9IGBkG449CXOto ovxKDJ5MxEdWRPWD9UgdAf2nKwH7H3VIfpT9xxwiw2HxeHc8dHPEwlB8tAS17O5XB9o= X-Gm-Gg: Acq92OECqbsm2mNIGjxGegoj9sPJnWXuO4vIPS+hZWYHbJzFVosyUR5CuACcEv+a+0N RmxV9NJ4ZfMkSn34et28r05XdBn5B6/BRRg0Rx3Frii359ROsyYKwo9RQb+RE9tvMP7MCDanKqZ uOaj+Iah9NToNMkD2exf6I3ZpThsshHKUn6hthvFuU2ecyMsOndlwq/wJ7CxYgZB16bBYsilAqF U/9lF6A2XlHSvctSA8TdeOn4rn/bBsRNCJ3QXqJ9Q6oelVA8uciJnzSjwWHVyxcaT4RILtzuCbO QZJ2h3YbbwTRVeujCh2JX7W3+CJ9KkhO7V8kFJUZg//pbHkJTd9tjKfieAD2GE4A7e1RqmDgZ+M dEZ0KPHoDtuOfi0o1VSpGjCJLcuX1O5rkssZbT4G2C8hW+d0adic5rGhL8rZQU2JnyuOdoN4m02 w9/5yy2qbmPvJvcKjRz9q1V2tdWNsobpGihXeJGbgIwPblAIjnMDJMClBYMJM= X-Received: by 2002:a17:90b:580d:b0:368:9021:3195 with SMTP id 98e67ed59e1d1-368f3ab5cccmr3651001a91.9.1778677785949; Wed, 13 May 2026 06:09:45 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:45 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 24/69] mm/mm_init: Skip initializing shared vmemmap tail pages Date: Wed, 13 May 2026 21:04:52 +0800 Message-ID: <20260513130542.35604-25-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org memmap_init_range() initializes every struct page in the target range. For compound pages with vmemmap optimization, the tail struct pages are backed by a shared vmemmap page. Initializing those tail struct pages would overwrite the shared vmemmap page contents, so users such as HugeTLB have to open-code follow-up handling to restore the metadata afterwards. Use the section's compound page order to detect struct pages that fall into the shared tail vmemmap range and skip their initialization in memmap_init_range(). Still initialize the pageblock migratetypes for the skipped range so the surrounding setup remains intact. This is a preparatory change for consolidating handling across users of vmemmap optimization, and it also avoids redundant initialization of shared tail vmemmap pages during early boot. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 9 +++++++++ mm/internal.h | 16 ++++++++++++++++ mm/mm_init.c | 19 +++++++++++++------ 3 files changed, 38 insertions(+), 6 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6f112e6f42bb..5fc968bac1f7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2264,6 +2264,11 @@ static inline unsigned int section_order(const struct mem_section *section) } #endif +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return section_order(__pfn_to_section(pfn)); +} + void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, unsigned long flags); @@ -2404,6 +2409,10 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #else #define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid +static inline unsigned int pfn_to_section_order(unsigned long pfn) +{ + return 0; +} #endif /* CONFIG_SPARSEMEM */ /* diff --git a/mm/internal.h b/mm/internal.h index 4a5053368078..1f1c07eb70e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1004,10 +1004,26 @@ static inline void sparse_init(void) {} */ #ifdef CONFIG_SPARSEMEM_VMEMMAP void sparse_init_subsection_map(void); + +static inline bool vmemmap_page_optimizable(const struct page *page) +{ + unsigned long pfn = page_to_pfn(page); + unsigned long nr_pages = 1UL << pfn_to_section_order(pfn); + + if (!is_power_of_2(sizeof(struct page))) + return false; + + return (pfn & (nr_pages - 1)) >= OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES; +} #else static inline void sparse_init_subsection_map(void) { } + +static inline bool vmemmap_page_optimizable(const struct page *page) +{ + return false; +} #endif /* CONFIG_SPARSEMEM_VMEMMAP */ #if defined CONFIG_COMPACTION || defined CONFIG_CMA diff --git a/mm/mm_init.c b/mm/mm_init.c index c64e5d63c4ae..3aaee1cf7bf0 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -674,19 +674,17 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ -#if defined(CONFIG_ZONE_DEVICE) || defined(CONFIG_DEFERRED_STRUCT_PAGE_INIT) static __meminit void pageblock_migratetype_init_range(unsigned long pfn, - unsigned long nr_pages, int migratetype, bool atomic) + unsigned long nr_pages, int migratetype, bool isolate, bool atomic) { const unsigned long end = pfn + nr_pages; for (pfn = pageblock_align(pfn); pfn < end; pfn += pageblock_nr_pages) { - init_pageblock_migratetype(pfn_to_page(pfn), migratetype, false); + init_pageblock_migratetype(pfn_to_page(pfn), migratetype, isolate); if (!atomic && IS_ALIGNED(pfn, PAGES_PER_SECTION)) cond_resched(); } } -#endif #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* @@ -916,6 +914,15 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } page = pfn_to_page(pfn); + if (vmemmap_page_optimizable(page)) { + unsigned long start = pfn; + + pfn = min(ALIGN(start, 1UL << pfn_to_section_order(pfn)), end_pfn); + pageblock_migratetype_init_range(start, pfn - start, migratetype, + isolate_pageblock, false); + continue; + } + __init_single_page(page, pfn, zone, nid); if (context == MEMINIT_HOTPLUG) { #ifdef CONFIG_ZONE_DEVICE @@ -1142,7 +1149,7 @@ void __ref memmap_init_zone_device(struct zone *zone, compound_nr_pages(pfn, altmap, pgmap)); } - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false); + pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); @@ -1982,7 +1989,7 @@ static void __init deferred_free_pages(unsigned long pfn, if (!nr_pages) return; - pageblock_migratetype_init_range(pfn, nr_pages, mt, true); + pageblock_migratetype_init_range(pfn, nr_pages, mt, false, true); page = pfn_to_page(pfn); From patchwork Wed May 13 13:04:53 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237839 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lC8x25jZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20874-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv3H6MHpz1yKH for ; Wed, 13 May 2026 23:10:51 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2B1lW4z2yVM; Wed, 13 May 2026 23:09:54 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677794; cv=none; b=OteIVCOKM+Kw+znmMD9jaKo81/WtcNA0PRhiRV+SSrI2+K9Cyb/10I5KBKJHCHjT3+sfhJlhId9UroHnGCMrNR+jYrnZC8k8ecyiP30nuOd+M63+Xz64PSKV4IFp6U/zRvW7L53hZgUNgVU6qnFtsx6tncETkz0oawPAm6azIPKTyr97hd4XoUmHG9HMZ8vbiwIbWFBSdg2HP2DHh6OAlrAlMChAZ6WFeUwe5xT07+uDbGoWICnUf62RvwHtYTvVtR/o2p1IMg4pN4uNCI5bF9D1rsVkG8v2dMGOgilNPZPf25oXB3tndWDb5kaKDiiWrgI+ZChqR8TvRJRdKPWNiQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677794; c=relaxed/relaxed; bh=1fKIdS9sH5LMxvvJtquNIgVu8JwU6IdO6x0CAgaX0p4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CU8UwIOnT2gAUXnQt8gVNRrzgeJpDAnlQiLR0ZMKsxiDk5wnnh/zxpTiGeZcS57OxDbqVRGbQg/VQAFj61NFROykG3XUhBdAxCc6DOanNMff/vwH1viOVUPSiREvK3ZZN0Xvga18I0+HLwGe0CbpTvVku1GugCi++p21lqRmByDK9CEZl9vbjKDTJVYWBpAQUY1sJkoKo0hY5Ok4iKusE7l3xlytgTFu62QSj+7IumMNuNDGqImYONqwUC0or6zPsyn8NR/uKbxp+EUCcyrUfK3hPTGlHMmt/WHaZY51P0UHcs/ziXgHsUNzpaN40tLSUNN3NoDS9Vox4KWXWfAvuw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lC8x25jZ; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lC8x25jZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv293qsPz2yTQ for ; Wed, 13 May 2026 23:09:53 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2ba17c8cfacso68319535ad.2 for ; Wed, 13 May 2026 06:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677792; x=1779282592; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1fKIdS9sH5LMxvvJtquNIgVu8JwU6IdO6x0CAgaX0p4=; b=lC8x25jZB5nQioWDh4l2mGI+H5vLiRnL2QH9qh3e/aOX2WLiVatjGbsUbFsJDRWeK1 oU0hX0XF3P6YL7TQbbiqjF/3OOzmlqgGtkluprPCyWrGh2poMP0GIh7mdClGOuewxJpS X5TSS/s7K+zfd5+YQUfhkrQ5v4Dw1JteN+LWRihZC+sWaHukDhKgYCnbED4IsoO8YaxU QYAecZfLF8Th5+DpJS4s6WA+U5zfjXp5PV8eRfUGgl6RvzXHYdffVzTQUgXG5FQm8nwj V0lGMgYxRYvsrR7fevE0MwXl00yTjst6+HkaFGuSPNKKqzBZSJ0wKXBaAdl/T5bBlYpw 440Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677792; x=1779282592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=1fKIdS9sH5LMxvvJtquNIgVu8JwU6IdO6x0CAgaX0p4=; b=ccCwWCrEkm4MeSNgy7wIyImInviD4DrM8gMkKYxzEZ5Imvbo5HqJPmp7lhx0I1XlwZ lgG4xCUbMfapV5fGC7G0I+XpsTF0PXlxgZ4yIoPwYI42+n4TwJC7W3JNJI0IUTYGBYL9 x+FMiPktFLmEnMxvDY3Wr8D7Pf2CyVmbhYr3M07GBqVuY5VC8erOt4tnbClc1Hm5xs2V 7c5nuE53uATsWhRJiVINaWYXj++h0lQ2bxDWFhzAiRhdfAUfik9a3/6GUsmNDsmJtHBi Ed9U+6mrO57EwhIERgp+z8IYvh8V6QhbdhtE2q2HUUNaH+IBk8wFrRoJw0PjKCKk0bIg CghQ== X-Forwarded-Encrypted: i=1; AFNElJ9S1SGJtR2RsydJaVlveWdzi4yyJQkddtbMdErt28LCpfdtkTtd5WIQSsJBrRRV9uRtxBpEC41OwI1wjtw=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yw8cMxKgNjjHHDLe3jQxYgZR4Tg6scnHAmsUU8Et+yx/SX6+aev hHfzyZuLEkq2/gpVcoOYTKg9dd4SU5VmpYyXqYvfdo+8ho2WdcevtjKD31429Q3OKjE= X-Gm-Gg: Acq92OH396jZEMZCqjnnCvyBn9DbVsVhcVtsIiFgeBrmEDt2/tXjzXCBo5Z7kl3OHrV yCRU65IL3grPyBTryFzGL3Ho23S6+bOHDFzW7dpy02c1y8IFRwHLHpFni96a6FDqwiT/7lc29qJ LBaCGQ7FRJcIFsr1OlLwrS/qFoBYH1ML7veLxMYwmr6ndCp6kms4kKFiBSbsNOLyWmkzp2duOp/ QNrJvoEWKyJh6rk3e6vayb0J5+QHZ/tYddaGJSnXPvM51DGg7T0SsRI7VBi2xvJUEOWie6+r8ax PcTHGU+uksPOneuq2g0Kt+tpsk3XSqgulBKTmIpyet8gFSO9+4duX6y32WmdBCmAZA8vtui2RTy HQCMF+NrdTLZVetIw8fpNlrRx8Bey7gg6RiJT12Nj4d4EE5Hf0UZjc+c83HdVPKalF6pz506BNy thdRuvA4Fg5gsu6EFUbnGXWX3Y8mUX0BxVehobfODPv2jtzga1Pbu71Cff9VQ= X-Received: by 2002:a17:903:b06:b0:2b2:50bd:83b3 with SMTP id d9443c01a7336-2bd27176414mr37032205ad.10.1778677791500; Wed, 13 May 2026 06:09:51 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:51 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 25/69] mm/sparse-vmemmap: Initialize shared tail vmemmap pages on allocation Date: Wed, 13 May 2026 21:04:53 +0800 Message-ID: <20260513130542.35604-26-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The shared tail vmemmap page allocated in vmemmap_get_tail() used to be left uninitialized, because memmap_init_range() would later overwrite it. That forced users such as HugeTLB to defer the initialization to their own setup paths. Now that memmap_init_range() skips shared tail vmemmap pages, initialize them immediately in vmemmap_get_tail() with init_compound_tail() instead. This moves the initialization to the point where the shared tail page is allocated and avoids relying on deferred handling in individual users. The remaining deferred initialization in HugeTLB will be removed once it switches to the section compound page order mechanism. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 39529245d790..60d5330a8399 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -328,18 +328,11 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * if (tail) return tail; - /* - * Only allocate the page, but do not initialize it. - * - * Any initialization done here will be overwritten by memmap_init(). - * - * hugetlb_vmemmap_init() will take care of initialization after - * memmap_init(). - */ - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); if (!p) return NULL; + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail(p + i, NULL, order, zone); tail = virt_to_page(p); zone->vmemmap_tails[idx] = tail; From patchwork Wed May 13 13:04:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237840 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OVHyTg7o; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20875-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv3R39QGz1yKH for ; Wed, 13 May 2026 23:10:59 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2N3Spyz2yVd; Wed, 13 May 2026 23:10:04 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677804; cv=none; b=PCzdJ6/vmtUBJBUQ9rJkgWf5dOXjUhA+M5BBx4bQjkdGlL1//fyrCvGcBG+eJYx6p7q7XfrieR9Ls9KU4MawvoDqit0NwooMu+CSE0dq8B7I0biyY8smtOaAldaEPM2v/angEWGBhJTbLwtEoEFpHS2O+PBw5jeRh6jUDd7VdDbtTzyMIW9z0p2BVk0Wxu7O0BqRYKCvne+rXs30ejme/pwFDSvMQtWkEUJSN1UegbYABEKzJcmJmhw2WEh4xBaGBU6cJGlDfnJ4JJAvOmyynoxiX4flNgJQJlmRGossR3+XTM3rRNJ3GgR9XAO0JbEUaEMYM0AdvaASXw5JAsQgjQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677804; c=relaxed/relaxed; bh=TcJbixi8CH0+fCoBl2UtMRXMhZzJhhVobMYZVtSUs0g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K/bHxkzGPnZd8S3XHmFs7AdcbZaxFtD1OYuSOPqBGbxTV8b5n/gUuEXKLRHZgSV7AVHedUR0kalmoyxL0bcuBPpvPTlhNI7flPNQqJn80o9jhmS+5oNaomvElNebRu0F85Ewc6t4ZcCkFHvehhx37smGFx9SxjLH+qbawvwOQDVlxCFh1HxHIVbcBIZEFJ2pzxA/o6ueb2f1bVTIXUxRRTc0Dzt6tUWN9QuPzt79w3zqYNj/k0xx6Zo4sJW7991aTQD/bhwLpxVNmrCnVd6SkZoqzvb4MQvolE4tWg++o4Rn2b+ISJtMvvyoT1/v+4FvQMxXlLnt49LE5Ap/RfDEpA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OVHyTg7o; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OVHyTg7o; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2M4gPHz2yTQ for ; Wed, 13 May 2026 23:10:03 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-2ba4efedbeaso47462435ad.1 for ; Wed, 13 May 2026 06:10:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677802; x=1779282602; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TcJbixi8CH0+fCoBl2UtMRXMhZzJhhVobMYZVtSUs0g=; b=OVHyTg7osLvQE0GcgRQC7VhyLt/0NF38M8lrEoJjwQvCULx+Id8rllbDXBtJlJUCzd Bv4A/y2dYhOMtyo6ucteuJJu6fHnx10UexYTcgFtHYKZyTBNk+Xz5UQfg09ItYpbuvZN oBqkUnLuqOg0BgALZvGA0y95lJxJj/zSUd2TbJye8CtOEeo82sOdXQ6nTRuM6t+GBAMw F/Zc0XQosCAyQkN6hkJvsJVvwcGPOfxCpzNOUSoZ3YHUPOb4slDBdJkFK2GVJg7obyPt V78SbCNolh0yM4mUIrQDYWwdkqa+7AuYmce70glrExSl0VAb2jvgU++gdioTTd/ey6dO ltNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677802; x=1779282602; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=TcJbixi8CH0+fCoBl2UtMRXMhZzJhhVobMYZVtSUs0g=; b=j8Bryk8WFv/ZFmEVApcWIsr6DsnL8/a/3TBVePn89MsoS540i5IULuQu5m3ZlPxZV+ boaTXJphCNC++yI2gLsXOxu5XkyGb6Gnp9bE0EJZt9BfUm42V0TOXtFWs9NEe4ii5Ir3 0ZzGpFn1sC2dmMGxF/TvhyVuUSyPCdyfJTIafz6E9ry//kz8xKAQMTMTz7uyOtLOlhv4 C6y0GGx3cP8PebzuaScQLn4ZrRSpZFpUjPpE/CccxIEE3pHphu5bqWbEHdICK4R3eDcQ dJHQJ50XjzpK8gELTTFKahCk3BJgPgUnfW7M9nBHLUtWWKLOVFM4EocqsgQaVyUYX01c /glw== X-Forwarded-Encrypted: i=1; AFNElJ/QdrjfcOw+tlciREApnPXlAONOG0PaFTwIZX7oaLdjVgJJNzUpMH/+XSm4nVjDksVW08tkxZRzxkZfno8=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz56UjwaP9x22fna6j8yEU4KOBLqAc9MFCZiWPURc/eow+Eo2Hv ZAvUoAlbsZM1NbBQrs3Jvru8Y4Q/Xgudnik1w1LkZkBKv7HjU5joZ7gSU6TPw1lUM4k= X-Gm-Gg: Acq92OHpnPe7vdD6JeVjT2wBie3FcLnNJExlKsKWy636UHFGokguH26o6G5OMJDq5L+ ugvdkfnUhggSeReegk9/riNU27IYkDb1tR482XriFE7c2MX2LddAbHnjPstdxULDHJAUHsj6elq T7/BMNCletiAfpEy8HC8LqmUDyBYDHFcD0JV/8s2vhI0yHrY7Z8pC16/Z0SJNYONNIPUqY3KqRR IViCu0U9ftMDmPYQRBd8UBA9kOIxf/hRMW28hAdA6l9/mM0nxlS6ZzsIuWbVy4TVmSa9HRybGTV 8xkdVt7Ete7CThWZrUZodtZsYDod9dwkQHnZReEutI8lb4dqv3m0J6+5BeDc+fsze+0uZy42Ke7 n4ec1SUWb6kHGCGK91i2R4KB3gH3tMFbJZfLpMWSPt2mFa4ct/Nmbzqbl+bzxoVGE/mgpe8YWYP G7RIltr3wvwDWdN5x+klO/UwVuuOM3mnRPLLhjmQsOVGaiAu3thDSXPKJXmzQ= X-Received: by 2002:a17:902:ced0:b0:2ba:67f7:9326 with SMTP id d9443c01a7336-2bd2f6080a2mr31395015ad.9.1778677796975; Wed, 13 May 2026 06:09:56 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:09:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 26/69] mm/sparse-vmemmap: Support section-based vmemmap accounting Date: Wed, 13 May 2026 21:04:54 +0800 Message-ID: <20260513130542.35604-27-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Teach section_nr_vmemmap_pages() to account for section-based vmemmap optimization, so the helper can report the vmemmap page usage for a memory section with or without shared tail vmemmap pages. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 8 ++++++++ mm/sparse-vmemmap.c | 13 +++++++++---- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5fc968bac1f7..0974205abd3d 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2269,6 +2269,14 @@ static inline unsigned int pfn_to_section_order(unsigned long pfn) return section_order(__pfn_to_section(pfn)); } +static inline bool section_vmemmap_optimizable(const struct mem_section *section) +{ + if (!is_power_of_2(sizeof(struct page))) + return false; + + return section_order(section) >= OPTIMIZABLE_FOLIO_MIN_ORDER; +} + void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, unsigned long flags); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 60d5330a8399..94964363d95c 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -629,24 +629,29 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - const unsigned int order = pgmap ? pgmap->vmemmap_shift : 0; + const struct mem_section *ms = __pfn_to_section(pfn); + const unsigned int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); const unsigned long pages_per_compound = 1UL << order; + unsigned int vmemmap_pages = OPTIMIZED_FOLIO_VMEMMAP_PAGES; VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - if (!vmemmap_can_optimize(altmap, pgmap)) + if (vmemmap_can_optimize(altmap, pgmap)) + vmemmap_pages = VMEMMAP_RESERVE_NR; + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return VMEMMAP_RESERVE_NR * nr_pages / pages_per_compound; + return vmemmap_pages * nr_pages / pages_per_compound; } VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); if (IS_ALIGNED(pfn, pages_per_compound)) - return VMEMMAP_RESERVE_NR; + return vmemmap_pages; return 0; } From patchwork Wed May 13 13:04:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237841 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SNQ876Oa; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20876-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv3b14hZz1yKH for ; Wed, 13 May 2026 23:11:07 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2P5wcWz3cQh; Wed, 13 May 2026 23:10:05 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::634" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677805; cv=none; b=OKuXiywTxPyO0d9TZFP0RtH5aoGXQgaSbB+uyV8B+zLr21x0pklVqsAKr1fmCxu5g3BJTFtZNZY3tvhpL/rscjYpiBN1se5g7jCHlfm0fTNf1IwszAy08NohyshslNjiEaKya+nXJgV+sWg8i4fCFPrUBTgLBL9lgxyVJekredCieU9BI1UoN92jtlICQKvvizuNoa3jK3AN2yGRStbx4/1d720dnfrH9Um9gHfAOqKxWoqEMhASRCFvuEg0tFd1YqZAQy5WXAfHwUu7J0tOCxOp/+T9wON+Zl22py/XK9jc0PZANN5e7pKwAkxAwzo71rKA4V1ElDlTTqSlm+dTvw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677805; c=relaxed/relaxed; bh=YTByyvfyvRQXp0QDzHJ+fbRqWZip61THGdhvQJfTIW8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OabchBcPLu87euiB2MMZgGLMqWRTFqq70e2vStQ7H143Sb6SX4b/lvlQcQVaThvRgFxVaIlw3ImG80p/hy9WViMCMpchfZoLAOipfOsLv0BghWwnLk+uV3r8PX7oxXfAqKzx5emtpmEdOtMb4S2deqYmKOXYQe7oajWKhTG0rS9yaB09bz+rQV4gIHHx1FuL06uY0otlTwKd3iw5mEBl7BV9bLwuonA8GKJhXpZC/Q9pb1kZT42esJAkefESMiypPxe+0wz/p7D3W7r0AySXouIE7bJm1juj9AHZ9/zH0wku/uVC4N3Qke1e5RkCAZurXIWoca/1YiJVzXcb+Mu6ZA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SNQ876Oa; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SNQ876Oa; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2N57RNz3cPc for ; Wed, 13 May 2026 23:10:04 +1000 (AEST) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-2b788a98557so50708225ad.2 for ; Wed, 13 May 2026 06:10:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677803; x=1779282603; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YTByyvfyvRQXp0QDzHJ+fbRqWZip61THGdhvQJfTIW8=; b=SNQ876OaKiuBbgG87XQ6hVUc2N3Y+Y7p/fj9s4K5jQxZ/T68MYpXFvrZUFmQLn2IBJ uPdef8BqYfgFmQMeTj6/4/AZeb5n5XlwrPUK7sfe6f7iQ1BDpeB/Bs+IJrKHW/jtkcu1 6M90ZCU16RdXc9OG8lOmn0JfXy/aqG3deRBjtPv4twDzBcxWnXHOSvzAUjWh9/vXjIao v2LJXOet0RwMehyLKUVTTm+YGhFLPg+cBmuET4gEOy/8gtFZsbEWD5rifmXd2U6zgbEA 1N2tv8h8MaNO+oD+BnAwkwQNl7jIWsfzfQvrM+oDrw0e0FuDy42sNHnKU/7ZPcaG0umR 36Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677803; x=1779282603; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=YTByyvfyvRQXp0QDzHJ+fbRqWZip61THGdhvQJfTIW8=; b=Cc5v9UDpB19b8GuSQdGJckOVB1zH9WXHsrgOJ89A0EZZ6Y/maoql7X8IfS7vp0gmzs pOQDUzNdQdIqil6ntfI6shGjWypJhBsjsR+vxzjDF7XVpJh+D5U34PaFA89wpEPJM04m yuxoYCN6VnFy3OtOxQIjqRU9RHBNb5VBxviWMHQZrSX/w5cg8bUooyiZPkt9yLmKVvZw GPM7Jo3FNvJn0+klqFgH8e1KVKE7Oa9lte1e1sa/6J6OOi9I+7u2lwkFWPRkGZ4KFEBL M+n2NxICkZ3kloLLQMy0TRYMw80s2Is8D0iUujo/DUk33W0VerafZnoC9Ss75M3MsRUl AHrg== X-Forwarded-Encrypted: i=1; AFNElJ/dHg8rhRMs+4ky9Yjhgaoc85XtVhyphVWfQzHyzSrINliIAYyWoTwp5pmVRSEOVGCKQAltlrxNPdiaJak=@lists.ozlabs.org X-Gm-Message-State: AOJu0YybnF8AxM3gcSyI6G7HfLFr1/U/nH/V8cnU2N2cEJl/0DLaT0fH 9WDsxD9IG8LNp5dL+emrkWOpEOGcipTl4OmOxqW5ao7F+WQvSUg9iBiYsAm4FoGbi9c= X-Gm-Gg: Acq92OGpoBJlygqs5R/v91oygRJPhQUhjEHI6LDai6fU8nAf/UgxtGOMA/BRVaTX3Z0 Q+dsio/kKSMvLrqSId3STB9j/WQHo3NdMEc+PH7S1wiH1lBuay6JwTC/ki1Cwt0El349tvruwN2 IleFX6XjcTrfZ8P9pDrCXWH7/UWaG2nf1u8ta30nwrEilrJYl1BJpnJoUy/N7kqnQ8esd+Rc7gV nhUCH7oOlskm6+xkNvi0s3TAx6PPCyh3OFFVBdDC7M+kdkfMeVmGVGq5a0Zh1dFPFsyn31LP4B9 YG07/S1BzXS8q5JeX3qcP1CPvs4C6evNKVB+Yvw13K0TUDJHC4p21zzwPgUseeZAfatu1a/a+lG u/Tprey0VPl/yY/if/hYr8ChyRNsTELkxXrMZSB6qXKqGPPM8S4mHmwYZK8Va7oCjdmWDXn/AD3 7uG2/yxU+rsDnw+ho/FISv9aaUY+zXQrfVCaezyKd7SN41iZMu1O4K5a2riEZRZxnep0BSUA== X-Received: by 2002:a17:902:c209:b0:2bc:f1da:4bfb with SMTP id d9443c01a7336-2bd275a75b1mr25305165ad.34.1778677802375; Wed, 13 May 2026 06:10:02 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.09.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:02 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 27/69] mm/sparse-vmemmap: Support section-based vmemmap optimization Date: Wed, 13 May 2026 21:04:55 +0800 Message-ID: <20260513130542.35604-28-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Teach sparse-vmemmap population code to use the compound page order when deciding whether a vmemmap page can be optimized. With this information, the common sparse-vmemmap population path can allocate or reuse shared tail vmemmap pages directly instead of relying on HugeTLB/DAX-specific handling. This centralizes vmemmap optimization logic in the sparse-vmemmap code, based on section metadata, and prepares for sharing the same mechanism across different users of vmemmap optimization, including HugeTLB and DAX. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 2 +- mm/internal.h | 3 ++ mm/sparse-vmemmap.c | 89 +++++++++++++++++++++++++----------------- mm/sparse.c | 34 +++++++++++++++- 4 files changed, 89 insertions(+), 39 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 0974205abd3d..bf4c40818b63 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1147,7 +1147,7 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP struct page *vmemmap_tails[NR_OPTIMIZABLE_FOLIO_ORDERS]; #endif } ____cacheline_internodealigned_in_smp; diff --git a/mm/internal.h b/mm/internal.h index 1f1c07eb70e2..2defdef1aedf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -995,6 +995,9 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } + +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 94964363d95c..69ae40692e41 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -139,17 +139,49 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } +static struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat = NODE_DATA(nid); + + for (enum zone_type zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { + struct zone *zone = &pgdat->node_zones[zone_type]; + + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + +static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone); + static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, unsigned long ptpfn) { pte_t *pte = pte_offset_kernel(pmd, addr); + if (pte_none(ptep_get(pte))) { pte_t entry; - void *p; + + if (vmemmap_page_optimizable((struct page *)addr) && + ptpfn == (unsigned long)-1) { + struct page *page; + unsigned long pfn = page_to_pfn((struct page *)addr); + const struct mem_section *ms = __pfn_to_section(pfn); + struct zone *zone = pfn_to_zone(pfn, node); + + if (WARN_ON_ONCE(!zone)) + return NULL; + page = vmemmap_get_tail(section_order(ms), zone); + if (!page) + return NULL; + ptpfn = page_to_pfn(page); + } if (ptpfn == (unsigned long)-1) { - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) return NULL; ptpfn = PHYS_PFN(__pa(p)); @@ -168,7 +200,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in } entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); - } + } else if (WARN_ON_ONCE(vmemmap_page_optimizable((struct page *)addr))) + return NULL; return pte; } @@ -311,7 +344,6 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, } } -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { struct page *p, *tail; @@ -340,6 +372,7 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * return tail; } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, unsigned int order, struct zone *zone, unsigned long headsize) @@ -388,6 +421,9 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, pmd_t *pmd; for (addr = start; addr < end; addr = next) { + unsigned long pfn = page_to_pfn((struct page *)addr); + struct mem_section *ms = __pfn_to_section(pfn); + next = pmd_addr_end(addr, end); pgd = vmemmap_pgd_populate(addr, node); @@ -403,7 +439,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return -ENOMEM; pmd = pmd_offset(pud, addr); - if (pmd_none(pmdp_get(pmd))) { + if (pmd_none(pmdp_get(pmd)) && !section_vmemmap_optimizable(ms)) { void *p; p = vmemmap_alloc_block_buf(PMD_SIZE, node, altmap); @@ -421,8 +457,19 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, */ return -ENOMEM; } - } else if (vmemmap_check_pmd(pmd, node, addr, next)) + } else if (vmemmap_check_pmd(pmd, node, addr, next)) { + const struct mem_section *start_ms; + unsigned long align = max(1UL << section_order(ms), PAGES_PER_SECTION); + + /* HVO-covered sections must not use PMD mappings. */ + start_ms = __pfn_to_section(ALIGN_DOWN(pfn, align)); + if (!IS_ALIGNED(pfn, align) && section_vmemmap_optimizable(start_ms)) + return -ENOTSUPP; + + /* PMD mappings end HVO coverage for this section. */ + section_set_order(ms, 0); continue; + } if (vmemmap_populate_basepages(addr, next, node, altmap)) return -ENOMEM; } @@ -626,36 +673,6 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) } } -static int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - const struct mem_section *ms = __pfn_to_section(pfn); - const unsigned int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); - const unsigned long pages_per_compound = 1UL << order; - unsigned int vmemmap_pages = OPTIMIZED_FOLIO_VMEMMAP_PAGES; - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - - if (vmemmap_can_optimize(altmap, pgmap)) - vmemmap_pages = VMEMMAP_RESERVE_NR; - - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; - - return 0; -} - static struct page * __meminit populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/sparse.c b/mm/sparse.c index 9457a4d6a6fc..3e96478a63e0 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -284,6 +284,36 @@ static void __init sparse_usage_fini(void) sparse_usagebuf = sparse_usagebuf_end = NULL; } +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, + struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +{ + const struct mem_section *ms = __pfn_to_section(pfn); + const unsigned int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); + const unsigned long pages_per_compound = 1UL << order; + unsigned int vmemmap_pages = OPTIMIZED_FOLIO_VMEMMAP_PAGES; + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + if (vmemmap_can_optimize(altmap, pgmap)) + vmemmap_pages = VMEMMAP_RESERVE_NR; + + if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) + return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return vmemmap_pages * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return vmemmap_pages; + + return 0; +} + /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_end) * And number of present sections in this node is map_count. @@ -314,8 +344,8 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(DIV_ROUND_UP(PAGES_PER_SECTION * sizeof(struct page), - PAGE_SIZE)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); sparse_init_early_section(nid, map, pnum, 0); } } From patchwork Wed May 13 13:04:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237842 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hBnMnqmS; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20877-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv3k2blzz1yKH for ; Wed, 13 May 2026 23:11:14 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2V5CGWz3cYK; Wed, 13 May 2026 23:10:10 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677810; cv=none; b=nzmi7hivEUEEIqXotqCQTuXclFBwPwTuuWLu1zPHPGJf/ixD/1tzJYolGGqRKwtuk0iXwr+u5ZZ8aM7Li0f7u7PGrtvsP1L2+iRSSxn6S2nLE+63Vj3cdbSBMCKEMxY3wKoH+Jj5n40reAFGmigHsHj01KwfgdWn/wZo0IIj3MNdzOEGtgC1w0wjxfqV5N1jrTCx9rOImS9kfSjQBkebvbGvIw1k3UexrFzvXgFG3zdvGC+4KwK/X+Si4+LZp5aRrr+CYlAtsHt7jeySD12Kz5fds58YTxQdF+Jo0C5/qCaMoG5ma/GDGELCgXu2dMGL1zz0xr9mlHwl4YQzt2z7FQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677810; c=relaxed/relaxed; bh=wj09+T2FktkdLwU79u80MiVhyJiQtKDMnM2x4Hjwdtg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UG4bvzXCyDZjUywohk7ZihBvzSxgxhGslOjtHR5e6aLFRUgtIczfV3oAUn8g70eh6wDU+2IOK2hxxqijB9JNw1tdlIoE2+SB3oTTqigP/+k71nU1C75hFvR5kaoj/NcRUf19MMPkhfVVwQ1L7Q7ajaaBnE/UF49NvYY+d71aCfCquWXYdOG2hCyKUAw6mffHXMmKC84z1y4i3wI3Mt0PXPWib0tn5ijjHFPEKKzw9uj55wwVBqYwRSJaQeplFeYUyja3gyU3I2YuKleoZuK0dewHAC7WK9ATqM9W1NXh8ASGC8/6ekTjVn9PknKzKK5gDr2rAodFZYO+rOuBTKwGRA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hBnMnqmS; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hBnMnqmS; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2V083Sz2xqn for ; Wed, 13 May 2026 23:10:09 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2bc7b311e77so17532175ad.0 for ; Wed, 13 May 2026 06:10:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677808; x=1779282608; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wj09+T2FktkdLwU79u80MiVhyJiQtKDMnM2x4Hjwdtg=; b=hBnMnqmSFB9V0QzHF4n48AWSWzmav5vdXOGcBvcAeGNyuL4kgI7J/SuHhrnGo7DAQG qZErG9wPWKhUKXyg5oSbUdEbeewQAu3wI7qW662oLUONbTXdab48K+K2n3Nj94wOU2Cl IhiV3Rl0oX1Q+aU/6WvQRBCBntSOmPy3NHmNhNiLQMRpjCyEphAYVD8pnMd5gDCPEI8P TA8ny28fi990np5EfbWqIc9vnNWYujRQLFVe4M0MU6H/Sond27aPF+TFP8vFOQtlzYN1 N+mhZ8Ctd8xovUm/TTCOF0+jyayaU0o83skFBNAvHGGfO6Ke0/IJ8nbLRd0pSLui7qWJ Yw4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677808; x=1779282608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=wj09+T2FktkdLwU79u80MiVhyJiQtKDMnM2x4Hjwdtg=; b=eQGJ9igPbcADpR9jCZ626ozc1cXja4FlXXsTKm+uxidbidGK1s4k03tqOoqdLt4CZA +0ZOHoGxF7fdNKiViIpGO7ZVb7jEThpS6u5k9LV8QTizvcSfh56mkGnTo8FW8M7HQv+y llEZg+aH4UN8x/L5qqOUEUHVkG9q6iijx3u1d9AuqnCiqbS9vCyEjZ2rM6PJiBojXOWZ lIS+M72azO4P8HFqWhXX7JqB+4Y6MZc/P5aRL/aoqQW6pT7G9IXTxvqVqqRcpbe1K9+4 i1Du1L+8yvBKpDUodW+KKGYf4A2qU2Gqi77gjTL29h9QxD5ywFzNRNRIv7nmcDoIpiy4 8Z2w== X-Forwarded-Encrypted: i=1; AFNElJ/q+eHITBQC8j2ngrlHe16jw8HfOHkzGO1v13v055cDjtD564SkrU1xkrgyo41HUBDDOPShzNeg5X4MdOw=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yz/FwDiJ8c17KB/hUfnEFD7jZ0B7bJQceoXflfWq8xnIczRLgUH Suth2cV7yBA4SOT+meF5nESSOZHwXd06YeaoqH8Oz7juDufGwfD2Zeo+EArMwd90+5E= X-Gm-Gg: Acq92OF7tpBsiArCIt0Ojrb7e3Tc1cWFNz9F7hWvrACSjCDzNpvOc18GPaM27SI5Fvu 5BuD+F0AXzoY+YldDjzB1KCHXf7WI/dDRt8bmOen8ua7+WRIgatTUBf/bIrg5/HvX/Iddp7anT6 zoJvvT2+jIV2fMI8hFnoLSbFSOr///WRfg4XuoVx4tqrvRzaVwqksRQ8TA6qxRWZxTxq0MiYSkv 2JdShj1Y1TuZnWVnkHLcxlLyKPuL8HCE+XPp92mHUi9i5DERBWLtLyEHoEDNr7I1dYTQ2eGGCVa UsH+sYFLKla74lD3lMc6xJ2qB52HpFfGGG0rx95NFx3c0h1dLzzudfpZt3GM9aCnDXTNdoWaaOj IWgx6ZDULIBWCUREW1SEGue8YnyXgyIkExgRbr1TM/qRLW/+uLaoNfLvinpvMe5pustcGGyJYJr d0HeuDhEO1zi65Oye1k53NJ0/ioOPX93mTEUxBlK6q2qnMyq97jjJXKGPvJMEIk25hOxAoaQ== X-Received: by 2002:a17:902:b713:b0:2b0:6068:4c5f with SMTP id d9443c01a7336-2bcfcc3ce55mr52172605ad.8.1778677807808; Wed, 13 May 2026 06:10:07 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:07 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 28/69] mm/hugetlb: Use generic vmemmap optimization macros Date: Wed, 13 May 2026 21:04:56 +0800 Message-ID: <20260513130542.35604-29-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Vmemmap optimization is no longer hugetlb-specific, so the remaining hugetlb-local reserve macros are redundant. Replace them with the generic definitions to remove duplication and keep the hugetlb vmemmap code aligned with the common optimization macros. Signed-off-by: Muchun Song --- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.c | 14 +++++++------- mm/hugetlb_vmemmap.h | 9 +-------- 3 files changed, 10 insertions(+), 17 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 53448b05ca11..8debe5c5abce 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3222,7 +3222,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, * be no contention. */ hugetlb_folio_init_tail_vmemmap(folio, h, - HUGETLB_VMEMMAP_RESERVE_PAGES, + OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES, pages_per_huge_page(h)); } hugetlb_bootmem_init_migratetype(folio, h); @@ -3261,7 +3261,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) WARN_ON(folio_ref_count(folio) != 1); hugetlb_folio_init_vmemmap(folio, h, - HUGETLB_VMEMMAP_RESERVE_PAGES); + OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); init_new_hugetlb_folio(folio); if (hugetlb_bootmem_page_prehvo(m)) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e9906d32a64c..4367118f8f57 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -407,7 +407,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, vmemmap_start = (unsigned long)&folio->page; vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); - vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + vmemmap_start += OPTIMIZED_FOLIO_VMEMMAP_SIZE; /* * The pages which the vmemmap virtual address range [@vmemmap_start, @@ -637,10 +637,10 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, spfn = (unsigned long)&folio->page; epfn = spfn + hugetlb_vmemmap_size(h); vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), - HUGETLB_VMEMMAP_RESERVE_SIZE); + OPTIMIZED_FOLIO_VMEMMAP_SIZE); register_page_bootmem_memmap(pfn_to_section_nr(folio_pfn(folio)), &folio->page, - HUGETLB_VMEMMAP_RESERVE_PAGES); + OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); continue; } @@ -779,9 +779,9 @@ void __init hugetlb_vmemmap_init_early(int nid) zone = pfn_to_zone(nid, pfn); if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), - zone, HUGETLB_VMEMMAP_RESERVE_SIZE)) + zone, OPTIMIZED_FOLIO_VMEMMAP_SIZE)) panic("Failed to allocate memmap for HugeTLB page\n"); - memmap_boot_pages_add(DIV_ROUND_UP(HUGETLB_VMEMMAP_RESERVE_SIZE, PAGE_SIZE)); + memmap_boot_pages_add(OPTIMIZED_FOLIO_VMEMMAP_PAGES); pnum = pfn_to_section_nr(pfn); ns = psize / section_size; @@ -826,8 +826,8 @@ static int __init hugetlb_vmemmap_init(void) { const struct hstate *h; - /* HUGETLB_VMEMMAP_RESERVE_SIZE should cover all used struct pages */ - BUILD_BUG_ON(__NR_USED_SUBPAGE > HUGETLB_VMEMMAP_RESERVE_PAGES); + /* OPTIMIZED_FOLIO_VMEMMAP_SIZE should cover all used struct pages */ + BUILD_BUG_ON(__NR_USED_SUBPAGE > OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); for_each_hstate(h) { if (hugetlb_vmemmap_optimizable(h)) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 7ac49c52457d..66e11893d076 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -12,13 +12,6 @@ #include #include -/* - * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See - * Documentation/mm/vmemmap_dedup.rst. - */ -#define HUGETLB_VMEMMAP_RESERVE_SIZE PAGE_SIZE -#define HUGETLB_VMEMMAP_RESERVE_PAGES (HUGETLB_VMEMMAP_RESERVE_SIZE / sizeof(struct page)) - #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio); long hugetlb_vmemmap_restore_folios(const struct hstate *h, @@ -43,7 +36,7 @@ static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) */ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { - int size = hugetlb_vmemmap_size(h) - HUGETLB_VMEMMAP_RESERVE_SIZE; + int size = hugetlb_vmemmap_size(h) - OPTIMIZED_FOLIO_VMEMMAP_SIZE; if (!is_power_of_2(sizeof(struct page))) return 0; From patchwork Wed May 13 13:04:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237843 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S2NmvPvI; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20878-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv3t0sz6z1yKH for ; Wed, 13 May 2026 23:11:22 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2c1ZLnz2yYK; Wed, 13 May 2026 23:10:16 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::535" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677816; cv=none; b=Wrc1OEfOl5xWUZrvfAezwI6rDUyQze8B53osjdUJxLw97p9B3MWK8aqB0N3vOssto3sB/Fh1jb+h2Y8wNTXoB183yx70tec5i0aZeSTdPkstaZXpbCf9TPgCK/thMgj4lVo3l8JOXFT2qsiPRtlm9wZtZ1zHrz75XS7kuimek1RYmgFvj86mF1sTY77fxeLsu8VzgZ9roBE4mkKGRAXdhyja/qZjKM3NBEvdxcJRZkFYR3GSaFG98SNeG/M0nKndPTOu57W5gkoqQq4ODKR+4WV71fn4xCS4O7G9llDvNRLQGZIg/mSC4dodsUO8hb/O/ZbJ6+9K4kfu9U4l62ZeMg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677816; c=relaxed/relaxed; bh=Y3uLkZc/zeI4wWlYRCnYJNtlEmmCl6e0Ymtz/E5HBFc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dEu2hWxGc9eq/LWB4ZJdOagR2bJ3C5/yqN6McxiRzMJ2auozUB/vc8hswJeUMWdbs40KZStdJyGGBO8K3HW8J24E0qskRh4BqnmhsLfvjG42Rqe+2ikoFDekO5DjhpAk3xz0tv1fSI3mzfFx87EGSeDl/PX5SvG2TKDelhYBahh9P4N8A4Ku3nMNnsQt9i6NRjDnzADCM2QhjexuJJpymEl5cIMv/qhTAk0BwEK3kOhsxfmPkBx7eDf6Hu7CzaDNfTqlAKZRXsnrHNRalRcUa4dlYrP8yGyosEJxlK6Cv4HsHQc8LQivpZLQBanm2cO71DnTlPoxoADbCnKekkI2Yg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S2NmvPvI; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=S2NmvPvI; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2b3TdTz2xqn for ; Wed, 13 May 2026 23:10:15 +1000 (AEST) Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-b6ce6d1d3dcso2739065a12.3 for ; Wed, 13 May 2026 06:10:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677813; x=1779282613; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y3uLkZc/zeI4wWlYRCnYJNtlEmmCl6e0Ymtz/E5HBFc=; b=S2NmvPvIUrYaxfHA5xSUa6KbDeC3WRk50c8OGVaWNdkojvR1eOEvpDEUl90MYySTUZ b/6OgHs6JKp1lZgMT2KSe5uCbPoWf5FG5yck6m98XtqkuLoOOIIVunh/9p9xntboC9J9 0s3R1Aibvy+vMnxUiLyTnPkN9fXFFqt5vkwKVgfDg+FIq2nl4oGgbBMLa81r7v4PTlz8 IGEuXA2GrmZSYjkqMKThUa5iY7FE6qKMaW1TzzbmBUHuZhR4qEVSVuJtNErPvEMi3Qgy BSLimk2I6HQ00hGOHpfmcM5k02O33nrEIKX7D9jrGb2MIld2K/BeIJnNWFebpHCfn9Ca 97yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677813; x=1779282613; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Y3uLkZc/zeI4wWlYRCnYJNtlEmmCl6e0Ymtz/E5HBFc=; b=tM1W9Nez20Pm5DLuSmlA6wZeY976qd+UF7h+qHvRNPayFCCmlv1gRYNnjn26mDl+Zg mtLQRQ5IEcCy7E0CfSyOI3xFn3F49ahn5Mo+vc4vDOLHM+EuWFbiqGyjtuBkjKvgmr7a CB7dYHMwprlDWW/hKKg5RQ99y67jjmY5CatCqf8qNjJLwqgxwa3J446/eov53MkRtX2c KfPtY56zOzrPQyBfxJYZQVN84hKgghxqdqnMBgJzkj7OUJhNOZ01cIpfOCfzMyJcQAAu DLTvHcvc0pPAqJQ18kDoABdyH14br2ln5tqF4MDdZ6M9spzTD24XwNEfQz1nYi6puIZX Pmzg== X-Forwarded-Encrypted: i=1; AFNElJ+PuScKp4PGhvn7aL6ehhCkjHy20ge+7QqgVfmDllAOCWsv2nC+9kX/Bb8rPIoK6C5Wq06wQUFLzRJp6eE=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyC745E5z9o5hu4P1bojtLepkd2gKpYwlhBxPSd06Bz/rio4jlg tGBOUd2Ruma7r5EEZrO1vvD21EGIBlZOpaUI4fUBYlAbkpy1XC64pHjIf+7V5pDoOiA= X-Gm-Gg: Acq92OGakuZ3kLvUORyvBwrg1oihLtRGoXHKFsr1o9dPlzCjNmffScoN3jNqir+VZqU 9APEXe27CLysLeocGEshuOJGz5vRQOyW8KsC42PxTh/GycQJcleSRemtMu3+fBNP5Z5fcJpOZwB OmdyPMbpJPJ6mt88Wwi4BnSoegFbStW0QTk72ZyNO2mkoaSdaKpbW6uN9KZAoAhOpkyA2TRelAe cf+V/XAtXRXGTrBdYX4i3PFi1d1RHFcVrQO8Ve/RcYZS+UZBbhJTR76c/n9setReGpWW1GfBk3g 3vXbSTtDSqmQh0qf11aWFIpPMUBR9aSCUZUFHbCgMBYexpRJKwehAC1k+azpi2yi5NkhyxbTPKn zrHWrM6j86hI5pkOLoi33bZMEQdun5L2VFgYLBcb+E2+1ydi1h5ZHmvIiMCFo7PoNRMiG2l5pp1 9sXmFo4wHbsqCxU7kc4ExXoib4U+OngpiLLrAYIy2h3/1ebupNif5r8z3EJSo= X-Received: by 2002:a17:902:e54f:b0:2bd:4b8c:fd30 with SMTP id d9443c01a7336-2bd4b8cff82mr5004355ad.41.1778677813243; Wed, 13 May 2026 06:10:13 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.08 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:12 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 29/69] mm/sparse: Mark memblocks present earlier Date: Wed, 13 May 2026 21:04:57 +0800 Message-ID: <20260513130542.35604-30-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Later patches need struct mem_section entries to be available before HugeTLB bootmem allocation starts, so the section metadata can be set up at that stage. Move the memblock-based section present marking out of sparse_init() and call it earlier from mm_core_init_early(). Rename the helper to sparse_memblocks_present() while doing so. This prepares sparsemem section metadata before the early HugeTLB setup path. Signed-off-by: Muchun Song --- mm/internal.h | 2 ++ mm/mm_init.c | 1 + mm/sparse.c | 4 +--- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 2defdef1aedf..bf30617c78d8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -962,6 +962,7 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long, * mm/sparse.c */ #ifdef CONFIG_SPARSEMEM +void sparse_memblocks_present(void); void sparse_init(void); int sparse_index_init(unsigned long section_nr, int nid); @@ -999,6 +1000,7 @@ static inline void __section_mark_present(struct mem_section *ms, int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #else +static inline void sparse_memblocks_present(void) {} static inline void sparse_init(void) {} #endif /* CONFIG_SPARSEMEM */ diff --git a/mm/mm_init.c b/mm/mm_init.c index 3aaee1cf7bf0..6723c604eefd 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2693,6 +2693,7 @@ void __init __weak mem_init(void) void __init mm_core_init_early(void) { + sparse_memblocks_present(); free_area_init(); hugetlb_cma_reserve(); diff --git a/mm/sparse.c b/mm/sparse.c index 3e96478a63e0..33e89bf1ec0c 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -195,7 +195,7 @@ static void __init memory_present(int nid, unsigned long start, unsigned long en * This is a convenience function that is useful to mark all of the systems * memory as present during initialization. */ -static void __init memblocks_present(void) +void __init sparse_memblocks_present(void) { unsigned long start, end; int i, nid; @@ -361,8 +361,6 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count = 1; int nid_begin; - memblocks_present(); - if (compound_info_has_mask()) { VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); From patchwork Wed May 13 13:04:58 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237844 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ApQzJ/4T; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20879-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv411HmDz1yKH for ; Wed, 13 May 2026 23:11:29 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2j6Qwmz2yYf; Wed, 13 May 2026 23:10:21 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::535" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677821; cv=none; b=GgPKr4ONtNkhxvuIX1gb1wum+O7rlfTSMd03OypdYQZ2B39GbdMzy2EzlLDgwrFFXosUEbKb3St4LQQ9fcMpAlNiN+SN+yHyAaoEAF+8R1F3WNxL58xs694jR/nKj4I+N080Oe0qz8mn/FsHkqpi2qM6G99gOX1N0k6qJjCYGSA1e2yrWC0k3WcbAvY1ZsHRXpb2SnsfBOX4pv4yTkixnHLsYMfddIbLi2qbf6gcLKB9NA4SECUn0R0KFVjCJTHPj3R/WQI90krApctxvvYXlNaqSNOgNQOU8ucpTnaf4+43Tzo0yIMgmDQ4Ainx0vm09jIOtPCqApI9JTILXjdpVA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677821; c=relaxed/relaxed; bh=OR7HeIKD4yKpxE8z6TINwIVqXU1D5zEjTSKwc02PFRM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fn06bnrY2W+vhg1Wtr4fSXzUUuwWYRURxqEK8TD1gAeBv74bESo5u1DGwrQNy1FIcBjeOXpLFuimY4bXlpCyTiSsiFrCvb8spJBv5UkYpCHiBZz9yGx5Vei1djvdHqgvlwUgvJSEBpGbAEItI26b5EBiPJ07KP1f1Q53wb/bJnZJpYAtWleE17Z6lH0hPPQDIAuJzTw7s/iGIxwkXEjnqCULmQlKCeuBbTI2Nhnw7vpOCDjEUCplu16pOnjFWqpFpsznVejV9W0D/5sz5/YCHngmXdMzmj7js6Pr5Wwf8tAGS9cvVHejutL0+FGUeR3XgfwtUwCfmRcBMyyHu4hqAw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ApQzJ/4T; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=ApQzJ/4T; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2h5LWrz2xqn for ; Wed, 13 May 2026 23:10:20 +1000 (AEST) Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-c82471904fcso2969256a12.2 for ; Wed, 13 May 2026 06:10:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677819; x=1779282619; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OR7HeIKD4yKpxE8z6TINwIVqXU1D5zEjTSKwc02PFRM=; b=ApQzJ/4TsuPph72kqqn8d1rJUkhM2DWjqTvwc7gLriwWvdk9cb8+aoghA5wsUQyD0P 7nK+BSX4fIm16+y8K48SVYicRvOhOhp5rlWiwrZhmYpnFxp13uenw7Q9tKWCeCOc9B0p z/QOAfeXaV5IYGg9q9dk+AWNWUVitA1l1NQ6FIIOSsFukXzfIbDuLlD9Ci4H3an8rkdH uA4U4hp17mDoWWdTfu2w2o9r47zLc1MNcrlEG1GeWXaSJ/BE/yJL0A382FQuPrWPEleH E8nKO9CX5TGAx2kGuDgKrAORsQ6DwaGXqMS8VQH+gWYZdI0smlRKDSvhFUF9zXsCorEH rW7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677819; x=1779282619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=OR7HeIKD4yKpxE8z6TINwIVqXU1D5zEjTSKwc02PFRM=; b=AlyD5yeumqlAWdaRilWXq/SLspR4w4XEijROuhm3T+3/PKVpxcU9bzg35OzbvwB/Rw 4nGCianSgTTb2Pl2KT5tGIsDXKhxTZgYAWzGTAfBJ7R2ERO9DnIpo9AidRB/PxBgCJ/f pRAHWqfMrp6BfX10CXKCaGZSBXuGGIvNNRJ+6FCxjOun8oejJvfEaXgIrnB+inNGKpE+ cgxZQOZp1Fq80Okj1eWLlC9fH9fqr7VfLBGj/BbFRt8dh4qK0ev/CXpos1PWpfxabSTg sfAL3z6p3wGmDHnnn5FDe48o+0HSBFFPrW8dV70quFsk3+9CtjK9GvfBnzRfUuVkViY5 b80w== X-Forwarded-Encrypted: i=1; AFNElJ/0ttzKMYsIoMEtsEnNJ4j051SLx9YCCfQdFxR67fzWiv/h0G9IcKTYS7U77XZvQljSDjzkq7T8uoMSBBk=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxzyTi1zTymAUgoYYxrMi+kCuVWBzC4fMf9H9m4ZlwL+GEq72zq QAQYxRZH9WAojHcbRYD9YJ5SlZvUQsOout6CvbyUPZS2z79Hd6KXGTZ/50P+fxzkqqs= X-Gm-Gg: Acq92OGADZu+yhap++bM9qG+3c35GmJjvTPuewt3E3GYHck8VNuSEI2JZo/6W2JxrWa 5Duki7dGwdeLBBs1h8/N0ygvV4P7k5yPvwM5HZRoS2kXLF/JYG0WqEBkpzu4DpfSz5sOIPIcU9G izQvLrj2vmGLteTQkWEzGjxbnD7lwAdd+mveZCAuqR1NSlo5P5tViV19HESuDnh+wToXTCWLLIe 3bITnIk0QEhcX09BTChGnC1NhxwrcWGzMM7GAUMu6y1Lvajrd0A9UWCacGzhPpZH9mPGvVmH7mn IV8tAnI0q9WNyhRrZzbjkv9S/WD1spdkZdjGdYApkTXFdfmVUJX7Xl1nAnDTJQNuIx6oPfwdDKF OUW+54vs1R24MGQiom5CCR12oIkSu2inv2+TJLAExMQCETysKraEW9NjVC3XkzIG99l0Wp8hnfa ruhen5IRvJtjjiOjMcZ9E2G9RNcx8otEYXy53oOunLb8Zv2yVEuEm0RtJa5Wr+Zmqod2wRAQ== X-Received: by 2002:a17:903:3905:b0:2ba:c453:ed3a with SMTP id d9443c01a7336-2bd2718da07mr36777045ad.16.1778677818683; Wed, 13 May 2026 06:10:18 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:18 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 30/69] mm/hugetlb: Switch HugeTLB to section-based vmemmap optimization Date: Wed, 13 May 2026 21:04:58 +0800 Message-ID: <20260513130542.35604-31-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HugeTLB bootmem vmemmap optimization still carries its own early setup path, including pre-populating optimized mappings before the generic sparse-vmemmap code runs. Now that section metadata records the compound page order, HugeTLB only needs to mark the bootmem huge page range with that order. The generic sparse-vmemmap population path can then allocate and map the shared tail vmemmap pages without any HugeTLB-specific early population code. Do that by setting the section order when a bootmem huge page is allocated and dropping the dedicated pre-HVO helpers and related special-casing. This removes duplicate early setup logic and switches HugeTLB to the section-based vmemmap optimization path. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 1 - include/linux/mm.h | 3 - include/linux/mmzone.h | 17 ++++++ mm/bootmem_info.c | 5 +- mm/hugetlb.c | 26 ++------- mm/hugetlb_vmemmap.c | 124 ++++++---------------------------------- mm/hugetlb_vmemmap.h | 13 ++--- mm/sparse-vmemmap.c | 29 ---------- 8 files changed, 45 insertions(+), 173 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index fd901bb3630c..dce8969961ea 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -171,7 +171,6 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio); extern int movable_gigantic_pages __read_mostly; extern int sysctl_hugetlb_shm_group __read_mostly; -extern struct list_head huge_boot_pages[MAX_NUMNODES]; void hugetlb_struct_page_init(void); void hugetlb_bootmem_alloc(void); diff --git a/include/linux/mm.h b/include/linux/mm.h index 31e27ff6a35f..f39f6fca6551 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4864,9 +4864,6 @@ int vmemmap_populate_hugepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); -int vmemmap_populate_hvo(unsigned long start, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize); void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, unsigned long headsize); void vmemmap_populate_print_last(void); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index bf4c40818b63..d6a5dd042c25 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2264,6 +2264,18 @@ static inline unsigned int section_order(const struct mem_section *section) } #endif +static inline void section_set_order_range(unsigned long pfn, unsigned long nr_pages, + unsigned int order) +{ + unsigned long section_nr = pfn_to_section_nr(pfn); + + if (!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)) + return; + + for (unsigned long i = 0; i < nr_pages / PAGES_PER_SECTION; i++) + section_set_order(__nr_to_section(section_nr + i), order); +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return section_order(__pfn_to_section(pfn)); @@ -2417,6 +2429,11 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #else #define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid +static inline void section_set_order_range(unsigned long pfn, unsigned long nr_pages, + unsigned int order) +{ +} + static inline unsigned int pfn_to_section_order(unsigned long pfn) { return 0; diff --git a/mm/bootmem_info.c b/mm/bootmem_info.c index 3d7675a3ae04..24f45d86ffb3 100644 --- a/mm/bootmem_info.c +++ b/mm/bootmem_info.c @@ -51,9 +51,8 @@ static void __init register_page_bootmem_info_section(unsigned long start_pfn) section_nr = pfn_to_section_nr(start_pfn); ms = __nr_to_section(section_nr); - if (!preinited_vmemmap_section(ms)) - register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn), - PAGES_PER_SECTION); + register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn), + PAGES_PER_SECTION); usage = ms->usage; page = virt_to_page(usage); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8debe5c5abce..080f130017e3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -57,7 +57,7 @@ unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; __initdata nodemask_t hugetlb_bootmem_nodes; -__initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +static __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; /* * Due to ordering constraints across the init code for various @@ -3111,6 +3111,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) } else { list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |= HUGE_BOOTMEM_ZONES_VALID; + hugetlb_vmemmap_optimize_bootmem_page(m); /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3264,13 +3265,15 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); init_new_hugetlb_folio(folio); - if (hugetlb_bootmem_page_prehvo(m)) + if (hugetlb_bootmem_page_prehvo(m)) { /* * If pre-HVO was done, just set the * flag, the HVO code will then skip * this folio. */ folio_set_hugetlb_vmemmap_optimized(folio); + section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); + } if (hugetlb_bootmem_page_earlycma(m)) folio_set_hugetlb_cma(folio); @@ -3314,25 +3317,6 @@ void __init hugetlb_struct_page_init(void) .max_threads = num_node_state(N_MEMORY), .numa_aware = true, }; -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - struct zone *zone; - - for_each_zone(zone) { - for (int i = 0; i < NR_OPTIMIZABLE_FOLIO_ORDERS; i++) { - struct page *tail, *p; - unsigned int order; - - tail = zone->vmemmap_tails[i]; - if (!tail) - continue; - - order = i + OPTIMIZABLE_FOLIO_MIN_ORDER; - p = page_to_virt(tail); - for (int j = 0; j < PAGE_SIZE / sizeof(struct page); j++) - init_compound_tail(p + j, NULL, order, zone); - } - } -#endif padata_do_multithreaded(&job); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4367118f8f57..730190390ba9 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "hugetlb_vmemmap.h" @@ -478,12 +479,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, return ret; } -/* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ -static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio *folio) +static inline bool vmemmap_should_optimize(const struct hstate *h) { - if (folio_test_hugetlb_vmemmap_optimized(folio)) - return false; - if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -493,6 +490,15 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio * return true; } +/* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ +static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio *folio) +{ + if (folio_test_hugetlb_vmemmap_optimized(folio)) + return false; + + return vmemmap_should_optimize(h); +} + static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) { const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; @@ -638,9 +644,6 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, epfn = spfn + hugetlb_vmemmap_size(h); vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), OPTIMIZED_FOLIO_VMEMMAP_SIZE); - register_page_bootmem_memmap(pfn_to_section_nr(folio_pfn(folio)), - &folio->page, - OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); continue; } @@ -706,111 +709,18 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - -/* Return true of a bootmem allocated HugeTLB page should be pre-HVO-ed */ -static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m) -{ - unsigned long section_size, psize, pmd_vmemmap_size; - phys_addr_t paddr; - - if (!READ_ONCE(vmemmap_optimize_enabled)) - return false; - - if (!hugetlb_vmemmap_optimizable(m->hstate)) - return false; - - psize = huge_page_size(m->hstate); - paddr = virt_to_phys(m); - - /* - * Pre-HVO only works if the bootmem huge page - * is aligned to the section size. - */ - section_size = (1UL << PA_SECTION_SHIFT); - if (!IS_ALIGNED(paddr, section_size) || - !IS_ALIGNED(psize, section_size)) - return false; - - /* - * The pre-HVO code does not deal with splitting PMDS, - * so the bootmem page must be aligned to the number - * of base pages that can be mapped with one vmemmap PMD. - */ - pmd_vmemmap_size = (PMD_SIZE / (sizeof(struct page))) << PAGE_SHIFT; - if (!IS_ALIGNED(paddr, pmd_vmemmap_size) || - !IS_ALIGNED(psize, pmd_vmemmap_size)) - return false; - - return true; -} - -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn); - -/* - * Initialize memmap section for a gigantic page, HVO-style. - */ -void __init hugetlb_vmemmap_init_early(int nid) +void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m) { - unsigned long psize, paddr, section_size; - unsigned long ns, i, pnum, pfn, nr_pages; - unsigned long start, end; - struct huge_bootmem_page *m = NULL; - void *map; + struct hstate *h = m->hstate; + unsigned long pfn = PHYS_PFN(__pa(m)); - if (!READ_ONCE(vmemmap_optimize_enabled)) + if (!vmemmap_should_optimize(h)) return; - section_size = (1UL << PA_SECTION_SHIFT); - - list_for_each_entry(m, &huge_boot_pages[nid], list) { - struct zone *zone; - - if (!vmemmap_should_optimize_bootmem_page(m)) - continue; - - nr_pages = pages_per_huge_page(m->hstate); - psize = nr_pages << PAGE_SHIFT; - paddr = virt_to_phys(m); - pfn = PHYS_PFN(paddr); - map = pfn_to_page(pfn); - start = (unsigned long)map; - end = start + hugetlb_vmemmap_size(m->hstate); - zone = pfn_to_zone(nid, pfn); - - if (vmemmap_populate_hvo(start, end, huge_page_order(m->hstate), - zone, OPTIMIZED_FOLIO_VMEMMAP_SIZE)) - panic("Failed to allocate memmap for HugeTLB page\n"); - memmap_boot_pages_add(OPTIMIZED_FOLIO_VMEMMAP_PAGES); - - pnum = pfn_to_section_nr(pfn); - ns = psize / section_size; - - for (i = 0; i < ns; i++) { - sparse_init_early_section(nid, map, pnum, - SECTION_IS_VMEMMAP_PREINIT); - map += section_map_size(); - pnum++; - } - + section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); + if (section_vmemmap_optimizable(__pfn_to_section(pfn))) m->flags |= HUGE_BOOTMEM_HVO; - } -} - -static struct zone *pfn_to_zone(unsigned nid, unsigned long pfn) -{ - struct zone *zone; - enum zone_type zone_type; - - for (zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { - zone = &NODE_DATA(nid)->node_zones[zone_type]; - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; } -#endif static const struct ctl_table hugetlb_vmemmap_sysctls[] = { { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 66e11893d076..0d8c88997066 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,8 +9,6 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include -#include -#include #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio); @@ -20,10 +18,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list); -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -void hugetlb_vmemmap_init_early(int nid); -#endif - +void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -69,13 +64,13 @@ static inline void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, { } -static inline void hugetlb_vmemmap_init_early(int nid) +static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { + return 0; } -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) +static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m) { - return 0; } #endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 69ae40692e41..b86634903fc0 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,7 +32,6 @@ #include #include -#include "hugetlb_vmemmap.h" #include "internal.h" /* @@ -372,33 +371,6 @@ static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone * return tail; } -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP -int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, - unsigned int order, struct zone *zone, - unsigned long headsize) -{ - unsigned long maddr; - struct page *tail; - pte_t *pte; - int node = zone_to_nid(zone); - - tail = vmemmap_get_tail(order, zone); - if (!tail) - return -ENOMEM; - - for (maddr = addr; maddr < addr + headsize; maddr += PAGE_SIZE) { - pte = vmemmap_populate_address(maddr, node, NULL, -1); - if (!pte) - return -ENOMEM; - } - - /* - * Reuse the last page struct page mapped above for the rest. - */ - return vmemmap_populate_range(maddr, end, node, NULL, page_to_pfn(tail)); -} -#endif - void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, unsigned long addr, unsigned long next) { @@ -600,7 +572,6 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, */ void __init sparse_vmemmap_init_nid_early(int nid) { - hugetlb_vmemmap_init_early(nid); } #endif From patchwork Wed May 13 13:04:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237845 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dp3khBqD; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20880-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4853Y4z1yKH for ; Wed, 13 May 2026 23:11:36 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2q2RGKz2yYy; Wed, 13 May 2026 23:10:27 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::632" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677827; cv=none; b=FJPRA6xPL43Zm19B8xWVHmhbV3GQdl7kCsGZKvFZhQmisXdK07RAPsMD6twUbVkugEGxqvm0I97PfRd9A+kbnCG7DyWG+QcrouFbuV4RkhIxu+JWV5oBaOXc7uew1TGD0PsZEmSHMRgKyW0NhQsW4dMr15fedgapKt1or0zCGidE63ibYaLYZq4trZsSvcjPwnJvVD4IHlPyiKA40pMAkTrWxMxx2JuDe1dxoECiHYpwWVbKGdLamB+zUc+mE4/8XT4nymd5GxMwItVzKPBXRxQRf7cfyaRJTAbhBtG1te0VZZnLGtT2VSyNENzYOr5cYob68YFUhG5pU/RKZ9O+VQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677827; c=relaxed/relaxed; bh=MIg/xuUWJQhpBPpGQxQRETv9f5LqZHFddPqDq17HTHk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DmoD+TezRZAsmH3S7TrzlPo3q+wCnV3YQvz6yBQTHXYiJxQR7f/TgCH3u5nxNrppl0U9yv9EGqGKvb1MU6qHLackPxJ1vqwUpmW0OAXVE49MmyyzZrk71NaOfonkk1704pHIG0HFi6r3EVgkgIJamG7Gqw6Ilt3XsZxarNMs2PV47Rm/VUGc4epcClljxVQhduIpT4Mxpa04LDvHjwP2s9bN3qKrP9OyNXPqOrr2wssatbQ1m+uJWae2iZSQQi0mEYn4i91igiOkaGVXsBYgNcHx6CzBCojP5O6WjaOneCrVyTtF9nBrCOLwqsBhVmKRcAQMGxvpouNqmRkIZLv0eA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dp3khBqD; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=dp3khBqD; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2p0hYbz2xqn for ; Wed, 13 May 2026 23:10:26 +1000 (AEST) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2bc85eda6b6so20791205ad.1 for ; Wed, 13 May 2026 06:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677824; x=1779282624; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MIg/xuUWJQhpBPpGQxQRETv9f5LqZHFddPqDq17HTHk=; b=dp3khBqDa9iw+RXj3WO/zCmK0fmj6YL9P5EoEX9O0Wx5ag6t1uRCwxdI/Na5CgcA2v 2q7OY9GLhzeNc71iRc8jwjU+Tx1etAx1YMuqJzwsSTVXxZGrWtn7eDA6d/xwvC0RUv6r hAbNP8Wcg68gjOBrz8CfCYX1inZBX+VKHCC2ailt7okWXxtpNxtGRzcIn5pmxF8I2gMJ zI697XGIRNshSQH4QciaAhnOO8IOreeedUjYHck1e/LMpcJriA959YU7XHX/uivYfkZU bFrKVM0Uwu1Mvqgl4UgCIot/mqJOhV0L3weq+72f7QYD3r6J3Xxi+8w7uJbd0DFf8rNZ /8/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677824; x=1779282624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MIg/xuUWJQhpBPpGQxQRETv9f5LqZHFddPqDq17HTHk=; b=m4e6d9598/Ibx0CxPdYrpqGCGN+DaxwR2753VMfIK1yVRG7JY6EcpMYwCNg7IMyXcR lOZRV++PLCbu0IlCrkJNZvVWAL8AzkI9F8ymGHnlUp5LzLS5u+GJo69WhZbQx9BJnUcs j1fG+yMWZqwYdXERxx2IlTy5j4Nr7LTEnTQwdPHsMN57uio95JteEongi/s5oXnCn7Dn A3hpL8DR6Dm8eTHJpbTCXff5OGpPNMWQq4V8FDgVDVq7hgFFI81lsXCGwXTk3A8Vk37R Akh+CIPKksTjaILSyX0cTs0LSga+rLrwQxyH1ktb3QgHppVi1xRNXLdWnzTb+8tDLOoj R5hw== X-Forwarded-Encrypted: i=1; AFNElJ+/WaELyd1XIkMd6D/q1qZRUrgzgut06Fyv01/a5anpW7VTDqzJKWQKB4mSdtuKEO5g2XT6Eomd3CjtSEo=@lists.ozlabs.org X-Gm-Message-State: AOJu0YygqzxFaj0xW7LqqvbDgZN2Kj+Uj3W5JkAvWJ7wFU6JhJQQ3jKi 4uXvhweEw1jxRsWGJkcsOqvxmLULYRj+rwH4qR5FvT1LRPCwO9hk6+ZPzbS59DFY9dk= X-Gm-Gg: Acq92OGKVSo0+OqgDXJvgBLUhXPCVjkAWQficI9t9db0X00XPb0z4kghZvLpOl10eqD idjdqlHif7Y+D49WW1yziRairEalLUM6G7NWXzm2Yh0OxJ0MvkEqH3XxOX2cO8SKEp3aBvDKqQx qFRzfxhXL2FHRKAgSTqsTh5x8ab2/4KiL6ob81zPDh40o/Me/v96WfREblcUXxj2vBCGHTIwk0Q nG5b9f+XzYid8xgtm/Cu430aJ99Sz6lhHnIpKWde3ZUYI4wrVaqftSLkciCrpBmxVRbOaS7fDDR eJFrij2696U0zRt5P6zvCWJUb3Ywqqjs3x+jZI/pA/KkNhBC2DfGeKYAezun62DFqaJufiP1jqm 4LlxsfwrqaOsRvy1jUI6A2pk1/AWa1qVeueCaNQ2BxCgSSIycNpeSbq1nIx/pC8Cs6pn2YcSrbj c+BeFKa/YCjg4M6xQHy6cn4nKFsZ6l9jf4o/vuuAj8XIAzk5hikI6wA1pGZoQ= X-Received: by 2002:a17:903:bc2:b0:2b0:c45a:bc2 with SMTP id d9443c01a7336-2bd27139901mr29366285ad.16.1778677824098; Wed, 13 May 2026 06:10:24 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:23 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 31/69] mm/sparse: Remove section_map_size() Date: Wed, 13 May 2026 21:04:59 +0800 Message-ID: <20260513130542.35604-32-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org section_map_size() no longer provides any shared logic. After the sparse-vmemmap changes, its only remaining user is the !CONFIG_SPARSEMEM_VMEMMAP path in __populate_section_memmap(), which can compute the size inline with PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION). Remove section_map_size() and inline the remaining calculation. Signed-off-by: Muchun Song --- include/linux/mm.h | 1 - mm/sparse.c | 15 ++------------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f39f6fca6551..fef39be8acd2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4845,7 +4845,6 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) } #endif -unsigned long section_map_size(void); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); diff --git a/mm/sparse.c b/mm/sparse.c index 33e89bf1ec0c..47349f6f463f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -222,23 +222,12 @@ size_t mem_section_usage_size(void) return sizeof(struct mem_section_usage) + usemap_size(); } -#ifdef CONFIG_SPARSEMEM_VMEMMAP -unsigned long __init section_map_size(void) -{ - return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); -} - -#else -unsigned long __init section_map_size(void) -{ - return PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); -} - +#ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - unsigned long size = section_map_size(); + unsigned long size = PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), nid, false); } From patchwork Wed May 13 13:05:00 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237846 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LYmNfEFG; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20881-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4H73Tkz1yLT for ; Wed, 13 May 2026 23:11:43 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv2w1lDBz2yZ8; Wed, 13 May 2026 23:10:32 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::632" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677832; cv=none; b=k8g7AY/zE0E9DHf/PAoGEvek4nibveJfiaBQm3F/KWbJqsnCcnA7H7Y7SvZV7l6z2UDGjrVExm8Gxhe4yvba/is+Z5DqFdsCXTzNyUl+f/et5WXq8Mgt6f5yo4rRIiLwz3bnffJYkMrLOr0uzBOdOE8c9bwHOpAc5ybAxa3hgGrpBj4DeILT7MVZcjt/WXtWT1iQ6j16LO3UyH03pjOWUKBW+5MGsMvwKeEdZng5ZgSxcdYZHnn/5IN15F4PPmJ/jrSVEAkcuWbRdbxJuDFe4ucC60bVH34Vjow3HbcoBi8/bKca+b2ECK+tvY2UvHLZVIgQxXUjBujPPxoZFD0aXw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677832; c=relaxed/relaxed; bh=u1XolciG0Jcg/XeXppdFYJHitey8LmRb9A3PbbH2+hw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jMqbl0khaFD1bP1uEy47UMiYNQpGRZ89NAomq0/1BGoQWaozhaeL/euFjzipttjmG1Q50X2591iiXbE6vN5fn+CSeKriwjgPns1JA3j/QLllCkUNIOYHxq5/p+ZkTre+r5TsN0GMHwB2uTTxA85y4ZLYsyr9ZuBwYoaVLab0H7l/CvGADOKP6sMMT11WoeKif+qe5KRyN9KK8MpGMoo5FPozmCC9mMyb2Zl2W6Qu89/L7vk63RFgCfrKjErpr2wThzvLgqh5i4pDpu8gLNNaJwGb+KvvyIz4WFrPtG4R5MS4dfFm/6cOyckuCsbkPm7bJA/ljqD5yQ62o8phygEddQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LYmNfEFG; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LYmNfEFG; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv2v3Ckfz2yZ6 for ; Wed, 13 May 2026 23:10:31 +1000 (AEST) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-2baef9f5ecdso41265485ad.1 for ; Wed, 13 May 2026 06:10:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677830; x=1779282630; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u1XolciG0Jcg/XeXppdFYJHitey8LmRb9A3PbbH2+hw=; b=LYmNfEFGlt2GjGCWyZeEmCg7tu+eS/3ol7wtM+bj+yBbfYaLzM62RyhWOnHN1r1VXd qYjIokDjCHROZOHpCrIpcV3pEawW46Ew6F8GfSuAXtwxYgL5wiihllRpBZKN+fmna426 elh/VQaKyIXNcz7DjafMsNgo0mEVvPd4zL4FpPZA3kQxqgei0EyJtcP3Zs2Oe+dj/Nj0 yJBUWulbDuMeTX1saCQ8ZmlKiDNRK/JsVShvdE9HH4Ms7Cehp+RzyDbYMslpHC4tQn5x rW0c0JjNWcjmnK0GJHGmPMZAUmXl6Og77zrxqWGtG+zzuEW36NHTii5GOwpbEoHu4lL8 aL5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677830; x=1779282630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=u1XolciG0Jcg/XeXppdFYJHitey8LmRb9A3PbbH2+hw=; b=kcJ0DRlv+gCjeH+jcsRdjwpq8by/h7HUiHEpwY/LB9VtBPwNhTlyUhH/Y94MOdhXZq kdQ+R+oZqRm3K36mgMpZDVhqwaMg2IIxoNlNBmg1swUIg6k/GV1C3FqxGxi1p9qZDZvx /rtXJ/nfFMt5Md6SO6blkZ1ZKJ3TENgxsObukomD+hJHPZQ40SV8Tf9O6HO/xqnyzh+6 nkw8lR83ERvfzF3RIM8iX1uMxMpYtRdp6B3QF5PZPOFn7TTsqHux5rgo9BtCZJYi9Nts cGJy0DrOoGhvRjvhqlsr/TEu7xZE6Qsp3NAWAO1TdyfoZh3NCg86ao3SGYRyyKSd3nGe umoQ== X-Forwarded-Encrypted: i=1; AFNElJ++ygFyvvd5oMq5qInGvd+Js0gJq7CRmF6/k37AWQn3DZeDOU3C8bNYVBrkHG3uMUnart1Vg+flqfyLmhE=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyORnh5uCATbr55IaVI2Coz0g7Qnrkqg1YiQtqgmxShwpfL8tdq 4ZdoHIu/fnq6caOclwHNcEVXoeMeb6bHwQepFtokvi+BUS4/SmcDYmp4ucNGi7Wnwvo= X-Gm-Gg: Acq92OHR5/Pc+cagD793x6djEURWYOIToSKVq6YxXsdfYZI98zTcwNuzm3ZbBWNujzi jULq3ztO0I/8cfEk9Y/afgq65fzrWyRd1qczLeQu7qTRRNyHmehvsUKRN2+OOjCsdW5DQQ2MYRo MwFWKQwWDhNh3EXCWXm2Sq6kflLjcf85RgfTPi0hyhm6wM5hg8dpMwgLlyKaJm658uBEBzseVKr VgrZZLqEN8wl08KP+AHsHCa/wgTtL8E2XWh4dM5FSyBCCONNv1fTW6K4uAqhSHWcCbrE2UyFBhk z1GHsIJWcyegY3Cv3Z67LgQZHWl0KSzUT4xgNFGTydiK9wIY4YMT0TUA3mncdNUYBlHJuuRpoWT RYWEyjKuaRmvJ3Su4BJNx8CERyBNZ9n8EEtI63BhUdB4bh2CHvbPpqCOt2viFa6SZTqi1s3LIbb IqAlC6BdsKsvg5QOxdF5m24gL+4nnCFtfc6z7EoDcIsKOK0tjVP1k3d92uM8U= X-Received: by 2002:a17:903:17c6:b0:2bc:8f62:990b with SMTP id d9443c01a7336-2bd2773e38fmr35514035ad.41.1778677829549; Wed, 13 May 2026 06:10:29 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.24 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:29 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 32/69] mm/mm_init: Factor out pfn_to_zone() as a shared helper Date: Wed, 13 May 2026 21:05:00 +0800 Message-ID: <20260513130542.35604-33-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org pfn_to_zone() in sparse-vmemmap.c duplicates the zone lookup logic in __init_page_from_nid(). Move it to mm_init.c, declare it in mm/internal.h, and reuse it from __init_page_from_nid() instead of open-coding the zone walk there. Signed-off-by: Muchun Song --- mm/internal.h | 1 + mm/mm_init.c | 28 ++++++++++++++++------------ mm/sparse-vmemmap.c | 14 -------------- 3 files changed, 17 insertions(+), 26 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bf30617c78d8..18276cd15622 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1354,6 +1354,7 @@ static inline bool deferred_pages_enabled(void) } #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ +struct zone *pfn_to_zone(unsigned long pfn, int nid); void init_deferred_page(unsigned long pfn, int nid); enum mminit_level { diff --git a/mm/mm_init.c b/mm/mm_init.c index 6723c604eefd..35c99e5c215c 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -686,25 +686,29 @@ static __meminit void pageblock_migratetype_init_range(unsigned long pfn, } } +struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat = NODE_DATA(nid); + + for (enum zone_type zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { + struct zone *zone = &pgdat->node_zones[zone_type]; + + if (zone_spans_pfn(zone, pfn)) + return zone; + } + + return NULL; +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * Initialize a reserved page unconditionally, finding its zone first. */ static void __meminit __init_page_from_nid(unsigned long pfn, int nid) { - pg_data_t *pgdat; - int zid; - - pgdat = NODE_DATA(nid); - - for (zid = 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone = &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); + struct zone *zone = pfn_to_zone(pfn, nid); + __init_single_page(pfn_to_page(pfn), pfn, zone_idx(zone), nid); if (pageblock_aligned(pfn)) { enum migratetype mt = kho_scratch_migratetype(pfn, MIGRATE_MOVABLE); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b86634903fc0..f1c3b2d0f23c 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -138,20 +138,6 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } -static struct zone __meminit *pfn_to_zone(unsigned long pfn, int nid) -{ - pg_data_t *pgdat = NODE_DATA(nid); - - for (enum zone_type zone_type = 0; zone_type < MAX_NR_ZONES; zone_type++) { - struct zone *zone = &pgdat->node_zones[zone_type]; - - if (zone_spans_pfn(zone, pfn)) - return zone; - } - - return NULL; -} - static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone); static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, From patchwork Wed May 13 13:05:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237847 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JJCA3S2g; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20882-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4R4ctjz1yKH for ; Wed, 13 May 2026 23:11:51 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv315BjMz2yb9; Wed, 13 May 2026 23:10:37 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::629" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677837; cv=none; b=Oz7IsRuVplgdkZUucShs5o93YF/gkvIGkz36qfjFIjpK+B7N43wPsuckTZopyMlDT6ZOCylJfsWTdasRYMqk+UrB5NCl/mNPhvMvi/38HBfTJRTH3+NPdR412WqCvPeR+7xAdcQB0pjfkyoh0i1gcbvKEXdSkYQPdaZq7bazbUV+w0P669upQ88VhGoqFnLSG8WQMAiFIFLhJplV/aFxgS5N3Djpa+klu//Du5SEsTWM5Cnmkxtu6wP0YxPAS7YDBrGhzbjwir+beQAio9R5oYU2iNtidp/JIB7mIYdX8LCHxgBA4ZiJp+GzR2maI41S8g+tAD2+L3uqV4srQIVUhQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677837; c=relaxed/relaxed; bh=UOmi79uUam5ycb0mnRYplgXKnDDShon/X8fwxFGausU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IvlNEDqFKxSKIY6A7O6T0qPXNlcAJBZyV7b79BYR7H7/Kep4h7LQIJSBF4GvaepFvPpIWD7rd7mzDlnrN1uqfz1eJijNN/q4X6/TD8SwFhNdA4+zv0DueDvqVavRiypEKlxomPLgcvFHSiEkerWgmxcDUuduL1ENpnxluPsd+YzrP1hfYPvh8usU9CbyvwHMI7S64q2aSfYtmGNwJ85/jx4lOarT9bYY9beCYxXMGdBRUnplBKj5EVs/Qdps6A9cBAWAh/ccG3MSqcZhKbnPtFbAyKI8ae+3ssjE7dx1wGL42dCpXiyqWNay8qnrOB7cwI2gpvBpjpNbBryiTtibjg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JJCA3S2g; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=JJCA3S2g; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv310Jzyz2yZ6 for ; Wed, 13 May 2026 23:10:36 +1000 (AEST) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-2ba21d32776so45340205ad.2 for ; Wed, 13 May 2026 06:10:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677835; x=1779282635; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UOmi79uUam5ycb0mnRYplgXKnDDShon/X8fwxFGausU=; b=JJCA3S2g22ulFbDmdchBnwRZ0CTs8T8sHUwkUHCugcoaS0zrdDRKppFxkvE2LmzlQl S9RoK8adyt3DnH/+HVHmL7DV+a1FdG5WP7ID0T8iUnLz1BB+XVod26Ax63rA5p+f0LeO bF4qL9cd/p+dirdgWmD4BEH16+D5d2ei0Bf/IeXbb3WpY9qi/K+6nXlva3BCivNX9D5D OoO0oyn0NspPYWgIFpC9yS6j3mGZkVk+4ODi2VaUx7uBjUndu0B3IKCjILXk5D2+rj7i 9YmtUJNhvA49KPiIrMBIJLQUjMVDqJczurwVIK5ZxBs0Vd7HoK5+7dkCQ7oJz6Ph3d6y lwHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677835; x=1779282635; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UOmi79uUam5ycb0mnRYplgXKnDDShon/X8fwxFGausU=; b=iI8eERuBAefGshT2EaA/9HlpCCKr8QeDpcFU/otFxAoBhf19dv0XiGik72VyRhHpbe Pst5Qzi2q3fWEacADwwkaEd0nkwK4qVyiZPZKq5IuEeNnI6l8YF4HvY9s7I9J67iL64/ rvjzMWkCrAQ1lpP+6QPjcL7mvRip53AG3Nc0qVs5iicK37dGEFqdyU7KijmdCBwBuivU BrkG1hQ3HbXyIXpbbDHXkf+yQeBzUUVFUYQosGu13kEud07i0FcQiJq38KiMxBZ0A62o p55AgKj8LtaEG3btUGGf7AcHfyF9qiHRkaPmwB0pyJwgviavMxUB4uGdD3+Zos7Ab6h+ fzsQ== X-Forwarded-Encrypted: i=1; AFNElJ9sU2msMoyPeI/awLaaHRhrG70Btxzy0mMXTE/kPJo72mSWtMgpTrhfQgV/VN9VEzGNhu5FgfNn60LZKIg=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwBsaZTO6Kyb+JsRXXOPy+WWrOgWf56VBgZtLbjXbXaHgVYvu/T Lr8uYeyDC2chlTf8ZSWbkIvubNScYYuHLUI+L9S8qlc5+lrHnDBA4bTOur2JDHW5m+s= X-Gm-Gg: Acq92OH0RGdDKoeVCfmilr5uM1RtPw2YCAa8e4bIROvrM3g1T4jdRILAyHrAHECqWWY QOFB58iEMCgXb9MmU7JyPSzp3NeMt+t+83H0V2G3pVYaBMls0MbnDQ8yCQRZ0Cjk5S7S7zOeA2A YKmbjc7ptft8WAlh1ui++mSf/xvq7aFnu1SkBGgjf6gYAH7JwH1tYLyWu8uq6f+RlNora/7Escm WBKVcfq9wtiInFN6L2AnhUwWzOs/j8+7mcQYRnTeSBra9A2Evt5j1qaUmD0r6fHFbbDjNAq60u6 yOrlVf7RrrgMZaOzimTYC19D4rp+HSpfXtJrP753zH39jRYApHiS/gvX7m2Kd7b78ilADYX5HTz 951nprkaPGJrd457wNbI6qmv9FX01FAOU512ogQdsLm0Hh0EVYDjhN7RZKpdmVNQ8q8G31GkyWz d460VldQFW3n4/QLsmYdeQCOe/jLQOvhp8De1NQhkNWDyJzOu376MV+W6nUpE= X-Received: by 2002:a17:903:8cf:b0:2ae:ba5f:3ac3 with SMTP id d9443c01a7336-2bd27133c5fmr20492895ad.2.1778677834864; Wed, 13 May 2026 06:10:34 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 33/69] mm/sparse: Remove SPARSEMEM_VMEMMAP_PREINIT Date: Wed, 13 May 2026 21:05:01 +0800 Message-ID: <20260513130542.35604-34-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org SPARSEMEM_VMEMMAP_PREINIT was only there to support HugeTLB's early vmemmap optimization setup. Now that HugeTLB bootmem vmemmap optimization uses the common section-based sparse-vmemmap path, sparsemem no longer needs a separate pre-initialization mechanism. Remove the Kconfig symbols, section flag, and empty sparse-vmemmap early hook, and always initialize present sections through the normal sparse setup path. Signed-off-by: Muchun Song --- arch/x86/Kconfig | 1 - fs/Kconfig | 1 - include/linux/mmzone.h | 25 ------------------------- mm/Kconfig | 5 ----- mm/sparse-vmemmap.c | 13 ------------- mm/sparse.c | 23 ++++++++--------------- 6 files changed, 8 insertions(+), 60 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f24810015234..ed2aa0e4c472 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -148,7 +148,6 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 - select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select ARCH_WANT_IRQS_OFF_ACTIVATE_MM diff --git a/fs/Kconfig b/fs/Kconfig index cf6ae64776e6..ccb9dd480523 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,7 +278,6 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP - select SPARSEMEM_VMEMMAP_PREINIT if ARCH_WANT_HUGETLB_VMEMMAP_PREINIT config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d6a5dd042c25..b9baef8cca91 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2092,9 +2092,6 @@ enum { SECTION_IS_EARLY_BIT, #ifdef CONFIG_ZONE_DEVICE SECTION_TAINT_ZONE_DEVICE_BIT, -#endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT - SECTION_IS_VMEMMAP_PREINIT_BIT, #endif SECTION_MAP_LAST_BIT, }; @@ -2106,9 +2103,6 @@ enum { #ifdef CONFIG_ZONE_DEVICE #define SECTION_TAINT_ZONE_DEVICE BIT(SECTION_TAINT_ZONE_DEVICE_BIT) #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -#define SECTION_IS_VMEMMAP_PREINIT BIT(SECTION_IS_VMEMMAP_PREINIT_BIT) -#endif #define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) #define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT @@ -2163,24 +2157,6 @@ static inline int online_device_section(const struct mem_section *section) } #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -static inline int preinited_vmemmap_section(const struct mem_section *section) -{ - return (section && - (section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT)); -} - -void sparse_vmemmap_init_nid_early(int nid); -#else -static inline int preinited_vmemmap_section(const struct mem_section *section) -{ - return 0; -} -static inline void sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static inline int online_section_nr(unsigned long nr) { return online_section(__nr_to_section(nr)); @@ -2427,7 +2403,6 @@ static inline unsigned long next_present_section_nr(unsigned long section_nr) #endif #else -#define sparse_vmemmap_init_nid_early(_nid) do {} while (0) #define pfn_in_present_section pfn_valid static inline void section_set_order_range(unsigned long pfn, unsigned long nr_pages, unsigned int order) diff --git a/mm/Kconfig b/mm/Kconfig index bb0202cf8b15..c26d2d2050d5 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -410,8 +410,6 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. -config SPARSEMEM_VMEMMAP_PREINIT - bool # # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. @@ -422,9 +420,6 @@ config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP config ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP bool -config ARCH_WANT_HUGETLB_VMEMMAP_PREINIT - bool - config HAVE_MEMBLOCK_PHYS_MAP bool diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index f1c3b2d0f23c..dde4486195ad 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -548,19 +548,6 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, return pfn_to_page(pfn); } -#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT -/* - * This is called just before initializing sections for a NUMA node. - * Any special initialization that needs to be done before the - * generic initialization can be done from here. Sections that - * are initialized in hooks called from here will be skipped by - * the generic initialization. - */ -void __init sparse_vmemmap_init_nid_early(int nid) -{ -} -#endif - static void subsection_mask_set(unsigned long *map, unsigned long pfn, unsigned long nr_pages) { diff --git a/mm/sparse.c b/mm/sparse.c index 47349f6f463f..eab37504819d 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -316,27 +316,20 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, if (sparse_usage_init(nid, map_count)) panic("Failed to allocate usemap for node %d\n", nid); - sparse_vmemmap_init_nid_early(nid); - for_each_present_section_nr(pnum_begin, pnum) { - struct mem_section *ms; unsigned long pfn = section_nr_to_pfn(pnum); + struct page *map; if (pnum >= pnum_end) break; - ms = __nr_to_section(pnum); - if (!preinited_vmemmap_section(ms)) { - struct page *map; - - map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) - panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); - } + map = __populate_section_memmap(pfn, PAGES_PER_SECTION, + nid, NULL, NULL); + if (!map) + panic("Failed to allocate memmap for section %lu\n", pnum); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, + NULL, NULL)); + sparse_init_early_section(nid, map, pnum, 0); } sparse_usage_fini(); } From patchwork Wed May 13 13:05:02 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237848 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lHcDCZRH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20883-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4Z5qJrz1yKH for ; Wed, 13 May 2026 23:11:58 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv370SQDz2ybQ; Wed, 13 May 2026 23:10:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677842; cv=none; b=CLSmIopyWNwLQMJSG2FpE2d3vZLcClMZVPMDv97SOOLlV8P2N913Aey35RCD4OtvbBC25CWt5wHZOcm5jAVEMreu24i6QQnq1K2TqmGt/6BW9wiGnDMiw268UoSMc4R895xezs2c5+s3oelRIG6V8Yd/PnQQ+n+JuD+2T5apgFao+0HyzgXuU2FV9GHy/qsoEDpUBbnXR5Pmi6EN2yZ52/cGUkC8wyfYBKpYYi3eDzzs4wfW/33nPjcQlJFrIxdLe0MsnsJswc/FN40Z0sQrP3vgXiE2ztqMO+/VE2zzV2Rg7DlaRAly7SBALAYx91X3UujIYSuyv0BRkYbIX5diSQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677842; c=relaxed/relaxed; bh=LjpGKbpR87QdpAXzElJEcltafcXUZyW4c5xpVY2ebxg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MuO+h2auuwRU81Ca8HzCcnMr0IfNXsxeUBfkRL+UPwoPDG6fqqssO83KQr7flUsfvs3lnSx3E/wi8yOL0kbbFrVGm4vTw30r6dMJNkpecXIa3GkyDge21rdEYSv/VQr6wAAW87eJDGH7Jf1M5A2RIi25hHGhn1dMmxNCs2zilAlRxWRz8ft19YteyX14NEYZnGPN5qAl4Y070Ey6ockaX7I9RjVgFO48MSbG2yBp9Swd7XadFJugdgn26JHEZMmr04ALZGt/eXQLIJczrbRcKBezO73XsxZkT21m1iaSRg8xNbLP2tdeqV6R3z9FqywunBuqdY0L9HKxN7IPKLuZRw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lHcDCZRH; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lHcDCZRH; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv362R8jz2yCL for ; Wed, 13 May 2026 23:10:42 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2bc763e2ba8so21663805ad.3 for ; Wed, 13 May 2026 06:10:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677840; x=1779282640; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjpGKbpR87QdpAXzElJEcltafcXUZyW4c5xpVY2ebxg=; b=lHcDCZRHTBlitEYvtPTFj0gQ2fKVSYsNK+eUgH2SmNqLdHgAaRXP6EAVDSkeynrpNY P/evUY/DrTQCj1BzymiLsSUfa6l8ml+XpgjeKu6Et2duxPLfl9B72pjVT0zWbaRrgcdu 8sJcaOqsF90CMNtbqhwvRbmGqSj8RZZGzg0Wxw9IymE3dQh4idMRHnZTc2RSdxgrlDII RuSi5De1X0bsr0+1lP09DqVNGgCEZh/t6ZrGAXqWAxTa8aeoiBKRwJUwok3gCiepXc0A A0YyQXveGs9G2w14X+9Xsib/13/16jn7R6qk6LA06hZp8WZpYm9CbQQXfZ2k5BXuEj8P HLHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677840; x=1779282640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=LjpGKbpR87QdpAXzElJEcltafcXUZyW4c5xpVY2ebxg=; b=ZCeMsr1co2gm6FKshEclWLK4ANwWHa4unjJDNMfaC2UiVs9B3MIRY6e/sdoYNhv3mv 2tzgXWjYAZFAi8GFnIQjUK5DqXeZH0XuYrGsDnACD34UyMHyJZ+ARiHcypx6dBAZTDF8 mYO8d3Bh49yyyrRXgf8wPkgjOycGNA89O+j2gxnoVDoJAABj59AO7hIFMPEwg7LwkcjQ c7y8BuG/svtEDwBXSejWnTNVbm/jkeRmQVrXhknrAfGaoqQwWBmpw9JXCKpXJmLqY8Tx ClI8MbaFY55TmZShIUm8cmEZJDr9jKZnserVTweuic8kDZ4XeIpr+8qx5FdB9VkIaV3x sjIA== X-Forwarded-Encrypted: i=1; AFNElJ8sfIYYZhJeFtbHhCCZK6njgMSHjTR8J70wVKY0rhkbxJEDYGCFy9AnRoFAjxoxsEraNowqJ2yQHQvYbVI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxdeF/4zG7aZZCcTmwbR/Muehn1mRi/czN6u3+KTbOgdCtfF1SG dECV0WQKLRlQzl9arN+m0KmU1Tc2o88MI21hSqTS/Tr5ohR9Ox9O7fqdDUQXJ3eZNkE= X-Gm-Gg: Acq92OEgtQrIt0u4+rqktpVfnQQNCUpvR67BOCK4jsL6rOlbJmYkKf14bPZ6GQIcnjl aVkSnv+cZH2Zc99cENGzZBMM4gCIlBt/khinZVXEti6oBiLzCZXnMD/ruKrOzzsR0X7jLOtix6k NUorBw0PDroAcayV5EQ4IXGuTkc/PQzzMVxxCg5SAkHxbWTa1jKFxljE/IIbrQb/XbHBEgEP/J8 PhDmhipeTAHtIMY3A5LK8StHlgcSjqTNGQQrV9suLJi+0ouopLbDGJnZkfOKs5q/504aZrD62hC rx0qUP7KJv/X1Yyq4hoiQeftlXo+h/n9aKMomKL3CjBI6FvSWJXIMSeY98GDTK29jVb7q9g1i+D qTUT7HoHfeya1//tvt3KG2mTQ/YBc5oV2LLpxABSatwOoWDpPfRebWKwOqCeX6fe257xfhvf66n uF5peRdGFtlN/DsoqvdgGvV16J+Q9SYxc2aQja4xnjP8ofVkLMaqnZcsymM8rgA1DV4WJIew== X-Received: by 2002:a17:903:4b03:b0:2ba:99a2:c44e with SMTP id d9443c01a7336-2bd275c6b33mr34992375ad.21.1778677840351; Wed, 13 May 2026 06:10:40 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:40 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 34/69] mm/sparse: Inline usemap allocation into sparse_init_nid() Date: Wed, 13 May 2026 21:05:02 +0800 Message-ID: <20260513130542.35604-35-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org After removing SPARSEMEM_VMEMMAP_PREINIT, sparse_init_nid() no longer needs the transient sparse_usagebuf state and its helper wrappers. Allocate the usemap buffer directly in sparse_init_nid(), pass it to sparse_init_one_section(), and drop sparse_usage_init(), sparse_usage_fini(), and sparse_init_early_section(). Signed-off-by: Muchun Song --- include/linux/mmzone.h | 3 --- mm/sparse.c | 46 +++++++----------------------------------- 2 files changed, 7 insertions(+), 42 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b9baef8cca91..a60fd5785fa5 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2265,9 +2265,6 @@ static inline bool section_vmemmap_optimizable(const struct mem_section *section return section_order(section) >= OPTIMIZABLE_FOLIO_MIN_ORDER; } -void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, - unsigned long flags); - #ifndef CONFIG_HAVE_ARCH_PFN_VALID /** * pfn_valid - check if there is a valid memory map entry for a PFN diff --git a/mm/sparse.c b/mm/sparse.c index eab37504819d..54c38ea08190 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -237,42 +237,6 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -static void *sparse_usagebuf __meminitdata; -static void *sparse_usagebuf_end __meminitdata; - -/* - * Helper function that is used for generic section initialization, and - * can also be used by any hooks added above. - */ -void __init sparse_init_early_section(int nid, struct page *map, - unsigned long pnum, unsigned long flags) -{ - BUG_ON(!sparse_usagebuf || sparse_usagebuf >= sparse_usagebuf_end); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, - sparse_usagebuf, SECTION_IS_EARLY | flags); - sparse_usagebuf = (void *)sparse_usagebuf + mem_section_usage_size(); -} - -static int __init sparse_usage_init(int nid, unsigned long map_count) -{ - unsigned long size; - - size = mem_section_usage_size() * map_count; - sparse_usagebuf = memblock_alloc_node(size, SMP_CACHE_BYTES, nid); - if (!sparse_usagebuf) { - sparse_usagebuf_end = NULL; - return -ENOMEM; - } - - sparse_usagebuf_end = sparse_usagebuf + size; - return 0; -} - -static void __init sparse_usage_fini(void) -{ - sparse_usagebuf = sparse_usagebuf_end = NULL; -} - int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { @@ -312,8 +276,11 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, unsigned long map_count) { unsigned long pnum; + struct mem_section_usage *usage; - if (sparse_usage_init(nid, map_count)) + usage = memblock_alloc_node(map_count * mem_section_usage_size(), + SMP_CACHE_BYTES, nid); + if (!usage) panic("Failed to allocate usemap for node %d\n", nid); for_each_present_section_nr(pnum_begin, pnum) { @@ -329,9 +296,10 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, NULL, NULL)); - sparse_init_early_section(nid, map, pnum, 0); + sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, + SECTION_IS_EARLY); + usage = (void *)usage + mem_section_usage_size(); } - sparse_usage_fini(); } /* From patchwork Wed May 13 13:05:03 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237849 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VN8K/pLY; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20884-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4k57N7z1yKH for ; Wed, 13 May 2026 23:12:06 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3D4Dq7z2ydj; Wed, 13 May 2026 23:10:48 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1036" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677848; cv=none; b=Ry/P7VUTVc9ZF1/Q6lTBU5/16FH3XEJi+CS5P+s1VtJ+kY0RQMJMNp18j6nF8Ui4srth3fj7g2/pXvZqJyiBn4KUpF53C5RCDf9DD3Zj7M9oE28MQuWlH8QWcLyA5drsTVN/LcU8zHNgxWwVBcwwgZWRW+hWdG7ePt2fnuwVf5t7JM+L81NI0RJHkIb7vozCj0eGf1HJt8Fx/x1luidP8/jwpOy3pNbs6aleaxPrKP48zw1vv39pDuKrzvFVVzH2x/KhKIqTZGwXwLdynTYKVzdMir47HiV54dLdiM4Q4dTTB4DpawlTKicGzTUAg377mS3ZYzLsziSI5Qcq5f7XWw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677848; c=relaxed/relaxed; bh=K6tm8+Tn6qMHHNbiluBNgNTiouQaD7hYoUCXgTq5wuU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HDD6sGRpolwNg6EBH9FhjGd/mgdr4qp2zSn3h0VnMcdrvRyo3nQTOcepD0sCSrr/ApSVCgk2iGJ1Gg5QmG4iEp8vheI6SnLvVGznYYOcXXcGL1bNsmHikS6A1VHeS0/GLolFsz/SzgAebEF0b46jnyBi2ZWhW9Sj0BrmVZ+jFyzzbAeyGYFYK4nsFvUR/Tezsj6f3lpDAwIgpFomtK0AYSSG6iyhojkyEMhCrAzl78QF2wY9dJi3Ry7C4hK+zxIUhcD6kt9XBRtuxwxEQILsca67AmOJck7saJ17J38gFu9+8du0aMHJboMmqC5VjnSY/VBnIXhNdx+PERKeCfcxKg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VN8K/pLY; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VN8K/pLY; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3C60Jgz2yCL for ; Wed, 13 May 2026 23:10:47 +1000 (AEST) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-367cbac9cb1so4509590a91.3 for ; Wed, 13 May 2026 06:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677846; x=1779282646; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K6tm8+Tn6qMHHNbiluBNgNTiouQaD7hYoUCXgTq5wuU=; b=VN8K/pLY8FjrdZNYDG095U31ceZ1PvacGg29faauripeU4F3kgWAz7SIom52wdfxbp tNgV2lhydlu8EfW4yV3fu4dU0FsqGmY/d5ndKnlOLyp1Dx4cCiaTOdDIzAPvIqEnWYZi dMQGOUe/SOMYtD+zJ+ifRYzlf+7ykOja7m9t3Q9ZWKWBQSHzICeUsJgbnt3flHC5UBzo RrLlxPsHDU6agACzXBcFZRNqtA/QHoaZbO+6c2Y2lOD5YtmcrItRV5AAxh0tbMClJoqc QDhw0Vyy2Wt1MLm403yYC0Bl1r6L2xwJrJCaxJ5Z/9oaCZO/UfhfD0hoLCHrrKdvxO5+ DpIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677846; x=1779282646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=K6tm8+Tn6qMHHNbiluBNgNTiouQaD7hYoUCXgTq5wuU=; b=Lq+NxMfFdpEhZ3Mu6Xfd+DNp8AnwD81PZo8pVpj/3jCea1Gva60uTJ9WTYvDO8kPSr D+lMFd3rZSV3Fy+ZRAIAZfTJrrXgrEfmFfh+oI2OOqX8w3SkW2uyl1lFEsKirn9lhO8P o0gmFdqzdlN7UxiSczgzr/08p1TluyIY8R5rt5PxC3w4WMtujmgH96ClYZYXbZtn+6Ja myriga9rTKYYmtSnrtrvkURugfXkjJLoF/ixITGRkdscNlKYRyOo4vR2h6dKvW6zldqf VL+YbDpr7gm2WqPnz5na1rQkcR/AwqgE/EoTaFtYn57ta4MIbxTAkM0FD+98u7NuWScZ 0njw== X-Forwarded-Encrypted: i=1; AFNElJ+4XexZf8JuykuPE//5txXgz2pbHhAlcOqvei403hGENhY7GE1ItU2kmgMzOebV2NuIOJYJPCXmUUdXKYM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzV2VA78Oj4UpeSY7yNtPtZobbAyZIwULuby09Hk1eWlVoFqNCJ PAUky+RqiKtHB4eq1MspIyIjxwy7RaOckRPQ7EoKLL0X0UGDNzO3lNMdzyVsNdZCAFA= X-Gm-Gg: Acq92OEjg59pFESL/yUcIe4C5XYKS5d/yKqSBgdX/55KXMa2eybCa0IQVWmsLK3jNXM r0QxdtSSCN3g0sTl6xvOtSbieVSnPvnnu9wEl9yb17BPVh0xZ7E6YuUvK+D0hP+E41pG4KQJeVL YN5ok/jLwKfxeme2YhkxhcjUXU55Ir/4nWsu9Q2SO3ljFSUdBuhkKpQr4DbudmqTulhiALOtA3Z JoHOpFdF9EW+Qc9fKVpHs6qgjuEFpwnwCEBB6XS/UK+tP3ONkfn40eOiX+7em9iEGw7v/3o6h8G HhXsbF8F9CIS3B8t2JvXshmFk6puSgsoOBHicxMKGafib/7Ie6aaip7+jmKyeecQsxBqm9wDr6/ eM/D7ZASmIcqBbdVPV1kmc3ZdC0SUvw1yrwlo9fW8Eq9YORb5nnb+JgH9BXcXspcd6D8sInR089 /wbf2eFFVn1u0uatSNkiDBnv2pH25dXPBqrjDN9Axl4SzMkl/wdS6zMWDJllc= X-Received: by 2002:a17:902:d584:b0:2b0:ac1e:9720 with SMTP id d9443c01a7336-2bd30210d95mr31196025ad.23.1778677845712; Wed, 13 May 2026 06:10:45 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:45 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 35/69] mm/hugetlb: Remove HUGE_BOOTMEM_HVO Date: Wed, 13 May 2026 21:05:03 +0800 Message-ID: <20260513130542.35604-36-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The HUGE_BOOTMEM_HVO flag tracked whether a bootmem huge page had already gone through the old early vmemmap optimization path. Now that HugeTLB uses section-based vmemmap optimization, that state is already reflected in the section order. Remove HUGE_BOOTMEM_HVO and its helper, and use the section state directly when deciding whether to mark a folio as vmemmap-optimized. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 5 ++--- include/linux/mmzone.h | 7 ++++++- mm/hugetlb.c | 12 +----------- mm/hugetlb_vmemmap.c | 2 -- 4 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index dce8969961ea..18af8f304b95 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -695,9 +695,8 @@ struct huge_bootmem_page { unsigned long flags; }; -#define HUGE_BOOTMEM_HVO 0x0001 -#define HUGE_BOOTMEM_ZONES_VALID 0x0002 -#define HUGE_BOOTMEM_CMA 0x0004 +#define HUGE_BOOTMEM_ZONES_VALID BIT(0) +#define HUGE_BOOTMEM_CMA BIT(1) int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a60fd5785fa5..9b87d798a365 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -117,6 +117,11 @@ #define NR_OPTIMIZABLE_FOLIO_ORDERS \ (__NR_OPTIMIZABLE_FOLIO_ORDERS > 0 ? __NR_OPTIMIZABLE_FOLIO_ORDERS : 0) +static inline bool order_vmemmap_optimizable(unsigned int order) +{ + return order >= OPTIMIZABLE_FOLIO_MIN_ORDER; +} + enum migratetype { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, @@ -2262,7 +2267,7 @@ static inline bool section_vmemmap_optimizable(const struct mem_section *section if (!is_power_of_2(sizeof(struct page))) return false; - return section_order(section) >= OPTIMIZABLE_FOLIO_MIN_ORDER; + return order_vmemmap_optimizable(section_order(section)); } #ifndef CONFIG_HAVE_ARCH_PFN_VALID diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 080f130017e3..abd79bb85b1c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3169,11 +3169,6 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return m->flags & HUGE_BOOTMEM_HVO; -} - static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) { return m->flags & HUGE_BOOTMEM_CMA; @@ -3265,12 +3260,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); init_new_hugetlb_folio(folio); - if (hugetlb_bootmem_page_prehvo(m)) { - /* - * If pre-HVO was done, just set the - * flag, the HVO code will then skip - * this folio. - */ + if (order_vmemmap_optimizable(pfn_to_section_order(folio_pfn(folio)))) { folio_set_hugetlb_vmemmap_optimized(folio); section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 730190390ba9..66362e553870 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -718,8 +718,6 @@ void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m) return; section_set_order_range(pfn, pages_per_huge_page(h), huge_page_order(h)); - if (section_vmemmap_optimizable(__pfn_to_section(pfn))) - m->flags |= HUGE_BOOTMEM_HVO; } static const struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Wed May 13 13:05:04 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237850 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Lde5Fexu; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20885-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv4s697Wz1yKH for ; Wed, 13 May 2026 23:12:13 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3K61jnz3d8d; Wed, 13 May 2026 23:10:53 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::62f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677853; cv=none; b=a5eSxiTjx8XjuTYUWE6xfBcUgH9B3hfTBCaGH0QwdbZDvlb7wtTu32FASq2u+xvQyXleQSYqIr0x+WQYyCQ6ZUNZTFUmhCFQ/GdJZhULF4FVTRM6r34+PiSQ7G60xrl16/IFAYdmPhTMnt1f89jYL1zeNBaibA0C9ikPImMk0UiSAirbqCHA08riigv1DzIZg49VUkvOuwjln8fgOnLzOL0BKzcs2M9BZsd3NOuSzLeBtjkFxqEiC6eKOuoTQSbvDogIah9XW8HFg0Wyl84410PbOnv2cbXaWKqtPmUUhYRxdTKYzrS0RztlLLwynUehXQ92ZSTjF8A+Hn19l7QtHQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677853; c=relaxed/relaxed; bh=8LL1KUkSPgRngQo7FrouyGUEJqkrJiFH1mc8jZNgwIM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P59g9JWJ5EZr+vXgfyaY8aAFiZfMRXhzmaa8QV+V01rTvn2/s6vnneYeBRmraYmqIe3hWS+KlFOmv4o3ZH47ZgdO+qBc5zJBapA5r/durShcrYiH3QP2TcfB7PoFZbUpstIhz//NYvp6SYUoJStRVOEm5e5Ese+7HBODqepEJ1ri/turISAnDLzCBIZP/wBlwo39ipPjfBykAT9wmNjPOCnX3rFrEw6KruuX5Z/ZnZyOiKU7bxcE3USvAS4LisM3UzAFVEl/1Ki0euuRB5eu6HtuHIP8W/+xjnY7SBSt7eAMpKd3pkZy8boWg1W6SIYwVRndt7CvCjtKCAsg/7astw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Lde5Fexu; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Lde5Fexu; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::62f; helo=mail-pl1-x62f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3K0WYSz2ydn for ; Wed, 13 May 2026 23:10:53 +1000 (AEST) Received: by mail-pl1-x62f.google.com with SMTP id d9443c01a7336-2bc763e2ba8so21665215ad.3 for ; Wed, 13 May 2026 06:10:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677851; x=1779282651; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8LL1KUkSPgRngQo7FrouyGUEJqkrJiFH1mc8jZNgwIM=; b=Lde5Fexua/xyw3U6cyYT/RhhBq9LKleFphXQXG0PRP3YN+R0/HZnpXPGSqtJ2chBd7 AHzaFjln7EkG8cAE7MWWv6QHCXDNRQfDMUseBPjBUwoeK3krCG8w3Hh3Y4ImOtwfV6B6 2WQripsO/UYIWl/kxeRgYAVGHUA+f4RHtpEAhUKlh5ViuFz1dKunCHWKU/WMl+tO9utk UQNIOfrX4qxso4Tf29lGG1PUDmLPYufbhcDB9rf4iZ95e5ABNTtYv19o5VvunVy9qDcM 6uI3ZxE9CzwAvDtl+rccln0Oy49MR8x5DVJfJSHj4Jhq/t+tWhFjnqzbWkNmgrbUWiAV aE/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677851; x=1779282651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8LL1KUkSPgRngQo7FrouyGUEJqkrJiFH1mc8jZNgwIM=; b=KKGAkDxFEOmuVLYrk8/46+Moec/03vapGttGEo/b6pjcp9fGyMX/KOlOwBrQ82Errh heewksFjRciRbX6OiVvDYcKfYOrq5K0NafWFiMotobYyYqSYMp4wVZQk1tVGymp7EPEx qq7biEuvFc+AKXR+1VjFAtjvtlrU3D39wmEu7JSwPX7zyQz4x8C1MerUNntH6MrJ8jpK HHMAooXsEL8vZbxU8NZmzAxy9R4LMdiSnEz9214lCp29GFUkOBAJQtPqJ4YED2Kcod9k 61x0+QC/AYmROKsRtHkqkdl4jcubzY1/LUjg6MmnulH5HqSmmI2lIRpbk1vsIVAxknco nXBg== X-Forwarded-Encrypted: i=1; AFNElJ+dMZ0J3Lyxfh54F1pHYFh7tahvA4H3QEUAbYC+oKlr8oFCnEz8ynuCX8rPD2HxXQ5+DsNDx9rvVpThpBc=@lists.ozlabs.org X-Gm-Message-State: AOJu0Ywv2RjB2aYZ/2j6hBv3f4nBXQ1JTKOGUK6W2GTIDOMOZ2x4ayG4 e3drv/wzWbjOtkq6ly+ORm2+u0kPnPFM4tYJOE+XMEWl2HRzIuq8956C2xIs442Tfhg= X-Gm-Gg: Acq92OGKe6ESBVSIyYVTn7/U6ZagQ3cx40Q6piPuHb+b+VDetfwBZIjfCE+VVw7uDhI AYDjllydZyqW6h03AFDojeM6Gc8p4s4h1RTUZmOFY9Nkg56tL4T1ZhAS8WdRvMkncFvMFaNRr56 5IwXa6xNh5FQw4NNtbPikqvHTixKaIsuFopL4kevkCztBKttLfpclwZOq0g071BmsVMcgwbYH/Z +vx74yYwSHDVWsxE1UFM9fufJOzu2ycXZ5nt4kAWvNcTugQwSOQRSUSX93PrZvgmwuhZC5dc1FO d0q7vxbKF+ZEvDNY6RC9Tvi88sv7eS340OyexAPuQNZG01ZbBC/+roIGNI6r80NFxtAqWLlUmyt L4FAGcE2KSLLrn/G4/pr0PMw1yWvwu8bokL3Fsd1iD9VjiUu0wuWC31hRY/s/1RiEwjML/KORSO LBS3r7529rTKRCpxVUZB7R8KtBJXovtptvWfEkNdgcXQmIQ4Cm4JtKsjAy4bs= X-Received: by 2002:a17:902:b401:b0:2bc:fe65:4e9e with SMTP id d9443c01a7336-2bd275d95bdmr26715515ad.27.1778677851113; Wed, 13 May 2026 06:10:51 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.46 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:50 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 36/69] mm/hugetlb: Remove HUGE_BOOTMEM_CMA Date: Wed, 13 May 2026 21:05:04 +0800 Message-ID: <20260513130542.35604-37-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Track early CMA hugetlb pages from the hstate instead of storing a redundant bootmem flag. This removes the unused helper and keeps the bootmem metadata minimal. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 1 - mm/hugetlb.c | 9 ++------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 18af8f304b95..82dbb9ebead8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -696,7 +696,6 @@ struct huge_bootmem_page { }; #define HUGE_BOOTMEM_ZONES_VALID BIT(0) -#define HUGE_BOOTMEM_CMA BIT(1) int isolate_or_dissolve_huge_folio(struct folio *folio, struct list_head *list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index abd79bb85b1c..74770c1648fc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3093,7 +3093,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) */ INIT_LIST_HEAD(&m->list); m->hstate = h; - m->flags = hugetlb_early_cma(h) ? HUGE_BOOTMEM_CMA : 0; + m->flags = 0; /* CMA pages: zone-crossing is validated in hugetlb_cma_reserve(). */ if (!hugetlb_early_cma(h) && @@ -3169,11 +3169,6 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head(&folio->page, huge_page_order(h)); } -static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) -{ - return m->flags & HUGE_BOOTMEM_CMA; -} - /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) @@ -3265,7 +3260,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); } - if (hugetlb_bootmem_page_earlycma(m)) + if (hugetlb_early_cma(h)) folio_set_hugetlb_cma(folio); list_add(&folio->lru, &folio_list); From patchwork Wed May 13 13:05:05 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237851 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fHn6RvcZ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20886-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv511jQSz1yKH for ; Wed, 13 May 2026 23:12:21 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3R1Jlwz3dHw; Wed, 13 May 2026 23:10:59 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::634" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677859; cv=none; b=N7WMPqXmuaXdGHJZc6gn+kip+jZZsukbKupUgzueMf7uy+E3g6L6EHaWl+PdVnhIcjRgipRxR4eA5EhZ1KEol+oHgSjiMyWsCV0U2dY0HKB386DEY8lIYZ/PdmTYHVRfp9wYboDd4k/mndQ08SubMCpLNyqfPhL19w/jrECwdERMRJEfr/JyqZrV5/kiWNhrZC35nrxutgRRK4GVnWWLTGI6faHcOktkgr8ffcRXtL3rdgWMw1aM1teHSe/+LwzTX3dVbqaXjhbY+un0mJTdpKLgnF3FvIaLuB+3KFLQ+uX4ihoAbnt2cagmcCpwA5wrcgmv5yeZw5SB8v9Gs6F7bg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677859; c=relaxed/relaxed; bh=dToyrDGV72K3gHJcuzQsCSmu29LS3Njm8M96u525pzE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BrdMeGEg84nGG2GaGhxOGR5F7MX5Yk+R2+dZb9TyLzuZ9jEfFlTctBgRkMXkIPPEQ7uKAx8+O74hdIGKOdLHDFaF4DYa89BA+p4ZXxuSSRMJ7GBFo11YQoQh00JzaH0+gcYjq/fXmZ1rQCKbYOLtlH2lBHCDLC5DaQ6WSIw6V1z+3CHyj2CiMYuB/8Lg8JLgrybP8a95X7zVFN6Qf/0Nk6VfPA8Iy1Wo1M2W7eHfsU2o4nmvL1C633JkvwroOKjqGwGC2YPeh/nhcm+PPL4lX9YujRgERMz5ix1vYVue4+/0XgBc+mHLM40KC/z8apCZLwXoNgRWrXQRa6p7DwUtaQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fHn6RvcZ; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=fHn6RvcZ; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3Q2VgJz2yfN for ; Wed, 13 May 2026 23:10:58 +1000 (AEST) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-2b7d3ecc10dso65526405ad.2 for ; Wed, 13 May 2026 06:10:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677856; x=1779282656; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dToyrDGV72K3gHJcuzQsCSmu29LS3Njm8M96u525pzE=; b=fHn6RvcZuw5Fw3Wy1T+yJr9M73TGJ+XUCnyhvxYcXb/cjcmKFFqt93+YgK3KYhADFE AmwCrEth1RtkzwCEG5cyGBgWqxlG9y1+eNsSgcoE42k9nxUQ2FoQLuOR5FI7V1M+f717 MrSi35hXXh4/dgqk2a7WymxTOj8ZpPLHu7rgsnBF7ENR9ssF5HQfTXWrDIfGQCRFDriC Z1S9hvvx/+kJ4s5WbF6wmZPxBzXk/pFQEkPeS9IVZFg2DBaSk32oi0XeHR7R/j5FpRI0 /7T+Hu77pChfxA0lpw4tauyPG5Oo9IAR8XWmGIxi/i7Q1pROlk+9Q1c1I3n+mcpP7dtw 05/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677856; x=1779282656; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=dToyrDGV72K3gHJcuzQsCSmu29LS3Njm8M96u525pzE=; b=eOJMIxWQWdBzAkf6nX2qz3roV+jK1lcd200DbsvIY5sLsWSWldToUjKG4/c1GQRBLr b6iyUr/SE6PkZPAG2NoMR09qwiu7WHjWYbTsX/YZWemjj5aiD+bXkBoPicT9yrscDVlb a27kZeXaem8VXyhmRC0PzT4PPciNMHLgSTTebd+TvrBtUdQeTNc019aCNOgLqzcFUtAo hcedBqE4utnX95TVa73Z9RC1XPA5Y3wL0UrjifoF4BJN8gy2LSpZsleodpeq1iZhvfqx e0N6nH46y4NYT87AEWI+aflZ+pjfo859VcHgYEUWOQKgZiMg7TmtOaGtPlaxXsStQH1c 70vA== X-Forwarded-Encrypted: i=1; AFNElJ9zVsyRySz5u3xACMzxPW73o590ntEAqcQUHrzuRJA9dgJW8cSoRM4wlbXgErsNDG0CVTlAkimUZLh1aJY=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxNX1JPH4HrsPEruWlLCN+xLs4kuxchO3jv6p7nZIdJNamWsF+w ScU/+Pm6AJmMvCp3c10JCBh3DztjKY3ovE9JSlk+wSBvckAz+uZAyzbV7uSkmYoq+2I= X-Gm-Gg: Acq92OFwDftPbywbLc423Y9zGFz/d0wj+M/4neeWjx2nW1U6NouF6oDT+1Mmjuhqbg7 BNWhufc1OW4ngapOY6HUVsDa0WdKcDxHRuNLwfJlnmm+WGr4WRCr/S8h8xh5Sn3CevjKM6DYBYq zq+rfy3gOgtFnwyV7K2i29mSfg2lgK+3pcavDAaTaxQ9b03pLl85zjiVFbL+PGcseuy6kIgonK6 I116gvNFGK6sRUEiO3mAomg0x1N7klMRz7C3y2k/8It+BnFDubkaFTZ4hzWiQyLK96V9jZM2r0V FJVyybFdSpx8v00hvxSFrJHz7M+4GNa8ro5q8Hu4xXnGruZSHcCerAwhZppPgux/EToHrDluruV pBVfwbRDKcvH+kEgvEA1DpkxNIsQ29Rkl5f9LWBrFbD1AATiRCKpXSfwR6n38dlQrIsEM5cmWwv 8rTn3P9rP9kSGQQS95Qcghy40B3Q/Q3uG4wnf5DMM4GaxedSWoKhu5faMNsSI= X-Received: by 2002:a17:903:1b28:b0:2bc:d026:eeff with SMTP id d9443c01a7336-2bd27188006mr39565445ad.8.1778677856477; Wed, 13 May 2026 06:10:56 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:10:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 37/69] mm/sparse-vmemmap: Factor out shared vmemmap page allocation Date: Wed, 13 May 2026 21:05:05 +0800 Message-ID: <20260513130542.35604-38-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HugeTLB and sparse-vmemmap each have their own helper to allocate the shared tail page used by vmemmap optimization. Factor that logic into a common vmemmap_shared_tail_page() helper in sparse-vmemmap.c. It allocates the page through vmemmap_alloc_block_zero(), initializes the tail struct pages, and uses cmpxchg() to install the per-zone shared page. This removes duplicate allocation logic while still handling both the early boot and runtime paths through the same helper. Signed-off-by: Muchun Song --- include/linux/mm.h | 1 + mm/hugetlb_vmemmap.c | 28 +----------------- mm/sparse-vmemmap.c | 67 ++++++++++++++++++-------------------------- 3 files changed, 29 insertions(+), 67 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fef39be8acd2..5281f073230c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4866,6 +4866,7 @@ int vmemmap_populate(unsigned long start, unsigned long end, int node, void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, unsigned long headsize); void vmemmap_populate_print_last(void); +struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zone); #ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 66362e553870..d24143dd6051 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -499,32 +499,6 @@ static bool vmemmap_should_optimize_folio(const struct hstate *h, struct folio * return vmemmap_should_optimize(h); } -static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) -{ - const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - struct page *tail, *p; - int node = zone_to_nid(zone); - - tail = READ_ONCE(zone->vmemmap_tails[idx]); - if (likely(tail)) - return tail; - - tail = alloc_pages_node(node, GFP_KERNEL | __GFP_ZERO, 0); - if (!tail) - return NULL; - - p = page_to_virt(tail); - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - - if (cmpxchg(&zone->vmemmap_tails[idx], NULL, tail)) { - __free_page(tail); - tail = READ_ONCE(zone->vmemmap_tails[idx]); - } - - return tail; -} - static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio, struct list_head *vmemmap_pages, @@ -541,7 +515,7 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, return ret; nid = folio_nid(folio); - vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio)); + vmemmap_tail = vmemmap_shared_tail_page(h->order, folio_zone(folio)); if (!vmemmap_tail) return -ENOMEM; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index dde4486195ad..53a341fcde74 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -34,27 +34,13 @@ #include "internal.h" -/* - * Allocate a block of memory to be used to back the virtual memory map - * or to back the page tables that are used to create the mapping. - * Uses the main allocators if they are available, else bootmem. - */ - -static void * __ref __earlyonly_bootmem_alloc(int node, - unsigned long size, - unsigned long align, - unsigned long goal) -{ - return memmap_alloc(size, align, goal, node, false); -} - -void * __meminit vmemmap_alloc_block(unsigned long size, int node) +void __ref *vmemmap_alloc_block(unsigned long size, int node) { /* If the main allocator is up use that, fallback to bootmem. */ if (slab_is_available()) { gfp_t gfp_mask = GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_NOWARN; int order = get_order(size); - static bool warned __meminitdata; + static bool warned; struct page *page; page = alloc_pages_node(node, gfp_mask, order); @@ -68,8 +54,7 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node) } return NULL; } else - return __earlyonly_bootmem_alloc(node, size, size, - __pa(MAX_DMA_ADDRESS)); + return memmap_alloc(size, size, __pa(MAX_DMA_ADDRESS), node, false); } static void * __meminit altmap_alloc_block_buf(unsigned long size, @@ -138,8 +123,6 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone); - static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, unsigned long ptpfn) @@ -158,7 +141,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in if (WARN_ON_ONCE(!zone)) return NULL; - page = vmemmap_get_tail(section_order(ms), zone); + page = vmemmap_shared_tail_page(section_order(ms), zone); if (!page) return NULL; ptpfn = page_to_pfn(page); @@ -190,7 +173,7 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in return pte; } -static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) +static void *vmemmap_alloc_block_zero(unsigned long size, int node) { void *p = vmemmap_alloc_block(size, node); @@ -329,32 +312,36 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, } } -static __meminit struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) +struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { - struct page *p, *tail; - unsigned int idx; - int node = zone_to_nid(zone); + void *addr; + struct page *page; + const unsigned int idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - if (WARN_ON_ONCE(order < OPTIMIZABLE_FOLIO_MIN_ORDER)) - return NULL; - if (WARN_ON_ONCE(order > MAX_FOLIO_ORDER)) + if (WARN_ON_ONCE(idx >= ARRAY_SIZE(zone->vmemmap_tails))) return NULL; - idx = order - OPTIMIZABLE_FOLIO_MIN_ORDER; - tail = zone->vmemmap_tails[idx]; - if (tail) - return tail; + page = READ_ONCE(zone->vmemmap_tails[idx]); + if (likely(page)) + return page; - p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) + addr = vmemmap_alloc_block_zero(PAGE_SIZE, zone_to_nid(zone)); + if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail(p + i, NULL, order, zone); - tail = virt_to_page(p); - zone->vmemmap_tails[idx] = tail; + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) + init_compound_tail((struct page *)addr + i, NULL, order, zone); + + page = virt_to_page(addr); + if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { + if (slab_is_available()) + __free_page(page); + else + memblock_free(page_to_virt(page), PAGE_SIZE); + page = READ_ONCE(zone->vmemmap_tails[idx]); + } - return tail; + return page; } void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, From patchwork Wed May 13 13:05:06 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237852 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CK6vPSLf; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20887-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv582N63z1yKH for ; Wed, 13 May 2026 23:12:28 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3Y2jt3z2yft; Wed, 13 May 2026 23:11:05 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::535" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677865; cv=none; b=LPJgCNYx/aJujwM8JI45R1dHhbRiBLiphwJ4Dgj7LIIMPECSlGgtbp/zD1BnXBqAbBsqZjgKeF5/FtgBxMGaV0xb2rHJ5rjPHy6oG95d2oX+oyGDTeg9os0duOZaNUyQ36tIqDMiE8KY1Y6As/H27P3h7cbBgmy1iwxMyRbO2+kMla5SwG/L2KwlVi/VLItPkz/nSF7F6u31AN48XipnwNwW4+CVaOq5GA3O7ckHF4ThoU3FcR8DAJYKXwAT1avzkdpnpXgK52H1VtHXdEDFB6A1dmKlpcZJ4on/v/jA4xLfN1MMio8pFHFB6XiyHh+O3StPdzkn+AP6YhuLMKl08g== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677865; c=relaxed/relaxed; bh=LpwPxpfie30YAMOHUn6jeAG+IgoJLqSekICSQMqQJSE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fM248VcF/OQazsW8wMbQchrjfvjSL3Mwc7o6OnrTs56Ori2Ayd2fDlQrwYzuOOETylrA8xo3BKn8x4cEqLpy0NZLJIzCzpeicxJwqbAsvIk8gZvTqvAZ3XdesYN/hGEWz/1tSmVjdPJeymuZb4E+rgTgxD5AhdHnIld50ERvLhJoJD0qstjKkSJVYWxEqya3JakEKLPNFYQd2CWoiiwleiJHfIqbSeJy24jZMQj/GRCMLW6tcTHamnCRKPwoiAzIMHap9U4FuEHM7G3uJ1+HB7XL9BhUIx5xuc4+w8Qf+azDV2DVnqMrCaoh05yh8f25y4oIrpBZIF+IZttFQiwgcw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CK6vPSLf; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=CK6vPSLf; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::535; helo=mail-pg1-x535.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3W6kS0z2yfN for ; Wed, 13 May 2026 23:11:03 +1000 (AEST) Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-c7b9f54d3deso4627557a12.1 for ; Wed, 13 May 2026 06:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677862; x=1779282662; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LpwPxpfie30YAMOHUn6jeAG+IgoJLqSekICSQMqQJSE=; b=CK6vPSLf3v0D+lCKQkq1gkRRTIWvQG+z+c0CDAyGeeM2Cv16JINR1zyNe81sxJ4Frk aXoFs+pEbEIzwCkjHn45whnS7AbCxf2YoUc62FivDU0O3rY776Yd89lfeUVNh1ZWuY3u LzKTnG1vn5bnfVe8Mq3aub1mTydv3uRhs1HVjVGWtyZ4RTVA8Ro+nGBzKHhH0WtUFMq8 m0TpWNKXelk289cSw/8cg5uJ0noSps7Yt/FILV6MEUDUmQl9YEhr9JgfPW0kLYvtz7S3 GbHcRhw1epRSawpKRjQ8Hp7Hl3O3eaXvdp9FAdGpez+BvcsBQkWT5K3egHHHRUUPMF1f BbTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677862; x=1779282662; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=LpwPxpfie30YAMOHUn6jeAG+IgoJLqSekICSQMqQJSE=; b=LVmx2ex95DQYnRjVPb75coXYKzSQyYNdWt6oMyDLu9dgeKVZFntIRTCWxP+9HTOcNa G8WFPr1YYh4U8piWmtz7HKy5vYRcfuDOemZGzgIlZ2gv3M4LGPJF/caxuT0GweuQBLBd mc6rPGdD8smKfOyLpMgoMjC6WOObMz/P3l7X4Nx76WIvpgwPMzd1iVe3R/P5pS/kDnkF aUQdVq4OkVQL2AwbYaXSf4E09vj4esAZ9d0Fs526rL6YuYKD0b0xsuBWJ7HT/u6p+eqq mZjkJNSBhcSzBC7p6Za4716RRljXXXUUIfiKADQUpLRIvCNvVZqbJMHq1c4ZLKNqKmB9 ksAQ== X-Forwarded-Encrypted: i=1; AFNElJ/nd7KAvZ3bk0SDaLP4LWFToPG+rCKVIwwMVVEvE5MrXS2GbAxuqIAJR1nuWXq0jamjyDiztivqbUhIyf4=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwGO929UOlIrgQM0Cb1zrckxSaQVnVcCOM7B0aK795Z8m09K+ik H/edRMVlh8V1Ym9lgqOmXZxt6WbLa9FHz8TYOr9TZfB/PqXk4hKB+Ybhj798ekWQLXc= X-Gm-Gg: Acq92OEzvKQEFM1vPeK+kAsPpz6mpxXwMjzVONrxr5VnJODVB/WaMWeWPNI+us2FGKh QVQlf6Fp/bHYfHyxm21y3ZS57MGJBfiBJygNmLTBl2j/hY5XenG7k5Tepb+kplbrUIlp9r35oNZ pkRj9pqQ1pXNtB3bo++Vrf22a0Eg6sf72ikWo560JpUUV44gV3Up6WMAnkdLE1zMshNS9Xc2XWj WWZ6mUC3EYi45BSOsCdI2wGGX3qhnvno6mLRZi78SHgf6nxwA3zyQietr3PYmp1aUrLwDgs4f+g pcw3oG0vXdSLdZ86dWPzru/6x5Z2WSghBoJmHaEyDrpSNAcGVYfq+q+bTibgrW2lhoSneJU/z2W hxoyTbfSYesWIrSY1RBKhnM1IUTKI7FYDn8SwPH8s99XkahnI3ffHuj7/lfGtSpJM+3Aynb9hj9 4EssjDQC5EJjGAfMtd9l/DdwuGnCJIOQSq2V0SKnBUbcxCRiof1NYkZY5zqqw= X-Received: by 2002:a17:903:22d0:b0:2b7:abc0:3bd7 with SMTP id d9443c01a7336-2bd2716ecabmr35771225ad.9.1778677861863; Wed, 13 May 2026 06:11:01 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.10.56 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:01 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 38/69] mm/sparse-vmemmap: Introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION Date: Wed, 13 May 2026 21:05:06 +0800 Message-ID: <20260513130542.35604-39-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The generic sparse-vmemmap optimization code is still guarded by CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, even though it is no longer HugeTLB-specific. Introduce CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION to represent the common vmemmap optimization infrastructure. Have HugeTLB and DAX select it, and use it to guard generic optimization code. Signed-off-by: Muchun Song --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 2 +- drivers/dax/Kconfig | 1 + fs/Kconfig | 1 + include/linux/mmzone.h | 33 ++++++++++--------- include/linux/page-flags.h | 5 +-- mm/Kconfig | 4 +++ 6 files changed, 26 insertions(+), 20 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h index bc3e549795c3..5f8424eade2b 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,7 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#undef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 602f9a0839a9..60cb05dce53d 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,6 +8,7 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE + select SPARSEMEM_VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP && SPARSEMEM_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index ccb9dd480523..f6cee1bbb1fc 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,6 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP + select SPARSEMEM_VMEMMAP_OPTIMIZATION config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9b87d798a365..5285d53b0c53 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -102,9 +102,9 @@ * * HVO which is only active if the size of struct page is a power of 2. */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && \ - is_power_of_2(sizeof(struct page)) ? \ +#define MAX_FOLIO_VMEMMAP_ALIGN \ + (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION) && \ + is_power_of_2(sizeof(struct page)) ? \ MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) /* The number of vmemmap pages required by a vmemmap-optimized folio. */ @@ -115,7 +115,8 @@ #define __NR_OPTIMIZABLE_FOLIO_ORDERS (MAX_FOLIO_ORDER - OPTIMIZABLE_FOLIO_MIN_ORDER + 1) #define NR_OPTIMIZABLE_FOLIO_ORDERS \ - (__NR_OPTIMIZABLE_FOLIO_ORDERS > 0 ? __NR_OPTIMIZABLE_FOLIO_ORDERS : 0) + ((__NR_OPTIMIZABLE_FOLIO_ORDERS > 0 && \ + IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) ? __NR_OPTIMIZABLE_FOLIO_ORDERS : 0) static inline bool order_vmemmap_optimizable(unsigned int order) { @@ -2033,7 +2034,7 @@ struct mem_section { */ struct page_ext *page_ext; #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION /* * The order of compound pages in this section. Typically, the section * holds compound pages of this order; a larger compound page will span @@ -2213,7 +2214,19 @@ static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn = (*pfn & PAGE_SECTION_MASK) + (bit * PAGES_PER_SUBSECTION); return true; } +#else +static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) +{ + return 1; +} + +static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn) +{ + return true; +} +#endif +#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION static inline void section_set_order(struct mem_section *section, unsigned int order) { VM_WARN_ON(section->order && order && section->order != order); @@ -2225,16 +2238,6 @@ static inline unsigned int section_order(const struct mem_section *section) return section->order; } #else -static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) -{ - return 1; -} - -static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long *pfn) -{ - return true; -} - static inline void section_set_order(struct mem_section *section, unsigned int order) { } diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 0e03d816e8b9..12665b34586c 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -208,14 +208,11 @@ enum pageflags { static __always_inline bool compound_info_has_mask(void) { /* - * Limit mask usage to HugeTLB vmemmap optimization (HVO) where it - * makes a difference. - * * The approach with mask would work in the wider set of conditions, * but it requires validating that struct pages are naturally aligned * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. */ - if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) return false; return is_power_of_2(sizeof(struct page)); diff --git a/mm/Kconfig b/mm/Kconfig index c26d2d2050d5..ddd10cb4d0a3 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -410,6 +410,10 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. +config SPARSEMEM_VMEMMAP_OPTIMIZATION + bool + depends on SPARSEMEM_VMEMMAP + # # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. From patchwork Wed May 13 13:05:07 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237853 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=O/ME9Nln; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20888-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv5H2rmbz1yKH for ; Wed, 13 May 2026 23:12:35 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3f0N2Sz3dL4; Wed, 13 May 2026 23:11:10 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::629" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677869; cv=none; b=IHH9Q2Qy+ctjTqgcBUTHpU2176SRn3kF5wNAxcmLG+FkyEkpb0sEJBnvt14a9uN3tGV+kZema1RF2AWY9JVFD9DqFNNSFUvZuvVDlMvZvkvTRAtA0ujIsJY2ceSVMkfZtDAJv0IzwUrlr0sg4ALfKWUghNyYKG7p8xnE1uDHPPWYVmjUIYT+d5rZMagdxkMZ8E3Ps4oZCzktvuXGMpUF72DwZmyl9YgNFVp1yb8QAYkj2kyNYzzg2/Sq8QWVrZJQ7IAY3lkXhO8VreBqpeftLOWhm9kzxOhE16a16u70KBjeoio3f/12j+XZyd8P5StBOtexKU5cHp3G3doiDmJGZA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677869; c=relaxed/relaxed; bh=lVy27NFdLuqlZKA5DziPKCwbPsRWePBCMeJ/pOD9y2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iRIh6A/mE2es52Suu3YzUebCZsorO00ZjEwZrxt62e4sBXWQmLw08qFmKvr5h4qmmlqN4N8aMJH3wYCBoq7eEwI3oSfLIgrGCvflSbqm2PHExW6TE7dBNQEi94dqvsXxpUi1KZtXwq6CBFzwjzPZcUjPlhPY21bPWfM3WO0Q6IiLtP1T9t+SQAEL8IjGKzmqfoEqoBjmhBqAGnYrUvgR7NXxG4BO4eVJmiEL323or2UGDVaSgOTvR0DIFuxEW+mMlcJSYFrGbxwd5kiVGWLxU9iOozunc0vaE4SnLV46zYv0OZlA4aGcwmyloBr3JCCHlXoM65kQrL6KzuW21ZCqYA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=O/ME9Nln; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=O/ME9Nln; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3d264gz2ygG for ; Wed, 13 May 2026 23:11:09 +1000 (AEST) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-2b788a98557so50724235ad.2 for ; Wed, 13 May 2026 06:11:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677867; x=1779282667; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lVy27NFdLuqlZKA5DziPKCwbPsRWePBCMeJ/pOD9y2M=; b=O/ME9Nln60w6awItKzhTlD6MJuEB5MaHSJiyTDRUbdIgHw/4WoleFDBrKHBqdzuGMS eLuufurH49Zf5WjpTeqnL5OaQy205hsp64PxdYqk7PTzUtiSWbML+a0cIST3gql3Aesf Jgtl79IOvyw41ZXuEOmgNd9uFG5kKJHhi1Glmz79HyEtI3fDVdc5M+kX8bvsy+dLHJfH Ri/lgiBBWK3TTG3+KtmTt8Z+QEXsUqxgKHT8q3gE9zJ8YHJ1NrwWEZacgWd+65pQFI+3 C+D+FzrKpYf9S7t755T5IaqzefoiratozGngPx8gYH1YTYBdpHkt5E3FFFflX8m8v3ir Hd8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677867; x=1779282667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=lVy27NFdLuqlZKA5DziPKCwbPsRWePBCMeJ/pOD9y2M=; b=MiEG7ZuEn/kb18EUlsY5XjqQynmxcrl6sgHxjB2uscDTYzhy8d1nXXmq/nbz4ZGFPZ aun0MFsNmHgqN5hZ6KF6quDa7QhQndMIIm32AIfuuMMX8oQjYxmBtpH3tF8Vgn8cKXAD 02SuAMIUPXF/oKgL+MbiAOWaoLqXlhSg1HzeELHGTth1X9O7ksTUGneaWQH/IpPMlD5n niwx9EzAIS59FjuZt6pe7csguP1HerJxgM4fQ9ACDhxu1V9srWHlzLcrT3+UrgwoKIDD tCxkzjdM8BrR9SURuh2ze42E3EPRV+YYmRjT4eg2FueKaBB7fNlL3gK2DI5UCf+8+u3V 0rOA== X-Forwarded-Encrypted: i=1; AFNElJ/fTymL0w6HJfcTcKyaSgUA0THUwfWt6DzZEvtKubQXE6cvBLe4CX4FinH+qXtFO0ApVsvqAkZKl6M5yws=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwjfR7IMOMngCZ+glf0BD1CbOqpmSfYTkQq82SuWsOoIjGVtKiF mbfJ+n1DoyE261j5JrAx0SHQYp/cOHhI2bfgVX7VXQ3vFw405bzPJy8a5NMFUk/+hLA= X-Gm-Gg: Acq92OHFZWhyQ1YWVk60dIj7Wy/H6+40A9qfRoQgaiP0GM2IRfuUbAvtZ1+gk2yJoD+ UuNA1JgieIOLPzBzwov4s+YtjIJ2w1EB8p7vIfZfWjMismVxWapq3K7FWDYr7BSwsYlytmOz6pc 4kkoym6UB0m8AnmyFaiAD8zeavEJaXRY54rHHRJwHt7NrIPLBwVXFC3FgbnfbRcDRRR0WKr8oiR sgtQSoyeR8OUIgwBUBisFyHYlt2NqSbadBw4Bp1a0vUKTljG0wXQCCiW1/zEfsgAypr71OcMdrf e4Y9XqHYp0iFEM5xK4fSgdpvVXJ6V2Qpb+Y1vluls26aq51he2sZ8crZet2WoyFM54pQyn9zY5a ZLU3Z0d1qzULhkF+iPErFyB+wYNsQAuBtkXC0rDVQuXEpyBVXPVXqsD2cKUc3EBp6QlUkirzYi7 mbjh7kZR1ZLiial1bou3peW42deIqEmjZvDyf8Jh/yrcEEjC+P+oB6V5RHGRU= X-Received: by 2002:a17:903:8c8:b0:2b4:65ab:57cd with SMTP id d9443c01a7336-2bd275c0c48mr35313305ad.36.1778677867293; Wed, 13 May 2026 06:11:07 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 39/69] mm/sparse-vmemmap: Switch DAX to vmemmap_shared_tail_page() Date: Wed, 13 May 2026 21:05:07 +0800 Message-ID: <20260513130542.35604-40-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org DAX compound vmemmap population still has its own way to find a reusable tail page by walking the previous section's PTEs. Switch it to the common vmemmap_shared_tail_page() helper instead, so DAX uses the same per-zone shared tail page as the other vmemmap optimization users. This removes the PTE walk and lets both the section reuse path and the populate path use the same shared page directly. When the target zone is ZONE_DEVICE, mark the shared tail page entries PG_reserved as well, so they match the initialization requirements for device pages. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 10 +++++++++ mm/memory_hotplug.c | 9 ++++++-- mm/sparse-vmemmap.c | 48 ++++++++++++++---------------------------- 3 files changed, 33 insertions(+), 34 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 5285d53b0c53..7484e7be7b6d 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1693,11 +1693,21 @@ static inline bool zone_is_zone_device(const struct zone *zone) { return zone_idx(zone) == ZONE_DEVICE; } + +static inline struct zone *device_zone(int nid) +{ + return &NODE_DATA(nid)->node_zones[ZONE_DEVICE]; +} #else static inline bool zone_is_zone_device(const struct zone *zone) { return false; } + +static inline struct zone *device_zone(int nid) +{ + return NULL; +} #endif /* diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 462d8dcd636d..9ff830703785 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -551,8 +551,13 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - page_init_poison(pfn_to_page(pfn), - sizeof(struct page) * cur_nr_pages); + /* + * This is a temporary workaround to prevent the shared vmemmap + * page from being overwritten; it will be removed later. + */ + if (!zone_is_zone_device(zone)) + page_init_poison(pfn_to_page(pfn), + sizeof(struct page) * cur_nr_pages); } /* diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 53a341fcde74..0c0b54e94c07 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -329,8 +329,12 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon if (!addr) return NULL; - for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) - init_compound_tail((struct page *)addr + i, NULL, order, zone); + for (int i = 0; i < PAGE_SIZE / sizeof(struct page); i++) { + page = (struct page *)addr + i; + if (zone_is_zone_device(zone)) + __SetPageReserved(page); + init_compound_tail(page, NULL, order, zone); + } page = virt_to_page(addr); if (cmpxchg(&zone->vmemmap_tails[idx], NULL, page) != NULL) { @@ -442,23 +446,6 @@ static bool __meminit reuse_compound_section(unsigned long start_pfn, return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; } -static pte_t * __meminit compound_section_tail_page(unsigned long addr) -{ - pte_t *pte; - - addr -= PAGE_SIZE; - - /* - * Assuming sections are populated sequentially, the previous section's - * page data can be reused. - */ - pte = pte_offset_kernel(pmd_off_k(addr), addr); - if (!pte) - return NULL; - - return pte; -} - static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -467,19 +454,15 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long size, addr; pte_t *pte; int rc; + struct page *page; - if (reuse_compound_section(start_pfn, pgmap)) { - pte = compound_section_tail_page(start); - if (!pte) - return -ENOMEM; + page = vmemmap_shared_tail_page(pgmap->vmemmap_shift, device_zone(node)); + if (!page) + return -ENOMEM; - /* - * Reuse the page that was populated in the prior iteration - * with just tail struct pages. - */ + if (reuse_compound_section(start_pfn, pgmap)) return vmemmap_populate_range(start, end, node, NULL, - pte_pfn(ptep_get(pte))); - } + page_to_pfn(page)); size = min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); for (addr = start; addr < end; addr += size) { @@ -497,12 +480,12 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return -ENOMEM; /* - * Reuse the previous page for the rest of tail pages + * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ next += PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, - pte_pfn(ptep_get(pte))); + page_to_pfn(page)); if (rc) return -ENOMEM; } @@ -828,7 +811,8 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - page_init_poison(memmap, sizeof(struct page) * nr_pages); + if (!vmemmap_can_optimize(altmap, pgmap)) + page_init_poison(memmap, sizeof(struct page) * nr_pages); ms = __nr_to_section(section_nr); __section_mark_present(ms, section_nr); From patchwork Wed May 13 13:05:08 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237854 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HWUaCopO; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20889-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv5Q3XNdz1yKH for ; Wed, 13 May 2026 23:12:42 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3l4cRfz2ygL; Wed, 13 May 2026 23:11:15 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::52a" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677875; cv=none; b=MVQC64cnRT/1Yu0t0BfMiZhg2AlhGlMJeWmRn8wnL4A9RLkwRoMmjeUWZBoSoA+29YMDp5FiYJO4cW2ZaB7TnGMDbehcRcCQp3+zK0NxScJintB8b+ELEJkXb2wnMNhHCW5D0xwQFBPtV6kE7uFggufI0Lb1JoFy5ngMkXdBu/+tyUWIFIrTNnlqAYitfmu3lWXUE26Sv6HHya3eQaNTvK5KK1sdxVBjwYIWsb0o21fihQzBqAm4VnXIu8jSqiEB9iMe+np/JoYCkxK40z7FXOqQkEmpsRv56rAa1yKZa3HRqsHHIpLSV8I5V7unP2nC9q9c54KK9ZwO9sgwFILYxg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677875; c=relaxed/relaxed; bh=zm8LOv4zY1TsfsxyQCLJwzqDtIAwoZz5LOnZLOy8d38=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ShegUKciAsnkWKjrDGcKVK9k2p2VnqwYWyOoFpo2cUxFtHOoSkRd8VyYNTX2moTux2zuuwBjTGX7eKUcvB3jDAbhhzBFQj+Lv5AQAw6AdHs7eiBPRthxA4mXBKqytFbdOyA6TVmsvKlmVjn7HDMqA0qOcCFzu4l+n3kcw8kaaHoEFcN+XXcABp+r7Xk6n/RFLpjcSl/lQypOjCaw/DSWkeUVdX0reOLqDqX7Bt8raVbNSMKbpYiIBaUJse7kkS0PsW42qKIqUvJnNlFA3/ZImJJxjByIEHlTez3TemFYz8xvlcXNobZ8bJK8bpOY4yiWXY173+ZSqir/YXp1KTVm2Q== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HWUaCopO; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::52a; helo=mail-pg1-x52a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=HWUaCopO; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::52a; helo=mail-pg1-x52a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3k5p46z2xvH for ; Wed, 13 May 2026 23:11:14 +1000 (AEST) Received: by mail-pg1-x52a.google.com with SMTP id 41be03b00d2f7-c70c112cb61so5197589a12.0 for ; Wed, 13 May 2026 06:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677873; x=1779282673; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zm8LOv4zY1TsfsxyQCLJwzqDtIAwoZz5LOnZLOy8d38=; b=HWUaCopO8bmuAIYEBMmn4y5jPEhpEPijNdW3hEZEPHF1GNSoO5zDRNk+zf8Me/lmoQ n+bXZXJj5Ad46qPEYIN6SH8MxpQJ/EkSBl7Wh5Drtx2KCH8TJxuTNnFtcYzjCGKNnnWS o7UtYUzZURytee6A3ITR4zxiP6s0KNzXdUXjUQCbeZOMHfS71F8e9qvED6yPDRWkX0Jb 0hm2oyLNEMwmmLEE2EZxxQsBDIFJ1YAflUNQKh7iuzTHUFEJQSiS5G2yGhL9bSRsfGh6 nRJZzF0Ca7PjktEs6H6Rqn4s3poks6d8h/LxBFQDe6YyYFVmipEAtJ/Sz7g+A02gw+ms Y6Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677873; x=1779282673; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=zm8LOv4zY1TsfsxyQCLJwzqDtIAwoZz5LOnZLOy8d38=; b=TTHsYdV8UBnKu254BO6U5UseneoTkzRLwyFfyB5hzU1xvbon0evUD/jn/vG3TL6oxZ 7Jpu12pMZIwcwP/ZvEWD4NI4G4XGVwhYtKl7KfEzVKobAukvSVvT75xnyn6JPQ4NU7Qu 58P68zJ54IRSgtH3TzUuE9d7AIO2LjOyNpNh9fyeFo/B8qSMxEyHhvVc2rs3dMj23wGJ oUBG2wzbu+5/DGj5IbOIZ+2qONR7h+Or77KWSSMSWS2dtbuiaIgU+AWolLAI9X/112kB nbhFJF4SW5N7sqZIww7I+XQg4N0d1t30iyKPm1jI3mm0l+kb1mnHek1C6/nPdNTbWKMJ 7Ylw== X-Forwarded-Encrypted: i=1; AFNElJ+1QZJ6pevDWH04JPSFiRl+Oq2BY6+fetnphIDVEr3+wXtUTWCPgwpsa6w5NjhHQXbiON27chr0mqeXItM=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx8EpYaMydcYd7rXRu46U8e5/EQqdo9QGn1efuA+OHCy43anLgj piu0Ayh7Siy5fUuhrIS2OvAk4e7ZZOLjxpNVeMnT9LU1KGiwWTLsUmsd//cJF2O9ygg= X-Gm-Gg: Acq92OGdihbYlEKVpkfsJv/ad8vcTCFXdU6eiA97ZHCAg6fu56ul4fGuUV6e1jwUBuo aKvUhDDZ8HlCF43LR3rf5YWBmKk5Bk9g05kKeXVIGE1ac4yp7+SbIM4e41zML5rRW4xDwka26yr qAZE6xa7CNB98gnlGAme/wiWivV0APg3sxzdtU/S4JsWfkkV2eSZENlch51Xvy9PvBRQKq5Yqql yg/uf0xwMYIIuaSSw9de1Y5O/tFmp6HL5csEqSCPdgu4x0UGEyYZJar8bkEOm+FgeZ28Mj0uTRW M/ISVQkCWHYDQohsuak5fa8RGwEFQajLpNWjkwf7T1XUPYWioef6iHCFP2poaij2UW1v4KX6Xxb dUNo8+n3jKRzM5B5o2Vqj1j86+kLQGlZqvsfAtl7s/NAtjDVFH0kOZrF302FSZT7MXSWsl+whOE IyPpzu0YfdyaVEWtKjWNYngcDqiYbQ4zkX95u4llL278hjEAphn2n9k/zlTxXMtOC45pSrBw== X-Received: by 2002:a17:903:986:b0:2ba:6ca2:bd9 with SMTP id d9443c01a7336-2bd30640442mr29210365ad.41.1778677872675; Wed, 13 May 2026 06:11:12 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:12 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 40/69] powerpc/mm: Switch DAX to vmemmap_shared_tail_page() Date: Wed, 13 May 2026 21:05:08 +0800 Message-ID: <20260513130542.35604-41-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org powerpc compound vmemmap population still finds a reusable tail page by walking the vmemmap page tables. Switch it to the common vmemmap_shared_tail_page() helper instead, so it can use the shared tail page directly without probing or populating neighboring mappings. This removes the powerpc-specific tail-page lookup and its fallback path and aligns the radix vmemmap optimization path with the generic shared tail-page scheme. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 76 ++---------------------- 1 file changed, 6 insertions(+), 70 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index cf692b2b5f7b..95e65ac8cdea 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1250,59 +1250,6 @@ static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int return pte; } -static pte_t * __meminit vmemmap_compound_tail_page(unsigned long addr, - unsigned long pfn_offset, int node) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - unsigned long map_addr; - - /* the second vmemmap page which we use for duplication */ - map_addr = addr - pfn_offset * sizeof(struct page) + PAGE_SIZE; - pgd = pgd_offset_k(map_addr); - p4d = p4d_offset(pgd, map_addr); - pud = vmemmap_pud_alloc(p4d, node, map_addr); - if (!pud) - return NULL; - pmd = vmemmap_pmd_alloc(pud, node, map_addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte = vmemmap_pte_alloc(pmd, node, map_addr); - if (!pte) - return NULL; - /* - * Check if there exist a mapping to the left - */ - if (pte_none(*pte)) { - /* - * Populate the head page vmemmap page. - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte = radix__vmemmap_populate_address(map_addr - PAGE_SIZE, node, NULL, NULL); - if (!pte) - return NULL; - /* - * Populate the tail pages vmemmap page - */ - pte = radix__vmemmap_pte_populate(pmd, map_addr, node, NULL, NULL); - if (!pte) - return NULL; - vmemmap_verify(pte, node, map_addr, map_addr + PAGE_SIZE); - return pte; - } - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1320,6 +1267,11 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, pud_t *pud; pmd_t *pmd; pte_t *pte; + struct page *tail_page; + + tail_page = vmemmap_shared_tail_page(pgmap->vmemmap_shift, device_zone(node)); + if (!tail_page) + return -ENOMEM; for (addr = start; addr < end; addr = next) { @@ -1352,7 +1304,6 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); unsigned long addr_pfn = page_to_pfn((struct page *)addr); unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); - pte_t *tail_page_pte; /* * if the address is aligned to huge page size it is the @@ -1377,23 +1328,8 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, next = addr + 2 * PAGE_SIZE; continue; } - /* - * get the 2nd mapping details - * Also create it if that doesn't exist - */ - tail_page_pte = vmemmap_compound_tail_page(addr, pfn_offset, node); - if (!tail_page_pte) { - - pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - next = addr + PAGE_SIZE; - continue; - } - pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, pte_page(*tail_page_pte)); + pte = radix__vmemmap_pte_populate(pmd, addr, node, NULL, tail_page); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); From patchwork Wed May 13 13:05:09 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237855 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=GzgxQwqL; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20890-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv5Y6tMlz1yKH for ; Wed, 13 May 2026 23:12:49 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3s1mz4z3dRp; Wed, 13 May 2026 23:11:21 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::529" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677881; cv=none; b=mpPa+/PdOgbkJzwxPtRkzIz//vdnZXebuU+HqE+N5DQ4FtxmJ50fgS8NIbTq+mvJpvPi6QUUIK8IPvD+FSspHYyRkIvkqQlWkUckPh32uMUtwskEzghRfpovqbO5pI85WZHqkAmYfPah76hBR79mlote37stf4JdG7eZEJqVqdN8zozADWV7wRxGZjvpsr6KjSTFB2+Ncuuyu9xJ8mZ6ZWMPo3hfgZR1w7h2KC36ikNDLR1f5f595JJju19f5Q9Uf9nUVdjHRRhyWpTGwcwhlIwog6fswJ6lnzkzm1KqByBeHEboqRiNJcjVwvAsk1r5DdKP0pGZS4eOTvuDgP/Wgg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677881; c=relaxed/relaxed; bh=RjQC+d3yUJ1wS02bTAKTrS34AR4NyzrcQRS2dfL6Y70=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ehp7jL540e7vk5ylZUAf7bBjoN3g1u22mo8K+cKb2Ng1V8RnMwUHkEoT2AiMsCnJ/O9By3wRS5TsdP6dTz8cfx1YqC7CyPG2Bt5xeZ46hVt8WrtbNBEA8rRrETgdYOXmcc31XJsAEiGO0Rt0eh4mu5TTTjKg9WkXRzc+wTQ1TJIr6ZNqhPxSudCyhqQ/hPmxX+9tK3h0B4QJglHG5oiT23dcxihPJcLK2kQTjKSNZTmcrYqTJKM2dhnfCj6IcGPl6CdFsxogXNblqtKrvt2radlUwah1sTxXV8lK6FifehEt5yGCsefjLVeUT+51xRaKc/vNJOlcZFTM/35HyLY6Ag== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=GzgxQwqL; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::529; helo=mail-pg1-x529.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=GzgxQwqL; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::529; helo=mail-pg1-x529.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3r0yYZz2ygX for ; Wed, 13 May 2026 23:11:20 +1000 (AEST) Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-c6dd5b01e14so3051413a12.0 for ; Wed, 13 May 2026 06:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677878; x=1779282678; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RjQC+d3yUJ1wS02bTAKTrS34AR4NyzrcQRS2dfL6Y70=; b=GzgxQwqLPBOm89ESn5h2oSX174v8O1/SsBZYmPCCkBfuG7ESM+9ZJskkeI6R5cOPaA VpNnW8eW/KvlxrNIbly4vXKmVvyYQ2Z+r3ay1LY7L9Sc7KPxSwBZQoFFoQ8gOtUYv4DD aP5fTU48cd2toszBedz8f3J5+Tw/y++smzoCjVLqp2j5Fxkh9VkEDAjysBuBXEqhASwy bCOnxNtUSCGYIhRsxLZZLLKJZnhB67CMs8e1jvY8qoT5RyjGr1Dom2UeTEKY3ZwYOYtI im9KFqElyWrjuG/iGETgjKM67EO6C40BQUdR8cO86iPdduofqynidVwShpkjRd8XP0Gm hIZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677878; x=1779282678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=RjQC+d3yUJ1wS02bTAKTrS34AR4NyzrcQRS2dfL6Y70=; b=Sn+42jrmGoEp3O9haCAND1jFiSD9fZy+5H0FG0zFsjVqTFcrQ9V4DZn6dnwm2CXYy1 CuEVtp/rY1wKnzVN3REipyaY/ajw3WcEcz5PDGGZ6u4d2xcVhXOlpNUeISAvVHvjMtaw 7OOQf/P6fzsveVyl8S3lokfQHRny4jIwMg0IuAl/Skd+uJbG6i2RugwOqam5q2Q6Fo7t k4ExkAdb/azJWBcaQIbiY8iRMEoKTUvotdmwcHk6bUQRlx7jfTuLebW5MGglVUkwFbV1 MtUDEncUginJOvWh9eMUbEBshEUM2Ok/c+XO060GTW0JAivlmMGxNTVLCHkHMra2hh7u 12kA== X-Forwarded-Encrypted: i=1; AFNElJ9OSMSmp0o6atlY0m0mccq7g0SiA9wPAbyjXHxl3VltzA0yehciFbrb4d8wzGkyT9bwEIa1Pws7YoIUMzo=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxAh04DRITLYTGQ8TkEW9UmOEAz0Mtiq0EwWEQQjkYsNBSTFWB3 9xDFw7KgezNdx03eNTIPUG35hyQKjoojnKelaut5ZsGHJy2pEkjHEsGpzI1xMEHLSKw= X-Gm-Gg: Acq92OGbo4O2fmH6t6Z+5BUnzDjYqetQv9WVztlxLGh4AKM/AHNOoPBGoohMKBMvHvL ixNerb7MKhcFN/rSFXK3lTE7vlMZsxrGagDyi8qHtqpGAJm5WwWk4Va/kzGVNnQfjYGubuJBVLY 22PfDKO7aYJh1QZke6rvIjmnt+xIOf6EpnXsPVxoHGE6qo917JMqEAFx7invsx2U6a7P2o2z5un EtJgPkfP9IZ2O5mQqPlU4Gb4OaW+IB/25fdQ1PaSxC72WgJxBjz6V3AOm7EYGZUGvy2CDQEau91 IPfSQn4ERZUMBC58Sx/fDRW5S+Fomdehe8EOkb4Ur46OlDZaOqCVD13yg+KonRaaZgy3UG86gy1 epZYYin5h/iyHm27Vk138Df20M+aCuVfJy/+NyeR8F9/0XG8YO3A4J7QtcwzpCyWQZkFfS3eoY+ KumIA1bABoC6xBWpZzgUWVfCIf9RRk5LkPVf47d0sQHdBEhB9SHTUv2b0i6do= X-Received: by 2002:a17:902:d2c9:b0:2ba:3e50:e3f5 with SMTP id d9443c01a7336-2bd2ff29802mr31509965ad.30.1778677878094; Wed, 13 May 2026 06:11:18 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:17 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 41/69] mm/sparse-vmemmap: Drop the extra tail page from DAX reservation Date: Wed, 13 May 2026 21:05:09 +0800 Message-ID: <20260513130542.35604-42-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org DAX compound vmemmap population still reserves one extra tail vmemmap page after the head page, and only maps the remaining tail pages through the shared tail page. Drop that extra reservation and let the shared tail page cover all tail vmemmap pages after the head page, so DAX follows the same reservation model as HugeTLB. This reduces the reserved vmemmap pages for optimized DAX mappings to OPTIMIZED_FOLIO_VMEMMAP_PAGES and removes the now-unneeded first-tail population from the generic and powerpc paths. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 44 +----------------------- include/linux/mm.h | 2 +- mm/sparse-vmemmap.c | 8 +---- 3 files changed, 3 insertions(+), 51 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 95e65ac8cdea..fb8738016b30 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1217,39 +1217,6 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in return 0; } -static pte_t * __meminit radix__vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - struct page *reuse) -{ - pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - - pgd = pgd_offset_k(addr); - p4d = p4d_offset(pgd, addr); - pud = vmemmap_pud_alloc(p4d, node, addr); - if (!pud) - return NULL; - pmd = vmemmap_pmd_alloc(pud, node, addr); - if (!pmd) - return NULL; - if (pmd_leaf(*pmd)) - /* - * The second page is mapped as a hugepage due to a nearby request. - * Force our mapping to page size without deduplication - */ - return NULL; - pte = vmemmap_pte_alloc(pmd, node, addr); - if (!pte) - return NULL; - radix__vmemmap_pte_populate(pmd, addr, node, NULL, NULL); - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -1316,16 +1283,7 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - /* - * Populate the tail pages vmemmap page - * It can fall in different pmd, hence - * vmemmap_populate_address() - */ - pte = radix__vmemmap_populate_address(addr + PAGE_SIZE, node, NULL, NULL); - if (!pte) - return -ENOMEM; - - next = addr + 2 * PAGE_SIZE; + next = addr + PAGE_SIZE; continue; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 5281f073230c..86d7cecb834e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4898,7 +4898,7 @@ static inline void vmem_altmap_free(struct vmem_altmap *altmap, } #endif -#define VMEMMAP_RESERVE_NR 2 +#define VMEMMAP_RESERVE_NR OPTIMIZED_FOLIO_VMEMMAP_PAGES #ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 0c0b54e94c07..b5c109b8af6f 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -473,17 +473,11 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, if (!pte) return -ENOMEM; - /* Populate the tail pages vmemmap page */ - next = addr + PAGE_SIZE; - pte = vmemmap_populate_address(next, node, NULL, -1); - if (!pte) - return -ENOMEM; - /* * Reuse the shared page for the rest of tail pages * See layout diagram in Documentation/mm/vmemmap_dedup.rst */ - next += PAGE_SIZE; + next = addr + PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, page_to_pfn(page)); if (rc) From patchwork Wed May 13 13:05:10 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237856 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=RcOKXBVF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20891-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv5j24dBz1yKH for ; Wed, 13 May 2026 23:12:57 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv3y38tXz2ygm; Wed, 13 May 2026 23:11:26 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::634" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677886; cv=none; b=FON37lrOhHy328zuVSvY1umHAR6Mvgj5hUuy3rIZbZt2F558prtHY/BTOoaKQhiX4NnDMhdUatmnEP38HUIfI5yGDBQU2AJ09MGbFP9c6vWvfE/nH3C/slP7DIyYNyc5j9RlK4LTL6WcS1ZKyEl3NqGrsMWfOQy8aAYz/ukywmxKhWQuhoUnpkwfe6cxaGPl69sYN/IdyrH1zL2VltP4LvJqVYuaY7AW9O7UmXhIqjjmG78oe3p8MIfAbnhzgNusH29s6ET4ljQAkVe+Kdw0kZbzH6oopXgTBwRBoSdGFVbDxqMyPqXwg+JbwrhRBHkMNElZpRkunfI+QwLCBlHRwA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677886; c=relaxed/relaxed; bh=4mBY0+2gPF5yrifjarXE+EeA1By9caS7ano4A0QDHP8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RVmFvbL+PMv2WgXGCTnjMBYYqT449CvEGgFjHuCrMgPhjw1UjJyyXDRur01SC6v/FzMOEHTmGdK24hcJvItdQYJY4oeP61FwKuK0Q2s45axfcIPA/T0qYhd0MqqYwMQG7938n7+4p0W8LVXEFmrKGj16z5Ujho0hCPAZDLmfzGAkcQy70loUvwzNY7HOJQiwJuckpAvOt9z9FvwUQQ6fUz58q5PS8/tHxt0DXU2qIz+vQjxcmYWK9gmUbekEphgOloA4bkhyPiv0viVy0nWwzBlBdmJvH4BLeRzl7X+vqAWtXuQlGb6SXhehz+Ha0C2zK9NNoW+Ofu6f6Dtmwy/X5g== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=RcOKXBVF; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=RcOKXBVF; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::634; helo=mail-pl1-x634.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv3x3pNLz2ygX for ; Wed, 13 May 2026 23:11:25 +1000 (AEST) Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-2bd266f6fc0so4815665ad.2 for ; Wed, 13 May 2026 06:11:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677884; x=1779282684; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4mBY0+2gPF5yrifjarXE+EeA1By9caS7ano4A0QDHP8=; b=RcOKXBVFexUZUGUhbJgg03mMii/wrHAoU6DZOVkOmiatz6bHyZ6PAsePqmYl9JAX53 wcGCDqZQ1ErVCb/E/o06Tg6k7tU4PNvg9x0FSh4s0AW2te9emrNfxZDvN3/lmB9HkUPR RzMfHwIoh9/NteXmHcGqHZyZXS2r1PLiKvyXHfrZZ8NhIbNwst6NtW1oxvZfN4+3T+d5 Wf0jbsthupOY9f9HTMc44fAS8ERyFN4IhHqu8Hs/Te0qEQ/wPFpXtrjrlABc9qIebVR/ f9fZWRvSfedywox7oB7XaoZYAqgjtY/jCjuKuRylAHHl7GdSeMOYqAoB4/pR2r0SMMIQ QKKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677884; x=1779282684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4mBY0+2gPF5yrifjarXE+EeA1By9caS7ano4A0QDHP8=; b=IcFASov9WsDBA4pFj6A/Tb7vCY3WgXqphR7GyBq2Xg7l0ivAB0eDzPcOhsLqRYPxxr NmxIJd9JOu+ZwkL3IGp9aGUBvZxAGlfnoJvqT20xs38ZzADGi353Kh2/eZvTigf1O27M cKaalmCbNlnSUG8HN+tq8RPwDMbP3fjTDfr7yjxPgNUOjXMpXuH/nFXzM8l2KWUk0rkd yqF6KinRYxkQntNZ8hcAQHYCDpI5A5elIN7Dvx0+EyMOA1JBPx7ZQpRaGjJPfW5lU/aD HaMLbjY1RxqIfI8Ld+OfC98vRtmG9cSBX549f8pw90qGFnzKgO1ldx43eLVphME/UppH fEFQ== X-Forwarded-Encrypted: i=1; AFNElJ/rk61n2yiykCsIEmR7/Mc/4KpBWPEIOvIYSWPGdWSrN25jEV63ZlwkvKyF73HXOVL927kkm+FyC1ckOpg=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxpAci3J9Nt0+KEtUPKDPVfri1H2FPBqPkxFL2qLWJiMsswv+qe s3q+hcP9qGlNQmHrvdfcnpiT2DR4zGMdndMVmlb3xEdHbYiQGsS1PcKw1bOpx4/o1Cw= X-Gm-Gg: Acq92OHIXbIDAhizPqYkN4BLWgsr9W69gjQJ0t2WQEw3F1IMKrKVZWo/ux/A3l6aJVs nwPChDbIZKEE3WfUSiW4uMJtB6Nj+LtE6hb++LN1xfNzr7XnBo7biqwU5MmhB65yTIIoBWKvHUS GGcekCU4esXoMmdVo/t7rlCGK+hTyjZoS+huNGgvHr1DM4RsXu37gowr5M1lPDnnHrEdtyrxEDf oG39jEN+C64QVvnNI7donAVbsCBmRjS8FPxhx8F8rj8n+LlCtJoiYW3taoacTlKXLc+kCoBkvy/ K7xqrbgcUIzdKTZk+b4O4OASfc2SUvx8QWSONLHBPJAeCQz/qh8cVJbUV7J88/cW3N4JpAveMLD JvwvVNqHMIiHV7ISYi+t/aPT2kejS8/Gtq5JHKYNSMp8fCtVhLbHC0ozccfy4n4THvlQjEcNsfP i7Gg0qKwHZVT7YQHm5WnrXXfxD/vMCn226edRA+AU+kuIk8V9zDaZaaBZo2fY= X-Received: by 2002:a17:903:290f:b0:2b2:539b:d2b1 with SMTP id d9443c01a7336-2bd2f6071femr19517135ad.16.1778677883529; Wed, 13 May 2026 06:11:23 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:23 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 42/69] mm/sparse-vmemmap: Switch DAX to section-based vmemmap optimization Date: Wed, 13 May 2026 21:05:10 +0800 Message-ID: <20260513130542.35604-43-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org DAX vmemmap optimization still uses pgmap-specific state to decide whether a section should use the optimized layout. Switch DAX to the compound page order recorded in struct mem_section, so it follows the same section-based optimization state as the rest of sparse-vmemmap. This lets the DAX population, initialization, and teardown paths make their optimization decisions from the section metadata instead of carrying separate pgmap-specific state. This makes DAX vmemmap optimization section-granular. Only section-aligned ranges record a compound page order, so subsection mappings remain unoptimized. The resulting loss of vmemmap savings is negligible. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 5 +++-- mm/memory_hotplug.c | 6 +----- mm/mm_init.c | 13 ++++--------- mm/sparse-vmemmap.c | 24 ++++++++++++++++++------ mm/sparse.c | 2 +- 5 files changed, 27 insertions(+), 23 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index fb8738016b30..f0043c57694e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1235,8 +1235,9 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, pmd_t *pmd; pte_t *pte; struct page *tail_page; + const struct mem_section *ms = __pfn_to_section(start_pfn); - tail_page = vmemmap_shared_tail_page(pgmap->vmemmap_shift, device_zone(node)); + tail_page = vmemmap_shared_tail_page(section_order(ms), device_zone(node)); if (!tail_page) return -ENOMEM; @@ -1268,7 +1269,7 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, next = addr + PAGE_SIZE; continue; } else { - unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); + unsigned long nr_pages = 1UL << section_order(ms); unsigned long addr_pfn = page_to_pfn((struct page *)addr); unsigned long pfn_offset = addr_pfn - ALIGN_DOWN(addr_pfn, nr_pages); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 9ff830703785..c9c69f827efa 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -551,11 +551,7 @@ void remove_pfn_range_from_zone(struct zone *zone, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - /* - * This is a temporary workaround to prevent the shared vmemmap - * page from being overwritten; it will be removed later. - */ - if (!zone_is_zone_device(zone)) + if (!section_vmemmap_optimizable(__pfn_to_section(pfn))) page_init_poison(pfn_to_page(pfn), sizeof(struct page) * cur_nr_pages); } diff --git a/mm/mm_init.c b/mm/mm_init.c index 35c99e5c215c..2b94115e6dd5 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1071,16 +1071,11 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, * of an altmap. See vmemmap_populate_compound_pages(). */ static inline unsigned long compound_nr_pages(unsigned long pfn, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { - /* - * If DAX memory is hot-plugged into an unoccupied subsection - * of an early section, the unoptimized boot memmap is reused. - * See section_activate(). - */ - if (early_section(__pfn_to_section(pfn)) || - !vmemmap_can_optimize(altmap, pgmap)) + const struct mem_section *ms = __pfn_to_section(pfn); + + if (!section_vmemmap_optimizable(ms)) return pgmap_vmemmap_nr(pgmap); return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); @@ -1150,7 +1145,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, altmap, pgmap)); + compound_nr_pages(pfn, pgmap)); } pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index b5c109b8af6f..ad3e5b54abf7 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -455,8 +455,9 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, pte_t *pte; int rc; struct page *page; + const struct mem_section *ms = __pfn_to_section(start_pfn); - page = vmemmap_shared_tail_page(pgmap->vmemmap_shift, device_zone(node)); + page = vmemmap_shared_tail_page(section_order(ms), device_zone(node)); if (!page) return -ENOMEM; @@ -464,7 +465,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, return vmemmap_populate_range(start, end, node, NULL, page_to_pfn(page)); - size = min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); + size = min(end - start, (1UL << section_order(ms)) * sizeof(struct page)); for (addr = start; addr < end; addr += size) { unsigned long next, last = addr + size; @@ -501,7 +502,9 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (vmemmap_can_optimize(altmap, pgmap)) + /* This may occur in sub-section scenarios. */ + if (vmemmap_can_optimize(altmap, pgmap) && + section_vmemmap_optimizable(__pfn_to_section(pfn))) r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r = vmemmap_populate(start, end, nid, altmap); @@ -718,8 +721,10 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, else if (memmap) free_map_bootmem(memmap); - if (empty) + if (empty) { ms->section_mem_map = (unsigned long)NULL; + section_set_order(ms, 0); + } } static struct page * __meminit section_activate(int nid, unsigned long pfn, @@ -729,8 +734,14 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; struct page *memmap; + unsigned int order; int rc; + order = vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + /* All sub-sections within a section must share the same order. */ + if (nr_pages < PAGES_PER_SECTION && section_order(ms) && section_order(ms) != order) + return ERR_PTR(-ENOTSUPP); + if (!ms->usage) { usage = kzalloc(mem_section_usage_size(), GFP_KERNEL); if (!usage) @@ -756,6 +767,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); + section_set_order_range(pfn, nr_pages, order); memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap, pgmap); @@ -801,14 +813,14 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (IS_ERR(memmap)) return PTR_ERR(memmap); + ms = __nr_to_section(section_nr); /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - if (!vmemmap_can_optimize(altmap, pgmap)) + if (!section_vmemmap_optimizable(ms)) page_init_poison(memmap, sizeof(struct page) * nr_pages); - ms = __nr_to_section(section_nr); __section_mark_present(ms, section_nr); /* Align memmap to section boundary in the subsection case */ diff --git a/mm/sparse.c b/mm/sparse.c index 54c38ea08190..6878f8941b4c 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -251,7 +251,7 @@ int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages if (vmemmap_can_optimize(altmap, pgmap)) vmemmap_pages = VMEMMAP_RESERVE_NR; - if (!vmemmap_can_optimize(altmap, pgmap) && !section_vmemmap_optimizable(ms)) + if (!section_vmemmap_optimizable(ms)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); if (order < PFN_SECTION_SHIFT) { From patchwork Wed May 13 13:05:11 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237857 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NUTQMRxK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20892-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv5r5FBLz1yKH for ; Wed, 13 May 2026 23:13:04 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv436r3Rz2ygp; Wed, 13 May 2026 23:11:31 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::636" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677891; cv=none; b=BLvS81Ud4lQ2XLPcuqz1ixNjSBGT/j+d4xIp+0pHCV08WFA6la+v2FU+JutTnGuE6MiA+A9y3ExXgxSPDPh/TqWGTfy1tQXYvjjrvoT/rFapc+otE34gYm8Rtk7gLchwcHILG8n1IO7myFCaWzJKAMgs2l8Vjedsa1QaXgcdqwdNmf3fbR6SNb7bbfHnVxeTweI/4WsfMuJpaNEY3c6s1HNmn7aKK9xeYe68DcZWHwOd/nPsRPXXChjBEMLM8aVqJgrcXf4hby1sLH3tQ8LciClssnk6qI0ZQAevk16XN7vGoi+pgJPtOzAFn9nBq0PtDwg7byVBhpQW8FpTQXoSnw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677891; c=relaxed/relaxed; bh=puCoZmQRDxwpateuYAk/P74OaRyzUWBl0IwVomJ8OUY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TFXg4SMU3zK3ZPHdZv1WK7Oz/BNCK5WWCJ7LNU7WfaBvvitjK+HQHw8LtZ3pMQTmK31lKS6d+3yoHaUog4kc3TKkiyWKBmuo7t94DMi5MW0fq0uIvoy8z+rWkKWfHevZe8smddZZ0uWcTcgN33xRdUssPVp5nZGgLb+Hx4r0w+Z3zd9R1w5Gqei7VqAS4Oyd2Mq2qocJghSY947HUYQQ60zbcVH4jkzKN0OCxWaiUfhAMtW6YbTYM4lWKs9cs6t2G7m7K/jV82N12eajWkgeYen9LSyqRRdFCLdPZ3ifPDHE8wt+PEXz/Gb5AD9+eS62H9ibRT/1yVJH5ikNzEavfQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NUTQMRxK; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=NUTQMRxK; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv430dgqz2ygX for ; Wed, 13 May 2026 23:11:30 +1000 (AEST) Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-2ba4a1a0325so46803635ad.0 for ; Wed, 13 May 2026 06:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677889; x=1779282689; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=puCoZmQRDxwpateuYAk/P74OaRyzUWBl0IwVomJ8OUY=; b=NUTQMRxKjOknaa1JBZ4wGrJNKd/OXIcjph7PJXqEOYNXGicBu8hMCUNPRVt/5LZRwR LOSOlKsz1iU17+FYVysKfz9DOu12u8eGDnXZIJ5Pl6Z5jT/KMKyTez3cjwgWlQmxmpmL pUSqbZtke3Uhfa/0qlYblewPB0z8OIsUSpNR3yV9jwBsBFqMiNadO7O64fEkhSz8KT9f qnL5Ve/VjyWOSlAGTYyJtQiaFC3PTXKVyoS4We0j78shZZxH6E6G2ILuevjz/lX0m9Sz 2W+UCdVkHPvP1pvQJgA5aNzBKgTuj64C2vn+NPY8l11nEr8fy/IGfDzvfjbGkdImU8/1 +opQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677889; x=1779282689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=puCoZmQRDxwpateuYAk/P74OaRyzUWBl0IwVomJ8OUY=; b=UnONgxT6a2SFqfyPf9x1hWh1XOK+MpyMktPHGQznxl7OP3Mbq3j6FmymB4PnuJLZig PrI4W7sWPosXfJNOHWPLXY5sEdPrrqyF8gFmvWI9tY514ZwJDBZZFi8ZUJL/TTImEXcm VYY+wnxvVM1aXki4oazlVv0wq4g8HOt0Oea5QtD0wLE00/J9WnwNUIkOlwIZ1BdTvdwG 0KhB/PgABWrUdAS8UCvjNdeo0Ppw+F/axEZY2GsqJJPDBF3A05xAFJHF9e/omQMwDmk6 BqJ9zfGLx5guYGveLJo+RMcL84fmCSKuejNqkUN5TtLWDBEVdZTolVBA17n11dfCGgHr YkEA== X-Forwarded-Encrypted: i=1; AFNElJ+9TjjY6fFdBtV+Bp+Jw7opXr9jG10WhKvnai+DkmEhQUqDJJEgSXNBO1FSLLTrK9lIGKPoNJFiO9kyNO8=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzrJ5oJ4WXUt9WRKuUPHBLvgJntP5myDWRAB3VxMIv/zsW1rqcC TxqbC7K4uVg4k5Zlett1a4p/3ZLCbYuxPdO9LhDX0GhN1hHFVukMNPiVY6cb6VdQdNo= X-Gm-Gg: Acq92OGGV3cQC2qLdB2BDqmBwRDA1/P8NuYuj+/KvJS0kYzZ/B2ITnu9/i3stMgpzYG emNvg3AzEb+lIiJJDU0IWVTNO/emidiySblL4NceCkkFj33AbzrfH/aqZ8KpTAS7R00hoUTWm4I Fdq56eivd1zys5OEuE0hDVoNQMJDF28oM8uVMpXwB8GCsYgPcBRxw10uuMIisBOTDqu1V87IAVV tzz6n3PijGCFdQKXDOj5C+IvT1U8YKP73ySjP7JaKC5DaoeSpnr5DcCVPoW0GBywrUwBG9PaL9G fCgmDPbFHw7JQDsQCA4OclInx0ceD11Tc6jLm0Uo4y0mJUMHn/rHI5pwrsJOGkLCO+gUdkAteFJ O1K6J3veRV2lBBDzEnog6rH1aqu3U2N1MWbC9kOg2jpOf4uyaCS1PQHO3Qx1J8MwEXDaHEmlY54 QxHH2sLc2PbpOBvbstGB9BDNINsXhXRyEHrqcpD8og0jeeH0iWPdKPpOGxoZU= X-Received: by 2002:a17:902:9a0c:b0:2bc:810b:5c0c with SMTP id d9443c01a7336-2bd2770eddfmr29915325ad.34.1778677888928; Wed, 13 May 2026 06:11:28 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 43/69] mm/sparse-vmemmap: Unify DAX and HugeTLB population paths Date: Wed, 13 May 2026 21:05:11 +0800 Message-ID: <20260513130542.35604-44-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Now that DAX and HugeTLB use the same optimized vmemmap layout, they no longer need separate population flows. Move the shared-tail-page handling into vmemmap_pte_populate() so both users can go through the normal basepage population path. This removes the compound-page-specific population helper and leaves the optimized mapping decisions in one place. At runtime, the optimized users are limited to ZONE_DEVICE memory, so use device_zone() for shared-tail-page allocation instead of relying on pfn_to_zone() before zone spans are available. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 + mm/mm_init.c | 2 +- mm/sparse-vmemmap.c | 183 ++++++----------------- 3 files changed, 50 insertions(+), 138 deletions(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index f0043c57694e..c7f2327681cc 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1121,7 +1121,10 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in pud_t *pud; pmd_t *pmd; pte_t *pte; + unsigned long pfn = page_to_pfn((struct page *)start); + if (section_vmemmap_optimizable(__pfn_to_section(pfn))) + return vmemmap_populate_compound_pages(pfn, start, end, node, NULL); /* * If altmap is present, Make sure we align the start vmemmap addr * to PAGE_SIZE so that we calculate the correct start_pfn in diff --git a/mm/mm_init.c b/mm/mm_init.c index 2b94115e6dd5..9ff118e35641 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1068,7 +1068,7 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, * initialize is a lot smaller that the total amount of struct pages being * mapped. This is a paired / mild layering violation with explicit knowledge * of how the sparse_vmemmap internals handle compound pages in the lack - * of an altmap. See vmemmap_populate_compound_pages(). + * of an altmap. */ static inline unsigned long compound_nr_pages(unsigned long pfn, struct dev_pagemap *pgmap) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index ad3e5b54abf7..4833a2295abb 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -127,49 +127,48 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in struct vmem_altmap *altmap, unsigned long ptpfn) { - pte_t *pte = pte_offset_kernel(pmd, addr); - - if (pte_none(ptep_get(pte))) { - pte_t entry; - - if (vmemmap_page_optimizable((struct page *)addr) && - ptpfn == (unsigned long)-1) { - struct page *page; - unsigned long pfn = page_to_pfn((struct page *)addr); - const struct mem_section *ms = __pfn_to_section(pfn); - struct zone *zone = pfn_to_zone(pfn, node); - - if (WARN_ON_ONCE(!zone)) - return NULL; - page = vmemmap_shared_tail_page(section_order(ms), zone); - if (!page) - return NULL; - ptpfn = page_to_pfn(page); - } + pte_t entry, *pte = pte_offset_kernel(pmd, addr); + struct page *page = (struct page *)addr; + + if (!pte_none(ptep_get(pte))) + return WARN_ON_ONCE(vmemmap_page_optimizable(page)) ? NULL : pte; + + /* See layout diagram in Documentation/mm/vmemmap_dedup.rst. */ + if (vmemmap_page_optimizable(page)) { + struct zone *zone; + unsigned long pfn = page_to_pfn(page); + + /* + * At runtime (slab available), only ZONE_DEVICE pages (DAX) + * trigger vmemmap optimization, so device_zone() suffices. + * Note: pfn_to_zone() cannot be used at runtime because the + * zone span is not set up now. + */ + zone = slab_is_available() ? device_zone(node) : pfn_to_zone(pfn, node); + if (WARN_ON_ONCE(!zone)) + return NULL; + page = vmemmap_shared_tail_page(pfn_to_section_order(pfn), zone); + if (!page) + return NULL; + + /* + * When a PTE entry is freed, a free_pages() call occurs. This + * get_page() pairs with put_page_testzero() on the freeing + * path. This can only occur when slab is available. + */ + if (slab_is_available()) + get_page(page); + ptpfn = page_to_pfn(page); + } else { + void *vaddr = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + + if (!vaddr) + return NULL; + ptpfn = PHYS_PFN(__pa(vaddr)); + } + entry = pfn_pte(ptpfn, PAGE_KERNEL); + set_pte_at(&init_mm, addr, pte, entry); - if (ptpfn == (unsigned long)-1) { - void *p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - - if (!p) - return NULL; - ptpfn = PHYS_PFN(__pa(p)); - } else { - /* - * When a PTE/PMD entry is freed from the init_mm - * there's a free_pages() call to this page allocated - * above. Thus this get_page() is paired with the - * put_page_testzero() on the freeing path. - * This can only called by certain ZONE_DEVICE path, - * and through vmemmap_populate_compound_pages() when - * slab is available. - */ - if (slab_is_available()) - get_page(pfn_to_page(ptpfn)); - } - entry = pfn_pte(ptpfn, PAGE_KERNEL); - set_pte_at(&init_mm, addr, pte, entry); - } else if (WARN_ON_ONCE(vmemmap_page_optimizable((struct page *)addr))) - return NULL; return pte; } @@ -265,30 +264,16 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, return pte; } -static int __meminit vmemmap_populate_range(unsigned long start, - unsigned long end, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn) +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) { - unsigned long addr = start; - pte_t *pte; - - for (; addr < end; addr += PAGE_SIZE) { - pte = vmemmap_populate_address(addr, node, altmap, - ptpfn); - if (!pte) + for (; start < end; start += PAGE_SIZE) + if (!vmemmap_populate_address(start, node, altmap, -1)) return -ENOMEM; - } return 0; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) -{ - return vmemmap_populate_range(start, end, node, altmap, -1); -} - /* * Write protect the mirrored tail page structs for HVO. This will be * called from the hugetlb code when gathering and initializing the @@ -425,94 +410,18 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, return 0; } -#ifndef vmemmap_populate_compound_pages -/* - * For compound pages bigger than section size (e.g. x86 1G compound - * pages with 2M subsection size) fill the rest of sections as tail - * pages. - * - * Note that memremap_pages() resets @nr_range value and will increment - * it after each range successful onlining. Thus the value or @nr_range - * at section memmap populate corresponds to the in-progress range - * being onlined here. - */ -static bool __meminit reuse_compound_section(unsigned long start_pfn, - struct dev_pagemap *pgmap) -{ - unsigned long nr_pages = pgmap_vmemmap_nr(pgmap); - unsigned long offset = start_pfn - - PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); - - return !IS_ALIGNED(offset, nr_pages) && nr_pages > PAGES_PER_SUBSECTION; -} - -static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, - unsigned long start, - unsigned long end, int node, - struct dev_pagemap *pgmap) -{ - unsigned long size, addr; - pte_t *pte; - int rc; - struct page *page; - const struct mem_section *ms = __pfn_to_section(start_pfn); - - page = vmemmap_shared_tail_page(section_order(ms), device_zone(node)); - if (!page) - return -ENOMEM; - - if (reuse_compound_section(start_pfn, pgmap)) - return vmemmap_populate_range(start, end, node, NULL, - page_to_pfn(page)); - - size = min(end - start, (1UL << section_order(ms)) * sizeof(struct page)); - for (addr = start; addr < end; addr += size) { - unsigned long next, last = addr + size; - - /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, -1); - if (!pte) - return -ENOMEM; - - /* - * Reuse the shared page for the rest of tail pages - * See layout diagram in Documentation/mm/vmemmap_dedup.rst - */ - next = addr + PAGE_SIZE; - rc = vmemmap_populate_range(next, last, node, NULL, - page_to_pfn(page)); - if (rc) - return -ENOMEM; - } - - return 0; -} - -#endif - struct page * __meminit __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); - int r; if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - /* This may occur in sub-section scenarios. */ - if (vmemmap_can_optimize(altmap, pgmap) && - section_vmemmap_optimizable(__pfn_to_section(pfn))) - r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); - else - r = vmemmap_populate(start, end, nid, altmap); - - if (r < 0) - return NULL; - - return pfn_to_page(pfn); + return vmemmap_populate(start, end, nid, altmap) ? NULL : (void *)start; } static void subsection_mask_set(unsigned long *map, unsigned long pfn, From patchwork Wed May 13 13:05:12 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237858 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MDdUSoq/; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20893-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv604C7Tz1yKH for ; Wed, 13 May 2026 23:13:12 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv4903SPz2yhv; Wed, 13 May 2026 23:11:37 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677896; cv=none; b=duD8sdsfX+EYQWoYGKXIj9O63lHapv7NVltihjGU3EDQz64pa3XzqBuwet1HM1V5XmgJYkSDkhcHGefs8DZMD0EXqg5et3F5HDRLsoCG/GhtmdcaCBsCYqQrmxeiWI3n2a/P/2Na/LCD1soRNvEVVPBlz/tic6ioDrEf/tS6993g+fQ/LLfE6nMCewWMTdV4mwXi0KlsYokq3TgOCKkuRpiZrdUHytq1YC8oyMSfdVryMgylDiPDhgIDo9fch3pJq9rEXY7LiDEpu0BWNLepx5gJqluFHchwu/TpYnigUyZZT3tvRwsLLhSylUhMd/MFrHcsXeHpv7DxrNj5ib803w== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677896; c=relaxed/relaxed; bh=r3qZ6fKvu/W22OcYYEqHvfVb2RH2sc3ZeigSJrIXA9w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=omzpQnhbToC6fx2t0q/6rKTQLa8lVz5WsYZy2eAaaOgwv/LL7gT+VyuSGbS2JNQ7VFoZNdhV3yCJnaq6KHOrRBXpgNQm9Y6puCr0adYJpl9uI0HL/JEpObmiuJxYZzyuoHgvOAyVUDOjVpOknc+Ano8dtLoZ+tDvqqRcumHqxEI5bAPYzVpojbvRhKy4W+pH3xGikBj2SPpWyEzrCeRyUFxtgeDsLDG2735JBrA2hvqSC6T7h40wwV05inhTjYAtiVmR9RCiFVJ14Hh0IKS8khLWQdPi+bzPEL0Fe9gedz2IyePE0C86+QixT6NKUIKupWM/zAxWiIbK2RbXlf+47w== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MDdUSoq/; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=MDdUSoq/; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv482DCLz2ygX for ; Wed, 13 May 2026 23:11:36 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2bcd730e090so24830705ad.2 for ; Wed, 13 May 2026 06:11:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677894; x=1779282694; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r3qZ6fKvu/W22OcYYEqHvfVb2RH2sc3ZeigSJrIXA9w=; b=MDdUSoq/dOKfrzBTN7cAXXfD3tyFne6UEqLDZQUT+GS8cwOSsXYb/25DGYKzzFDNvp bpMYY8bPVPXGJQFC4TE0csg1tCEoLmpL7tiedZEdR90tIhWGCjUEli0lU0mCNRqdPjNO m9EUZsc4l56hoGXpullOGd14BjifDqiPRYj14JZhNpEpiU+/D9dko9Y4fj+oGVxRvins lLT9PtDkbzxxuqZkNepEFuQzuAfABBGLJhKqc6GatFw90F9e0Lry0O4TXKaLvDJRAGT/ WU7sfRKj6Vc9+Z90f3QB7VsdRCc2qY3pd2ZcJa79mjuBRwm0W5ScrQIbMc71zTaT6kKL S0Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677894; x=1779282694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=r3qZ6fKvu/W22OcYYEqHvfVb2RH2sc3ZeigSJrIXA9w=; b=RfT/miRBpvhiRpFHv7MMeC4oe23oDamKSmOpTMnzqDboEEp09vOXBAe2p3VLN5ISur szjAXKJeB9F7zf3eeUO8VK5E3HcYz+fc6lEeU8HAbac7LeDLx/Xzm170ZZpgJH5d2JNl mB0Pg6+SQFFBD1RT5lQX3DnbTc8XONOldX0F0Iwr5VjDK5cgMGjroF0wfMcQdNbPll9k 3NsaLkfSTgxjcEiQr9//bTEAI0gUOMLUlWPNfrCaqePoRZ8NBGbwGjSO/od0nZpf3dXm wlojUGXIr68zOYuU3oNJNhiBZspTI73LHhqoexoMyBSYsdAyk4QDhy3wUxxL9CPOLVIw seFA== X-Forwarded-Encrypted: i=1; AFNElJ8JsWUMutqv3f/fBkNwhyeQvbRJpVum9NacSlALlNkV0dtENS25mIDqxUiJg3cwmAOkjfufFgCGCaatriE=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx8dR1fKOwERKJw5rRLKbyS8hMx0rNmCFrST76FjxB/8JxO6zb1 pUz0WAWVotmUwRsBrr7bZT/HhEfvQMl2EU6Csc7t+NQ/sDxO6/h+UC/Up3BW5Tg2gdw= X-Gm-Gg: Acq92OGKpdC5plKMchhuIoxymls7SSwnzIzFwjW29rDW5lIzeqHVGiiifRwUBflr+x5 lhhshHMbWOtmHdaCVuL3ryXW4tCgq6hmTv1wEiY84oBBrWCJWFtiCaZM5DjAswWWLkisufn8aT+ 4oJBDk0NcJDVf5M4zWFbWGROmg8c64f+7YsSosetGXeAQgjAaEklh4sCn+LbTSigPgr1ffPRk/s /vzEjJBC4EJ8qqTo+lZsqEBbqyRsmwRsISxdI6XU3v9EtQgEyaRaewFOig//gtPt6Y/7/VrjGo8 KKwQ+s8Yi1wLA5xRQJ01TR4Y50gFWpPdTr8FO3XSWn1Gyv3C1If6/bmG55HrU8iKexz5EiZBUoK 5nn5bff80YPxUIh9JbmTBi+E8C4ydaoZtRLK8an277x/tUQ7YvT0RvDtOeFGgTVOi8W6xH7wsc6 zmjokZyRpetqUdM9WAP2Pj5jT3wmY9rO2FszJeoLFBR/oPn4ULufWt5kkyNRo= X-Received: by 2002:a17:902:8648:b0:2b4:5dff:30fd with SMTP id d9443c01a7336-2bd2f4f734fmr21340805ad.4.1778677894343; Wed, 13 May 2026 06:11:34 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 44/69] mm/sparse-vmemmap: Remove the unused ptpfn argument Date: Wed, 13 May 2026 21:05:12 +0800 Message-ID: <20260513130542.35604-45-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org vmemmap_pte_populate() no longer uses ptpfn as an input. It computes the PFN locally in both cases before building the PTE. Drop the argument and inline the PFN computation at the PTE creation sites. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 4833a2295abb..182d0c7dd1e7 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -124,8 +124,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn) + struct vmem_altmap *altmap) { pte_t entry, *pte = pte_offset_kernel(pmd, addr); struct page *page = (struct page *)addr; @@ -158,15 +157,15 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in */ if (slab_is_available()) get_page(page); - ptpfn = page_to_pfn(page); + + entry = pfn_pte(page_to_pfn(page), PAGE_KERNEL); } else { void *vaddr = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); if (!vaddr) return NULL; - ptpfn = PHYS_PFN(__pa(vaddr)); + entry = pfn_pte(PHYS_PFN(__pa(vaddr)), PAGE_KERNEL); } - entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); return pte; @@ -235,8 +234,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - unsigned long ptpfn) + struct vmem_altmap *altmap) { pgd_t *pgd; p4d_t *p4d; @@ -256,7 +254,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn); + pte = vmemmap_pte_populate(pmd, addr, node, altmap); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -268,7 +266,7 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { for (; start < end; start += PAGE_SIZE) - if (!vmemmap_populate_address(start, node, altmap, -1)) + if (!vmemmap_populate_address(start, node, altmap)) return -ENOMEM; return 0; From patchwork Wed May 13 13:05:13 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237859 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=BRmWBD6s; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20894-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv6859Dwz1yKH for ; Wed, 13 May 2026 23:13:20 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv4G2jHqz2yjN; Wed, 13 May 2026 23:11:42 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::633" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677902; cv=none; b=mewCp0lhMOpqELYUJxZtFdhf03860OYTNhuKXzUbA1S6NF/tYrJ5RAa5DDaiGZ8L0aEcJ5xQBIIZpsP9UsxNyJeiGRf5QZ6L5DayVdveiaGpwyvApzVwCK2M0Y5g6vLR0/4IYgEXWG04VIG22AKnfts4BgE+7KGCRxwu5Do4sWnOLLYwHlaCHcxToOJvC5dc70VAOIQVgZo/RGc2kLlsm/Uu/NEaxrMkeJYjfvy90zPBYZBL1H+ucvbd8fB0hewvnpKwJaZCP6a2WvWh3rIRdJLVWXsFGUQGbwVDenEC33FhL5oa6KTvmkfDYli0rxJZ5aJ0tamhnlgoU31LojD05Q== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677902; c=relaxed/relaxed; bh=cQTyOkGmfxy6eUFBRtUP+rZk74jFDbmzM0hnLmumWs8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Dw65ZrtR1l0orohfcTuGgMvzJNz6b++aJjbG2iB2CnA6TXEqe0+u31156j6viRjXyREdgKdV4P8Ytr4HVerVHIf2dFvyRknCBzB1kOfiwLtH0X3ggP+mx9WLp+KwRiWMGKPAStxDJa9e8trHnxl7jUE7+LthNgerlfCWlT9vkS88Vfp8nTEudb4dVHk4Kenzi7sVqj/ZRog4yM89lzdd4UmVZjBmhteetG33/AzDhlDX3u3J5Dkl+xEIikrdaTIyKXK1+qS+FbQrvCHO0/b/d5LUXUvT0WY6mFqHSns43O1Uw0Hkfs5TXuy72PmBEbiGqFuOpOp51z7nAjP7ppjwnQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=BRmWBD6s; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::633; helo=mail-pl1-x633.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=BRmWBD6s; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::633; helo=mail-pl1-x633.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv4F4m2zz2ygX for ; Wed, 13 May 2026 23:11:41 +1000 (AEST) Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-2bc7b311e77so17545905ad.0 for ; Wed, 13 May 2026 06:11:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677900; x=1779282700; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cQTyOkGmfxy6eUFBRtUP+rZk74jFDbmzM0hnLmumWs8=; b=BRmWBD6sSJw9zSlzjRP+7G17/4HDI6sxDFTN7BkMre2sYNFRtsG7MowSFATVXP42q9 6ZQA/MKaf3FkNKUPyurIQaLYBlBqgli31ViulsVuF/AAGLR9wFsT8trI9AZkaaksW0Q8 HOdGiI7pYPTin/Fq/Iu5adUod1t530XVtE5L+g9UlgDBRVLoRphD/HbPo/QHbIoQSYMS dgAe1nCNUiq0AWgQZaVHGJaN7CQGwl0CjJv3pd/yekGCDlOvQqrJ6VaaVWROyNiDIO81 bA3a/gAIkorI25BGEAunScDpOGlpm3rH3xzNIjUxjMEtG63PVzBOH3s+Zv0FWSIdNEBn zVxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677900; x=1779282700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=cQTyOkGmfxy6eUFBRtUP+rZk74jFDbmzM0hnLmumWs8=; b=TiMmGUFMF9ZKJ5hm5sSL4lRcdoQsJJZjMkLZSyWzrsQ1kV0+Bs1YObbkgACLRgInUh ndlJPWgWLi7lIbgYdCP2N0kWFyha5MpJrWLiyT/bCVwkZYALSwnBtmZBi0oMYqw344In xiCz+fGfarfRn2YC6PnZgz+ZYgsAqyRFc8fOdc5dl1ljaJ6GDHRlOdwGOk48f+brMt0b BUVRqRfK8QWBqeMAGuRb6xQBlsbvyrx8a28rctB1owO7je5/JggHSxm4VV8hS+3u4EQO yNQbizH5LZcvLNLGJI500kPm31aqJOl8CfArnXJINqSEkGlN5Pyg4kYMf7BbM9v4yRgE 0mgQ== X-Forwarded-Encrypted: i=1; AFNElJ9KFsgbrClkXcFunyDty2dxLJhJssGhLaS4nJX+l8/oXKD3LlW31HIeqL1jhdTW8Kcx/uhbcYcegcahyYQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxiUgMctKzZfp4Huc+55q80OTxKWY3NjMn9ukXMcbMHNoBsr4J5 zea7pr4ghVuorPyHW9O8AlNePwGdAoAOQWS9ZkELtZyRApQOl9Z6WNGph1QcAHqFTg4= X-Gm-Gg: Acq92OGfDRNXcXo58Nwe3ZcXl97oa1/XCTKsMoBZlczhu3vxMMFLJlF6KSgAC1Xsl44 AiegQVWov5HfeLUhpKRTqmVg3gRIM+0N2vm+7IsJkGfGMoTfzBqRniu9kIZHybmemr/G2ZaJW3P 6IhCaVF9BGFPcaj7cH5Btn6ARDSyRdYCyrFx00jexW71klruCPm/HjhN0s+QfAJRc2IJLk7Vb4c kE2CeZohljyV6CdlrzDn4QSrBVq3BfgFTdgr9jKWchYzsK1TC0S+TPMqjNHJZb2y3iARfq9ue6U TcXrW7Qcp/imwfJvi6h0PNcZSyIFJqNi7G2HGo9hrcng3jf1zZh4ql7oeqRUxkVQVDMVfIIwxqZ 49bokiuPMtLL5pG0vm13bC6Wufuatpym15s+L2wVSav13NrKCBsV8YsSThGI2kI/jRa68Oyo02F W/bq/qKttNy4YGVd1gGSyUMCgV5eg+NRhiiyemmdLw15mEdRWRjvG7e++OtdY= X-Received: by 2002:a17:902:da2d:b0:2bc:e299:4b3f with SMTP id d9443c01a7336-2bd272a16c6mr34476025ad.10.1778677899685; Wed, 13 May 2026 06:11:39 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:39 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 45/69] powerpc/mm: Make vmemmap_populate_compound_pages() static Date: Wed, 13 May 2026 21:05:13 +0800 Message-ID: <20260513130542.35604-46-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org vmemmap_populate_compound_pages() is no longer used outside radix_pgtable.c. Make it static and drop the unused dev_pagemap and start_pfn argument from its only remaining caller. Signed-off-by: Muchun Song --- arch/powerpc/include/asm/book3s/64/radix.h | 6 ------ arch/powerpc/mm/book3s64/radix_pgtable.c | 14 +++++++------- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index da954e779744..8452a2714cb1 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -356,11 +356,5 @@ int radix__remove_section_mapping(unsigned long start, unsigned long end); #define vmemmap_can_optimize vmemmap_can_optimize bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap); #endif - -#define vmemmap_populate_compound_pages vmemmap_populate_compound_pages -int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, - unsigned long start, - unsigned long end, int node, - struct dev_pagemap *pgmap); #endif /* __ASSEMBLER__ */ #endif diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index c7f2327681cc..18b24bb891b7 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1109,7 +1109,8 @@ static inline pte_t *vmemmap_pte_alloc(pmd_t *pmdp, int node, return pte_offset_kernel(pmdp, address); } - +static int __meminit vmemmap_populate_compound_pages(unsigned long start, + unsigned long end, int node); int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) @@ -1124,7 +1125,7 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in unsigned long pfn = page_to_pfn((struct page *)start); if (section_vmemmap_optimizable(__pfn_to_section(pfn))) - return vmemmap_populate_compound_pages(pfn, start, end, node, NULL); + return vmemmap_populate_compound_pages(start, end, node); /* * If altmap is present, Make sure we align the start vmemmap addr * to PAGE_SIZE so that we calculate the correct start_pfn in @@ -1220,10 +1221,8 @@ int __meminit radix__vmemmap_populate(unsigned long start, unsigned long end, in return 0; } -int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, - unsigned long start, - unsigned long end, int node, - struct dev_pagemap *pgmap) +static int __meminit vmemmap_populate_compound_pages(unsigned long start, + unsigned long end, int node) { /* * we want to map things as base page size mapping so that @@ -1238,8 +1237,9 @@ int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, pmd_t *pmd; pte_t *pte; struct page *tail_page; - const struct mem_section *ms = __pfn_to_section(start_pfn); + const struct mem_section *ms; + ms = __pfn_to_section(page_to_pfn((struct page *)start)); tail_page = vmemmap_shared_tail_page(section_order(ms), device_zone(node)); if (!tail_page) return -ENOMEM; From patchwork Wed May 13 13:05:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237860 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IKXmOgFX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20895-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFv6J0WV9z1yKH for ; Wed, 13 May 2026 23:13:28 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFv4N6g9Zz3dhY; Wed, 13 May 2026 23:11:48 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::631" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677908; cv=none; b=c6IYlIc+RcMgOz3yjhzdv9/XIrsi1yXYBqVT1hW4CsHi/xpfDygnU615WXg0u+zO65WvNzhQc8zG0nCqQgUmwMh2+CfF86tRpYy94GUCWZNbiRZHI5wBJLyQghiyJrVunCQ89+lztWG37gcBZIULOs+D6hHAbvUvjR/0EeGZ88Pc/Msdn4TLxKommKYMDE+6Ayng9ZU4wXfHRBBXqEzUe1B/KryYICJR+fS9X8rxxSh4TtZMEH3a1HtKDHtmOH7vKdEEZIrzzYLAbWKjBZI2P7eiFM/ycUoE4Z8+7CZxM6OaLeZmoWj1q6w1xjyTTkMjb0rowqpzusDJBXC4k+SDqg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778677908; c=relaxed/relaxed; bh=j+OzHiBv3aopKwtOQIdTMG6i3QI+K667v1klhZuAIPI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l8hMaS6riQAcPM7JDCTukLFNYBtDr3GPz4Vrim5KWDD9YY1NhkMqSfM3QUdnDynJGJ4zbB5gDuBoopg1p3r4STbBjahr8FYRTSQI8my7Z6/zsvGhyfTxTlC7laUGJljLu94rvOHB27JRVTY4vmL+TqnMTEZTqOGSJ1aMcxa/tqrQGXQKXF4KfpqSBPkv6z1FSkApv7eSfyXCt4/ITk5AKylBUUzg1RiFLkrYFXguSNHhCyLQbC/IJ/dG4hKL3pL9WdmVD4xyVgyKkyqEgL6hOGIzC+Jrz6qPxESXKkQ7ldJgCPCegSWe3omvLgGWs8ph3aCYKf9QDhRP/eDiE6Sgxw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IKXmOgFX; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IKXmOgFX; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::631; helo=mail-pl1-x631.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFv4N0vhYz2yFQ for ; Wed, 13 May 2026 23:11:48 +1000 (AEST) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-2ba17c8cfacso68353625ad.2 for ; Wed, 13 May 2026 06:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778677906; x=1779282706; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j+OzHiBv3aopKwtOQIdTMG6i3QI+K667v1klhZuAIPI=; b=IKXmOgFXheYqA03qcTBDzUBZ5+w1NcEoqUYzyrLAQvT8h8vB6Z3myzxLoHbHQcYROC puQuZ2h28CjbV5b2DCUS2c99hA4rkAM4dvCZEyBjc8YrV4rn2ukE8dstCaKY91O5rHNe KDsww/qdY7L2bHG8ep8LvmB/DaTyswKynyyaL4WxEiUC7seEit7d0jTbiVWh7u8X5rx/ zLq/mmsytQMDLSqBtaupr2mgki6e511+IiS3kSb/sD6swN7nI1sQtFXmgKGTB0wIsqCh e52jleBhkgfrYTKj68G6lyasMGOMgHmptmnvweFlPArz54ll4vrUEuTC5cRRFk3eFqXA kUCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778677906; x=1779282706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=j+OzHiBv3aopKwtOQIdTMG6i3QI+K667v1klhZuAIPI=; b=Ftvu9n/OnFqGHS5VCQ9Qs8psdCSngKZfJMVt5GuYm2UzQLlt1rClhmtbhCmRxYnOEJ UsGmHJ1FeJjNtbnVbU0Y+2cpvR5Z2ilTw89+uce3T0K8bvXzoqZD5dzOKdCFScRDKrql YK+DJ/o7beUCM6ikCb//1B1JOijBHpQXpOP/Fi+gYYMp6AqSO0gBMRl2HmPwBLDMLJG7 3edxkGzhsdx6JXAYaIx7y+RnCPY6/8iHwvFCc6TjhkLyV6mPHAhgAIf8wE/GZ4CEifHE aCtXLLQbwSrmoQ/BqbUuWBoVtsab8FSBE6uCQnY6aDNXFs09WRrM/t7sLhQtvCZ4qKG7 dEtw== X-Forwarded-Encrypted: i=1; AFNElJ96QNQfn9MqwjhQqhJnpRJlA5kYVuROt+HAqWn4XJzuZJ/Gy06besK/+gXdr/IZCKvFBzTD/I34f+P3NWM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YySyAeglP5BjeR7LkSz92nHJhci2UvQtvdh4aoeIsdH2HIDSWTY lyVxGcUOUQukAi5YC0KB+uqetBosufPnSyL7naeFoOE2S5pu79hW2dM9ABZxvWB15GM= X-Gm-Gg: Acq92OG5miWELn/d2+xx6Uiplc8YwJAS2PEwrjgpdaYrVERxqgFhM7A8z9H9m/PFsEg h2bbUbUvtTeIgN6/v0eP6OvCuF/yuJJKCv6FewFdYi5dQnPxnwl39r/lJbUzMDCajor+ONcNe1U bOkP43/q4v1+29AydBEloQJyCQsEXlJVZUs8AGcNtcTaSRveXc8CdRvtOmaPBtSBcskc2c7veZ+ s3ewjA7AneYk0my2A0UN7sTIAFnSAD50lqmrw21eKagFbtEzoj/lZ0gMI3Y3GNjgOu5YVyJpSz3 FPRjElUwfB3UITD+UmBjC1xzobejP+7KnVJH9boJIaUf2m9B6Jr1SoIYhuXa9DJ1Q6bLYGfu56l MhRnPqb/8A1iJqdxSwhDAxYaNl5KQNStnK3Cz/ZQhzLf19q/bKdakGL7SyeDPlvB4h2PZfrSU9/ PXY/hMBOSc0Xtxq64ICMMpTDTEHgNOSoUB3Xw0ZADCszccpTO5mbWcBs2UoyE= X-Received: by 2002:a17:903:120a:b0:2b9:fb0b:b727 with SMTP id d9443c01a7336-2bd27714d46mr39411895ad.39.1778677905970; Wed, 13 May 2026 06:11:45 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.6]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2baf1e90854sm166641925ad.66.2026.05.13.06.11.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:11:44 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 46/69] mm/sparse-vmemmap: Map shared vmemmap tail pages read-only Date: Wed, 13 May 2026 21:05:14 +0800 Message-ID: <20260513130542.35604-47-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Shared vmemmap tail pages are now installed through vmemmap_pte_populate(). Map those shared pages with PAGE_KERNEL_RO so writes to shared tail vmemmap entries fault immediately instead of silently corrupting shared metadata. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 182d0c7dd1e7..9811c92ad258 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -158,7 +158,8 @@ static pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, in if (slab_is_available()) get_page(page); - entry = pfn_pte(page_to_pfn(page), PAGE_KERNEL); + /* Map shared tail page read-only to catch illegal writes. */ + entry = pfn_pte(page_to_pfn(page), PAGE_KERNEL_RO); } else { void *vaddr = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); From patchwork Wed May 13 13:20:12 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237861 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=jKZoaK8E; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20896-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvH30PR4z1y5L for ; Wed, 13 May 2026 23:21:03 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvH26T6Sz2xn3; Wed, 13 May 2026 23:21:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1035" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678462; cv=none; b=GvOTDSOF6hIlK+CuLNmh5o9pYVNeD+MN+5KHv6Ojrrdr0KaP7npXsn8usOl4Y7jfnbcyL0f9CSijIrHg2LD2LRI3+UyCeKB1japakKnVWMk0CnIrRa7o22lXZMLgDSE8hFjRARAaAX4r+DZm1Nv6OZYJCZP6/ASI9Vip9ToEq37diUpGXMlVe+8M3ZCoWt3z9bLUS5k7+mMiAeP28rD9Ez1w3J6Tnok2o9aYprw3jv/h/BLeRCHiJiR+nzeTa3tV9fWRyxrYfVDbKMqV5oEiiuFcqm3ep4tu103k0bePEHR8yznPBLAIePDSapb/r47ZuLz2eHammPD9yPO1/72dbw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678462; c=relaxed/relaxed; bh=8trDTW/nV8eHiIfDFU7p2YkBuzRrf5VwZ270y/B/pfU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QZu337J7QRsmUTtZBbr9BZk7u80B6y9BbumvDElRUPJMFP3mtHBYXtsZ+awEE+SySLjLciIJfntAlce6qnAwlS+XBzNg2ZNEBvAiHl5/YFCmqQ7d8o9JlxTWDbDs9X5+u40hZqTIXvNpJzT/0LAI4i5agmiIzQpEBtHlU11Oa0BPmWu8HS1036TcXPndQj4yWzKMKD+Wl11cR12kQdSnyxJtLFKc5+TDBdJkz+HvuraaioPuan4YD4h04mhhNNnJ279SEnfQr1MPI/RJ97Y2bYlegqzvF/f1BbC6Q1XofE+7guvfmw4DWZaFaMHTd8xqE47pWIHre3eQ34NGYOH6zg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=jKZoaK8E; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=jKZoaK8E; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvH06n9Wz2xSb for ; Wed, 13 May 2026 23:21:00 +1000 (AEST) Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-366375c43c2so3969728a91.2 for ; Wed, 13 May 2026 06:20:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678458; x=1779283258; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8trDTW/nV8eHiIfDFU7p2YkBuzRrf5VwZ270y/B/pfU=; b=jKZoaK8EgYOkqMuSBqflWkTJ3StI2wXU0J07/OHdJykymRDLP0hbNMSM6NZithCkcn gW9X1vADfkvrCX7Ejxl/6SkvtAU543DEY3h2CH59CDuMrwOvhUZibXFgZAPDkcGuHvf5 Wbvc68LoejfnYBFN33M54RRckHe5fPAYDO0Sc43496SClYHvoRYvd+gUgvKPyFZsuyZl wIrnbLSB5Tbca3JCDI3mDfhjNTJxHR5QvbGYfK9n/l47XczZHl4eeI+rXDE33jblCPTW eEJXXS1N/jC9lyM013UM3/wABNVGaom21Eag6bNXKHEo3kYtX8+etGN0tAySKbrwHk3p EPDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678458; x=1779283258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8trDTW/nV8eHiIfDFU7p2YkBuzRrf5VwZ270y/B/pfU=; b=TKj8tOXGULD27NcUnh63a2xijUQ0GRw4099m6Or24kHtKDdKpt6Bp75BghWlK1MCsl rfkMLUVqDPOedj0SBp/yHciwqMvcrxHElCPEN4j8n9Bg8jt4v1deCUfWPKBl2XuN4sEa GYUS0Bnjg5/Vhe3zBrPLvmtBwt9xqJYOALamH/Fe4JiiMX5dQ+lqHxNWrIv9sSmYmPdL hOjHwoF24OsdMs0JFoafYLTCxEs7HjqGxtk97SV1sjzlscX1DpyfpRl9AY153TAbvLm7 7mwkOjOTEzLeDpihkjuGrcNBZr5GJTGfl02+/u2z3Kz6dZVPldzcKqhVbKH8oDxJ62hL J+Dg== X-Forwarded-Encrypted: i=1; AFNElJ97hYPUcIiCFyuweGwnpRLs9wzvbApYE0g7MRmRXs3ugt5HSaS+/5DmHGxuWWAZSHTZvmYJdEUuIBFQ7DM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzNiwAbFvFzmXlhyKk5v+t5casgBUV3zA0vsKsrNXGRHhLg0HvK 9QEKj3SRa5+aS5brnzuynqMP4et2+V2P/zs/uLSb6SpkhXKJw7alUdedDNp+uQ/f0lU= X-Gm-Gg: Acq92OECOwSA62NyBPhfwiGeVlW7w6ZWeAglMHt2aYajOdSjLIguzTv2Hr9yWUGf1OG SxyIsbQPXZiiM6/NWPj756y8JlUS+aOzgO/GulXo4nyshMI+EulpNxQ4TOwNyVxB8vPnpWhRrLf MSaJUlvdbXxESdph6lZuhyzqtrwVmiMFLZcxVfslzUtMZW38jym+spKC+FSF6KMLpzitKzxoAUq hfQ5+e6l/W32oEms2cwZGyt2I2qoJnjndmutoVLLDl/0XxPl6o+uDmh3etPCbu826b7VCCJYMW+ agof/22p/YF7tQNRpsACZgFHehQt8ASUW9R8V+oaDnJzLb602GQ7ta4EgCmqozbmnoZp7nraddV OeCImRFLekDS9Smle208DOKdirC/bmo65x6NOHL//UTAfCi2JEgBC6h63xO5unKZRBSHY8JR/on +/eDE+YC6j+6/bO+HPa1z4UPyW65lGYsy4EIrWnQ8pUUv9RVKMJsCymUIv9evk X-Received: by 2002:a17:90b:380e:b0:368:b01c:85a0 with SMTP id 98e67ed59e1d1-368f3e680d1mr3685120a91.14.1778678457609; Wed, 13 May 2026 06:20:57 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.20.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:20:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 47/69] powerpc/mm: Map shared vmemmap tail pages read-only Date: Wed, 13 May 2026 21:20:12 +0800 Message-ID: <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513130542.35604-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Shared vmemmap tail pages can also be installed through the powerpc radix vmemmap populate path. Map reused tail pages with PAGE_KERNEL_RO so writes to shared tail vmemmap entries fault immediately instead of silently corrupting shared metadata. Signed-off-by: Muchun Song --- arch/powerpc/mm/book3s64/radix_pgtable.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 18b24bb891b7..4c3d027c823c 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1053,7 +1053,8 @@ static pte_t * __meminit radix__vmemmap_pte_populate(pmd_t *pmdp, unsigned long } VM_BUG_ON(!PAGE_ALIGNED(addr)); - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + entry = pfn_pte(__pa(p) >> PAGE_SHIFT, + reuse ? PAGE_KERNEL_RO : PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); asm volatile("ptesync": : :"memory"); } From patchwork Wed May 13 13:20:13 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237862 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hODH5IjH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20897-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvH80s1Qz1y5L for ; Wed, 13 May 2026 23:21:08 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvH65kFjz2yMK; Wed, 13 May 2026 23:21:06 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1035" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678466; cv=none; b=RD38xHBPe/lM2ZskvERWi3VvnrCYnEuM/6DrGbLjHoimsblP0HALTOPGu+uNZZsWvPgfNS8d4Iv3mzvCaj4WlRe+oDXyPTovqtqrATvUCmrhnVFXwRZu7zsOc6Ia4fVUHiXI3JvGQfg7NeLxEFzWGsRHt4wgbVeYA2RyrlUDsrsCnSVhJy/voX/eJT13y63QlAaQWd91ukE5Bd17HdyCpOAthBKcvjJyD8FLw9GxNPoDVriasLGndHXVietcyfYgUpaSaYjIUiE3o1ZL60/5u4IMYfaCOBS5WE6I3DjDJWVjwOMFgnH0rYtAuv4rSwRFZI+QhuHNDDPyRiaozcpZXA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678466; c=relaxed/relaxed; bh=kmMHI/Oobz3lupMdXI7U03Ar3ucRA9ahppmUoHlMfZ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d7DnWBuJ3csRJ8UAyjdnkT8V6GV51/QFeHl4muW42DWlHEESAkroVkTHLsr2n+t9LBTNEw3trgVPBUxmlxQf5VOSaqS08Zcnw1gLdN+uw3tcrjA9zfs4jSHMibTOb8y/VoN+hjVKhwEalC+uOvg6u0JJvORwJEo3fV12RS0aMxLAkXf4DCpevdatSVieHElXEl04unIpQA7aAnJGUf2vTNDE7Ykkz6yBLmRLc8ad9XtzyaCxPGKe/2Jhmpi5Ym1Hy8CyaqaxyVPZpM31bmhwuLx+KiPh0etJcHXKNLJm76E73vk8GBtv2Ju7FRFdotEy3JT7NXJLtjiC7TZgpAW9pg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hODH5IjH; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=hODH5IjH; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvH60x9Kz2xSb for ; Wed, 13 May 2026 23:21:06 +1000 (AEST) Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-3665b67ed66so3373110a91.1 for ; Wed, 13 May 2026 06:21:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678464; x=1779283264; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kmMHI/Oobz3lupMdXI7U03Ar3ucRA9ahppmUoHlMfZ0=; b=hODH5IjHPgl651hlSW+rmJIWfvVjZu6/mIwGopxvNKjNX4oCWLz0HQle5zZyJHELRX bLWumRVC5n2V9ejXr9lVpyg1RA/+zn7S0xPGcTlMqlMk6/cUfebjiV7ZdT+5XrBRJM+q Mb5FMuzaejmGnpKygc7dadvb8JdRVDCBfJC6HHkUQCIOStp5Y1ZkMTLyDSiQ5OSNW5gS aAKIoYFa92qJFLQZYYRZztLcCILwjBFSdKH1X2sILRq6N+rpm90GlcE/Vc1R9pRtDg3l pSUlj8ysxUCmGK99s9kn+9fTYH1nKLpf9DEqHx6rsUp8UNsKn9wpuTVmHYEOK4ej3/9Q v+Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678464; x=1779283264; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kmMHI/Oobz3lupMdXI7U03Ar3ucRA9ahppmUoHlMfZ0=; b=siIh/5DtZHyWEbXRdtNw1a709eq1zRV00jRzez6xh+xVP/IzSzJsHNer8EUIngq8f8 R/q5co10mlMr+XL79G2RozoNB5Iv6gbYhqCdkpRLWrpW3ilQ2s8D8Ogx7Zjz9urqH0r5 mqExvxhui/T46RGK4uHSUkUCLgtW/uYJk4rPnCkwdHnlEiDIsB4K6HeTOdmZLBu+UPpa xtiQrUZY6Feeoj5jreyAm1ZDAD9kKfpUd7U6APoIUVHRx/1Mkh/9o92KGxJ7zuq46xHF bUJqaUp71EitOSRoxari4Q4gAf2KGak+BD1lYvnbztRcKww9q31ymwtC0Fl42+w8G/Xx r7Fg== X-Forwarded-Encrypted: i=1; AFNElJ8/gNAaCKdXzFyIRndOPem7nQt9cvqYYpFB1hXvBJ1QtdAUcx9YK2g4k+l1FLp8ognl0j3CcrjoNbCFL10=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzD+4Tw7RWHyR40y2kIWJu2KK1ThYdIVbDoxgLixb9TBcLXPnyR cVZsPgJNddTjMx+p6FXL2/vSRdPmFV1vf60LiB1JCfsv9aSHgCP+cltCGNRLrVidzMA= X-Gm-Gg: Acq92OELRNEydRqAZpb+dmJu1kQCzjTr/UI7x7BHv3+ISH2QfnSOTjMqmTbtckEGa32 86CpQCrZ6xqRDkQmvzink4krIVskqer/pE4k4ilLVuShSUNJEqPtSbgn56uKgHDFHTmba9Cxnew lQIOOvbXoCawWyiPUha8kkj7C4XmWLe59YXxWwXl/jSzgqp8H0GL9bX3BJgFauwCExHicO4sWVr JIxAbrt0OmWMhrrOXxpv/UZxTBtj4ZCcQce/HIiPMR2wKfkj4+rqxqBh02ips4nJeG+RDPWdtdd NIbPWK6gm9OpjSfzQNYBLy+Lng0Os3DZM9o8a46NGMa/oITv7K9IM8gev87HM1Ne0DkJb51r2a9 Ttya8nmkW8j7qv0cYR1H2eVND0VPNn+LWpFo+jHIdb99lijYuE/1AC2r5EtpKMV9LTT3wKGC8Wx BzJ55L3XTA2Q0ousISPYVO5ej60VlTv2R8+V9TADjoKURjn2HRmxM89d27aSog X-Received: by 2002:a17:90b:51ca:b0:366:3df3:fe45 with SMTP id 98e67ed59e1d1-368f3e43e84mr3548880a91.19.1778678463950; Wed, 13 May 2026 06:21:03 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.20.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:03 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 48/69] mm/sparse-vmemmap: Inline vmemmap_populate_address() into its caller Date: Wed, 13 May 2026 21:20:13 +0800 Message-ID: <20260513132044.41690-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org vmemmap_populate_address() no longer has any callers that need the returned PTE. Its only remaining user just checks whether the call succeeded. Inline it back into vmemmap_populate_basepages() and return -ENOMEM directly on failure. Signed-off-by: Muchun Song --- mm/sparse-vmemmap.c | 46 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 27 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 9811c92ad258..5d5cd5f73365 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -234,8 +234,8 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) { pgd_t *pgd; p4d_t *p4d; @@ -243,32 +243,24 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd_t *pmd; pte_t *pte; - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return NULL; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return NULL; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return NULL; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) - return NULL; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); - - return pte; -} - -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) -{ - for (; start < end; start += PAGE_SIZE) - if (!vmemmap_populate_address(start, node, altmap)) + for (; start < end; start += PAGE_SIZE) { + pgd = vmemmap_pgd_populate(start, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, start, node); + if (!p4d) return -ENOMEM; + pud = vmemmap_pud_populate(p4d, start, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, start, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, start, node, altmap); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, start, start + PAGE_SIZE); + } return 0; } From patchwork Wed May 13 13:20:14 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237863 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lPkwpkW7; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20898-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHG6FY8z1y5L for ; Wed, 13 May 2026 23:21:14 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHF6cHSz2ySS; Wed, 13 May 2026 23:21:13 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1031" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678473; cv=none; b=kBeclQpbS29HnebVSG0RkFhNWBHFlU/HWLPmOfMeKOTwj/h/C2fQ32Km9/miJ7xkhN7nfHW8G2FHBWucRNjG11B5BIO1aiZKGD7Ew0UozeHOlTR+YR+1FaPqXE7uXUZl2M+wM6QJoAxzSlzAhHRNVuoRmBt5ocSvwZhsxWe7H02VAbZn7lBKj7eqFghceTc6wOtxW1sUhE1JYB8ZB9PjcKHv+lobnM8TmNlfNxc1Mzm8E170xxqYqcymwxFkWbzzqmvl1scqJKXhZ1zbtIuypqnc3bcx1Sm8tsSBBsUIIXF33jcHic0/JfT19JP/nU8yKCHu9s6SHejj7Huud9tXZw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678473; c=relaxed/relaxed; bh=e4zXGt+6aCEAt5utokxBlmTHRiyLkdnOANcW299C8MM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=js1GTKDXLmQLaGVrQSN9DcbhCpflhA/GhKTMuXm54mcJWi3IuDeTMVGDoabyGLlck30GbBBi+4g3bP9L81/SM9pIe9p23lQe9Sm2Rrd2QdLKCuyRk2e1PfeHOf6zWAfDYiEvMVmz0rz/2djYOKY4f2sGEm/o/CUpHHwY77LGB3UUHAtbM7cH8Ixqbb7VW6fIBaWIJfVQQevr363lRyYzZc23ys9yN4MnFYdb7a+MyuSPznDo9mAhuXpHdfUgJ+zQC8FEZNFv8gYZsZeunXt6JV6AnSQTEjVxqpST1z5Di+CFYDilUwDBWUpJ96KpPutm+Fz04GM0CxfHThDn0mYkUQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lPkwpkW7; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lPkwpkW7; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHD5z2Tz2ySN for ; Wed, 13 May 2026 23:21:12 +1000 (AEST) Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-36523acb0c1so4788021a91.0 for ; Wed, 13 May 2026 06:21:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678471; x=1779283271; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e4zXGt+6aCEAt5utokxBlmTHRiyLkdnOANcW299C8MM=; b=lPkwpkW7FBCoqc26tvHY+PCERUJ2WEUgN3T9gKYCglLWppb3MY+ikyCT53rwZJUxMQ ywIqPGBj3ebGv1jdJs+3ENgZTgOFYUG4kqa3yFyoBYYNbDyh9Ojl38dhR5sgDS6y8v5u //vRNfRoXbIFHMmg+Ma2ziqxAxJ9yDRCS98XukOpNCa7XT0zNXncfhn9FT6X6WDw6M+b YJLsRspAT3asri/g2hZHrCReEig+H8ZejrovEq92w/h2tCv7NDIn2VfFIUCsqiFwAeTc baAeGOLk7WLDduVi6y5JXi9ctK2kOnj8uPnTqzGoOQvhgFaJvFJs1MypxPDL3rXLrBXQ aAIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678471; x=1779283271; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=e4zXGt+6aCEAt5utokxBlmTHRiyLkdnOANcW299C8MM=; b=APTx2/yukt6AMVcP7rH/ruPI5NzI5Szca5zPZgpD/CPcSgNeps4mXUz1k7eMero7GP xKHp5ZtmgsoHHUs7Tv1W9y4RYZ3Wsxx+4Muevd+RLZvs1O/3LLlgqJkwSRRuYZBPL30w LHR0uwBYnP/4NbvmiDUszqgOY49xmC8RUXAF3yFPDcFD3hNDNYSEXydFGQCZadpUJo+7 INb+72eQBbvwdTy4Qfci7TDEI3aAEWeKHkr4ADeZXTlK9RaI8bc7sCl6F+C0lXPNCO5k vzmVwvH2s+OnIRf8X29j6pp1dgVCI2zjIqnTuqm+5s+KQPpnLc59DFxwncUpx4u4wn9L 2egw== X-Forwarded-Encrypted: i=1; AFNElJ/Z7iMvm+gA2pgCeQN6a7jJFRcFw+RHjf/EPoreo4yacEdTt8jYqWFf7zE66ZYGqUgqDoHCPM+eQ6xAGLQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yx9WEKM8SWBE/k+ZnnjpyhL8aEm4k8dDLqwomNoB+JJANXwRN2K SohXZxVjVXpPA6C9D1sN+cgS0wz4kb85SVYH6/xACcttDTBB07JBapGQ9POS2aAOXVM= X-Gm-Gg: Acq92OEjcpaAMM6ksiWJiTYkj+M6JubCw7CpHd0q3YwHvupKLMnG6Kqzt4OZgu6mBsS raf9BLfItf2K4tTgV6m14ZwEZnnaxTSp/mXF4POxGybnfSDZSZemJfIBvC0iaf0o+Dgwz6Hrehr eYh4CVhHcezQryBPOmSi44Pe4Oms/TW0zVOf0QwxNuosXaeQUkP9U5f1lwT3abDTa8R5jBPpyD4 jncaV+EpuC64aXc2q41YEs9X/Q63RvnO+FaEQ/NL5Ikm3SPWc/WHZrJC6WnFgmMlEv4m7FdyUFz WVqIryg5cfqz9rYe9+jMSmjAV/xdrfqxStHf1TraTMG02ifYYpD7IpOwayXLkVccsZHA9rq7HIb eQO05XuX4FRwmCJwqkSK/yzUoQE+7Dqe7KbP6+ekiybliQFN+V804cgH3itYztpi5Gdp/WoZAiX Y+OVJKdhX52RUZZyG+79BmMxugBwquYW1CkRrJ5tGqxgQzqL+OPKyMwKxnSjsc X-Received: by 2002:a17:90a:d610:b0:35d:a3b4:2f0d with SMTP id 98e67ed59e1d1-368f3bd73f7mr3546310a91.6.1778678470505; Wed, 13 May 2026 06:21:10 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:10 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 49/69] mm/hugetlb_vmemmap: Remove vmemmap_wrprotect_hvo() Date: Wed, 13 May 2026 21:20:14 +0800 Message-ID: <20260513132044.41690-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Shared vmemmap tail pages are now mapped read-only when their PTEs are installed, so HugeTLB bootmem optimization no longer needs a separate write-protect pass afterwards. Remove vmemmap_wrprotect_hvo() and the bootmem-specific HugeTLB wrapper, and let bootmem folios use the normal hugetlb_vmemmap_optimize_folios() path. Signed-off-by: Muchun Song --- include/linux/mm.h | 2 -- mm/hugetlb.c | 2 +- mm/hugetlb_vmemmap.c | 45 +++++++++----------------------------------- mm/hugetlb_vmemmap.h | 6 ------ mm/sparse-vmemmap.c | 23 ---------------------- 5 files changed, 10 insertions(+), 68 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 86d7cecb834e..5e38c9a16a0a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4863,8 +4863,6 @@ int vmemmap_populate_hugepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); -void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, - unsigned long headsize); void vmemmap_populate_print_last(void); struct page *vmemmap_shared_tail_page(unsigned int order, struct zone *zone); #ifdef CONFIG_MEMORY_HOTPLUG diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 74770c1648fc..54ef7d12c585 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3202,7 +3202,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, struct folio *folio, *tmp_f; /* Send list for bulk vmemmap optimization processing */ - hugetlb_vmemmap_optimize_bootmem_folios(h, folio_list); + hugetlb_vmemmap_optimize_folios(h, folio_list); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { if (!folio_test_hugetlb_vmemmap_optimized(folio)) { diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d24143dd6051..fce772e95adc 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -595,31 +595,22 @@ static int hugetlb_vmemmap_split_folio(const struct hstate *h, struct folio *fol return vmemmap_remap_split(vmemmap_start, vmemmap_end); } -static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, - struct list_head *folio_list, - bool boot) +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; - int nr_to_optimize; + unsigned long nr_to_optimize = 0; LIST_HEAD(vmemmap_pages); unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH; - nr_to_optimize = 0; list_for_each_entry(folio, folio_list, lru) { int ret; - unsigned long spfn, epfn; - - if (boot && folio_test_hugetlb_vmemmap_optimized(folio)) { - /* - * Already optimized by pre-HVO, just map the - * mirrored tail page structs RO. - */ - spfn = (unsigned long)&folio->page; - epfn = spfn + hugetlb_vmemmap_size(h); - vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), - OPTIMIZED_FOLIO_VMEMMAP_SIZE); + + /* + * Bootmem gigantic folios may already be marked optimized when + * their vmemmap layout was prepared earlier, so skip them here. + */ + if (folio_test_hugetlb_vmemmap_optimized(folio)) continue; - } nr_to_optimize++; @@ -636,14 +627,7 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, } if (!nr_to_optimize) - /* - * All pre-HVO folios, nothing left to do. It's ok if - * there is a mix of pre-HVO and not yet HVO-ed folios - * here, as __hugetlb_vmemmap_optimize_folio() will - * skip any folios that already have the optimized flag - * set, see vmemmap_should_optimize_folio(). - */ - goto out; + return; flush_tlb_all(); @@ -668,21 +652,10 @@ static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, } } -out: flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } -void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) -{ - __hugetlb_vmemmap_optimize_folios(h, folio_list, false); -} - -void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list) -{ - __hugetlb_vmemmap_optimize_folios(h, folio_list, true); -} - void __init hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m) { struct hstate *h = m->hstate; diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 0d8c88997066..2b0a85e09602 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -17,7 +17,6 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *non_hvo_folios); void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); -void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) @@ -59,11 +58,6 @@ static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list { } -static inline void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, - struct list_head *folio_list) -{ -} - static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 5d5cd5f73365..ce1cf5cdf613 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -265,29 +265,6 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, return 0; } -/* - * Write protect the mirrored tail page structs for HVO. This will be - * called from the hugetlb code when gathering and initializing the - * memblock allocated gigantic pages. The write protect can't be - * done earlier, since it can't be guaranteed that the reserved - * page structures will not be written to during initialization, - * even if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. - * - * The PTEs are known to exist, and nothing else should be touching - * these pages. The caller is responsible for any TLB flushing. - */ -void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, - int node, unsigned long headsize) -{ - unsigned long maddr; - pte_t *pte; - - for (maddr = addr + headsize; maddr < end; maddr += PAGE_SIZE) { - pte = virt_to_kpte(maddr); - ptep_set_wrprotect(&init_mm, maddr, pte); - } -} - struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { void *addr; From patchwork Wed May 13 13:20:15 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237864 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lbaf+pId; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20899-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHP22TGz1y5L for ; Wed, 13 May 2026 23:21:21 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHM2TwMz2yjv; Wed, 13 May 2026 23:21:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1031" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678479; cv=none; b=mwukz4mgYGzmnMkPBXhw6gyfiLQUz1CXo8LKYuBfztJWjQquyvB4f04ckLaIP58DudOSjEt5hT5/oFLVHJpe5T/xWJWUg91M+z27KuGs3QjjOORzo6IOc+nHirNJSkFU/SgOpzDIR3nZ8ToNz5EsffQT4RcnITZsVsKT9jbmNDbOO4mHjdIEBwVTEDN59KUlsgLCyGzd8EfXZ5tTF4MHV6GkaVWEwcW/7d2f0uhGvt3eRsOGHbUTkkoOoxEc1JaLKJ+kcWUFWC0zP37cEFm8bxzinw4v7KVMU9k/nimNP83NleaIMuHMhQFQRB5rQCt/IaGANCyQdo4Tqbq6uYHHyA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678479; c=relaxed/relaxed; bh=pQsJbUbh8y0BvTtiLUFG2twbapHy6ttd2Doxr7AiD7w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l+q4Ik6EYncQyS+foz3ybVWNwrGhIXuttWVnx+BWU0AKR5iAQDtcZ8aqd1vTTnIl6zEIQ/OYIM8rtfsIeLy6LwAtorpQXx/8Imus1aHIU4wqXJ+mLSzl7JKRoeG4mMt1MSGg1XgJrtEITHhKg/7yUi6dyi3K4xJrm+KfHNdm5fmVVNovxRATvJRe5ndkMimyB9/lNtng/jNiYTqoTqmsGqNNWspZgzAGux9TThcQsvOCyVOaGLCfZGs/Qsgo1tiXtFtx9ECmtuGDD5qgGPQPT3MmWO4Q0CE8Evy7wirVXWHAR16rYc+Of0kkrX6aoii0xxF2Wfz1KOtDxESL7r5lUA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lbaf+pId; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lbaf+pId; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHL3x6cz2ySN for ; Wed, 13 May 2026 23:21:18 +1000 (AEST) Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-3664df32e91so5492509a91.3 for ; Wed, 13 May 2026 06:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678477; x=1779283277; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pQsJbUbh8y0BvTtiLUFG2twbapHy6ttd2Doxr7AiD7w=; b=lbaf+pId0HED9hsFmW0uqu6rId8VkSI3ke+JFgj6hKCReCeBCYtQ/rUQNs4sazVFo2 QLVUYQVXZ9EM1UCF7gbDUuDqTbQbRUSRDYDnBqqGapsuR2ZbcMwp1Xuvm3S4LDQRuByq Tny486IG6iPGGKV0D7AxTJ1zCPjzAyz0JiOByNJgKduuo8p2iJ0Fv8KgcwuKT5U0krlp oLsVcgMoRiGtK2jtqruv3jOFFdbDMJeCLX9a18hcVCe9Yz1BwJwpwkMYgiBSaXzhIEyz zXbaB05oq/PJH5sjPd0cHzjb0IC5AHy9+sh+DTsxNVoAEuprICqsvJ++vHj2EnxdKFBG E1+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678477; x=1779283277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=pQsJbUbh8y0BvTtiLUFG2twbapHy6ttd2Doxr7AiD7w=; b=lC3l2pAn8Wf48lDcIWQzEkqqv2UZvxouZZZpqlwca8SZHHwTtURVVnPpAQuYKe4RN7 0Ug1uYUUsQjz2aYy4iddp4h+KrE2aeJie54gcQT9ddpdB4ZoZQPXCIpTD8ofQDsgL0uB iS7vjffQyY9gSNeOng8jtISyFHYqgK2FExtj17zZ3mrn/41JjsN2/gAOLGMTOXvo+8RZ P+SOfVhl5eiX3XmocxWpQbbIY5PVErTZi+lqWAdsiP9kUjHwAKtr/DgcNtEOcrFZoxZd rYWx9nBPkzYg2YD+4XEiELzFBKavEr0Guxe6I7lCJAYC4dTR64121GnH66yqch1uvf6M kLyQ== X-Forwarded-Encrypted: i=1; AFNElJ/8l/VSPakiLyiivp/0Meiwq5qC2xbdKHc408o3l0887xE/K2VdAkxEWqEiM2E5OaX1BDpu8alE73mKiLM=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwDZ7WRcPt+j1lLJFtWYVOzidW8GS87Ru9eQkZsauYOcPtfwYSd i4by0oupkkgHQBYALIqGqa/h8qDa1Z7otXvl/uSaOd0wmbq3naVa1gMeIsHGNS42q5s= X-Gm-Gg: Acq92OGkwX5iz9jc3b8/W8LfXmaZxTQY4rLiqjd2HmVVKK1wzr+Tjhpq5g9+1I7/cfw cdidmcvgxndkNKlaVAsSOKLQVTO8NzSh82C7QVY7tJJQry3VMBs5HtE/cTfJCQv9UGZ+X6U1f27 94IGce7rCnz0NhoTad69inBPZV7pRbq2ST45Nw7PLfi+zPeudEfLZLV3813gBhE2FsOQ6JTnJUm NQ4d7+wFgaeYt0tRJJ2YiZCVkxMJgsZkw7TIn6S2oBcu4W/H3HaOInyJmN8M3YCNw97DcyC3TZy JdQW4hRLH9AMFRbRvqx16GMwJSSu67Ytxej3RhMv0t4xMbdYDR90UFVSZXL0knGxJGYb3+zg76c Bvwwbxe1eoHw+kL23wVs0nU6YIZwTQnoAi0gUniVf9voJ+6wIQmgtmYMktKvPduxxiFHNkxiozr 4nzxgjQQvZhu4rTJbaY0+V4rZL+i+198rIP4TUyyFNzfQPVDv4P2XmKwuDY+eu X-Received: by 2002:a17:90b:3f86:b0:366:33a6:9921 with SMTP id 98e67ed59e1d1-368f3a909b4mr3549334a91.4.1778678476572; Wed, 13 May 2026 06:21:16 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:16 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 50/69] mm/sparse: Simplify section_nr_vmemmap_pages() Date: Wed, 13 May 2026 21:20:15 +0800 Message-ID: <20260513132044.41690-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org section_nr_vmemmap_pages() no longer needs altmap- or pgmap-specific state to decide whether a section uses the optimized vmemmap layout. Now that the optimization state is recorded in struct mem_section, use section_vmemmap_optimizable() and section_order() directly and drop the redundant arguments from the helper and its callers. Signed-off-by: Muchun Song --- mm/internal.h | 3 +-- mm/sparse-vmemmap.c | 7 +++---- mm/sparse.c | 19 ++++++------------- 3 files changed, 10 insertions(+), 19 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 18276cd15622..06022074ebcb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -997,8 +997,7 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); +int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); #else static inline void sparse_memblocks_present(void) {} static inline void sparse_init(void) {} diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index ce1cf5cdf613..793fd4ce1393 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -468,7 +468,7 @@ static struct page * __meminit populate_section_memmap(unsigned long pfn, struct page *page = __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); - memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); return page; } @@ -479,7 +479,7 @@ static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); - memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages, altmap, pgmap)); + memmap_pages_add(-section_nr_vmemmap_pages(pfn, nr_pages)); vmemmap_free(start, end, altmap); } @@ -489,8 +489,7 @@ static void free_map_bootmem(struct page *memmap) unsigned long end = (unsigned long)(memmap + PAGES_PER_SECTION); unsigned long pfn = page_to_pfn(memmap); - memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(-section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); vmemmap_free(start, end, NULL); } diff --git a/mm/sparse.c b/mm/sparse.c index 6878f8941b4c..3390cb82f114 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -237,32 +237,26 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) +int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages) { - const struct mem_section *ms = __pfn_to_section(pfn); - const unsigned int order = pgmap ? pgmap->vmemmap_shift : section_order(ms); + const unsigned int order = pfn_to_section_order(pfn); const unsigned long pages_per_compound = 1UL << order; - unsigned int vmemmap_pages = OPTIMIZED_FOLIO_VMEMMAP_PAGES; VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - if (vmemmap_can_optimize(altmap, pgmap)) - vmemmap_pages = VMEMMAP_RESERVE_NR; - - if (!section_vmemmap_optimizable(ms)) + if (!order_vmemmap_optimizable(order)) return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return vmemmap_pages * nr_pages / pages_per_compound; + return OPTIMIZED_FOLIO_VMEMMAP_PAGES * nr_pages / pages_per_compound; } VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); if (IS_ALIGNED(pfn, pages_per_compound)) - return vmemmap_pages; + return OPTIMIZED_FOLIO_VMEMMAP_PAGES; return 0; } @@ -294,8 +288,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, nid, NULL, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); - memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION, - NULL, NULL)); + memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, SECTION_IS_EARLY); usage = (void *)usage + mem_section_usage_size(); From patchwork Wed May 13 13:20:16 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237865 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=h0pyfIJd; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20900-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHX1xTlz1y5L for ; Wed, 13 May 2026 23:21:28 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHT5NLBz2yV6; Wed, 13 May 2026 23:21:25 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678485; cv=none; b=WAFHdCcCiYsyGonf/n+aH9hBEQWJUghiZDUKU79TKLz1DP6eY+en+OUsI/boPFnzE0MSjPUicmDlaanHENSAmmNdMGHRIucIDa3KM06KOYc5A6xEKVYYVdq0TJJAr+ULSA01GkxpkO6SbY8j7TfN8x5b/xXw3EzN5/AyLzx2kNKeyXCm6OpmOAGACYGvNqyQXaGgD9xhpg7453fAN3K7vV77sIQd6QcWpD3lhON90vkvqf46sBMdPel1AgPJv3Ss0k2WwiXQqG407TjrjI8+Zbo3eu2M5tJomCbx/ec0WxHGEMpEBN618aegSpbV9kS9bGdn+K9bVxU9vRduVvX8WA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678485; c=relaxed/relaxed; bh=awvhCSRpADYIDd9yqdYnymlqYSzgggMn8w5U+iOMJHM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oGNpBK2xM0gfo+4+nflEC6f/FT0VaACiETJwCw9kh+/gvR0vHnR2p8rZ+zM90P3CviAl8m01AQGswElca1p5wpH1ZK1FId5eWyuYrNtOlZup++utMzRj3DqVpIbXQy+qI8Ih0K9AheD2jk/YsfUlQ11oZF032wn07Qacc+A6YH0VgCBp82Y49EodwQafEfXBLJaHGcMigOiZtVH7PdVSdBOgGzksp5wkq3QgkPTlMpVJYRBgDrM9HYAXXTghjXaHXY7Kn34l+iZcPGiIQWuCJdOq5M9TOLEdKxmjVmeNhb8MWsKO1iWdsL/L0YZFJm3pQpNci5QHJZU7fL7QEby01Q== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=h0pyfIJd; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102c; helo=mail-pj1-x102c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=h0pyfIJd; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102c; helo=mail-pj1-x102c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHT0J4Qz2yTP for ; Wed, 13 May 2026 23:21:24 +1000 (AEST) Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-367cbac9cb1so4526775a91.3 for ; Wed, 13 May 2026 06:21:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678483; x=1779283283; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=awvhCSRpADYIDd9yqdYnymlqYSzgggMn8w5U+iOMJHM=; b=h0pyfIJds6c8G9jgC1OMMyPewngPErHeiORA3EZy1SH3xBvrsMta4d9Cit/F/Z8BTf Ru7vfcqFWGfrjHYyapvcVcinIeDs23XsZg8EaK9OVv3O/90KoJAVPRTSCmAtgUrwuByU TRYjciAs49fH5Qtqw2paKPtO4zAsRN1teEyfmb2+fg0CgZ0iRxFIqJmx9w9eW8qioFwH eiXje7+jjNrN4Wi2lvY1ibFcr0Hr6vkBdKrxyfgt1N7Yu5TNA1yn8I/r4Rz3YzDRa7fp 6YwyzL0VOGC8D1MtJWuKo1xqj2BJg98WmZbg4oky2OS5Y2POrYXOq71+OzsXonQwKZ4p 3S2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678483; x=1779283283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=awvhCSRpADYIDd9yqdYnymlqYSzgggMn8w5U+iOMJHM=; b=aGHY9KsLqKnIbxJ/KNaXWmJUmxCK/HDMHH6K40BL3ZWDdKCZxEDQo31SIlXTssvCKE 9jH7+x+HRIOJAuScjaMvJ60rQ44/J5qSbPcAOMtlmlC3/3Vo01co5wV5fzoJEX8oP+ya djTroUjX8y+OAJ/XwHvHj3VPcWDYtMQ3Ppu0JXrJlHhwD4bPfsIfMu+1sEZuC8GLnDjq Jhmaje70C0/4JxwZvML1+HT896hOby2qtPaD4sit+96d9nNaZ3yXoVqUxbhuUYos595V X0GiVmEP01FPhYluynrNJ0Db/aSR/HUby/7pKcjXsgJqeCkSZJt9LGKvQDQR313rYpWF d7nA== X-Forwarded-Encrypted: i=1; AFNElJ/eyokiH2ZfEuRhdy71IKnt0z3ZrWMSzBSUqtWHSJ3pK74eeRbzMx2OVAUInc/MvvX3Pfb8Y88FbjbxwIc=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yyni8tl6OfGCSHA/F0F4abRtQPup8nesfRtYMUtj8WGlmMdTphU NlzY72EXFm1oU/VeZwj2uEhhkdxhgXtwh0b/sw0yg9/mCinPR7uYTOhSp1uRfn0dy9Q= X-Gm-Gg: Acq92OFv28BhF17RLXjhEaaj000dMBUzWqDpM5gAvn8ZdRPkjVYzBC8PK1QxeG+9/EZ WWdTdCRC3Tl07L7cDk1gJt2vWfidzDcrcy4YWf22hpXuLoV8LizxDNqf7vlF72L2fFD/z5+r4s0 nGkslZlNVDOMve4JkkNDkzREEpAelgUPVBWb2RlaaKVq6bERgxQ8sf9YEhEoBAv2vgcB+evhyLy IB9xSpi0BUlSlsbwo1l5iG4K8+DCfb41tIBVpUUlUjSLjg7oGFu6vUOnW93K4nMLo79Hu4zRz6p xRoXZmVXV/itPmM4j2EsuEuSH6bpNr5YW/0rtYFQzgd5Ali2Qp+VTlxuccXB0eYmXW2E4AzVAWc bG2jEshb4jy9BfSXdAN+coEVB/zAXUo2p1zD4zirzEkA9gNfKigojxc2PQ+oDWy8CjjMcF1P5JN EvancO/AYyKrdws5TG7VYa0ZI7PNtY73wDVxh61NqCec8ne9DAbj9g6DlVE7Iv X-Received: by 2002:a17:90b:2ccb:b0:367:db13:aca6 with SMTP id 98e67ed59e1d1-368f7a26e42mr3016818a91.26.1778678482924; Wed, 13 May 2026 06:21:22 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:22 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 51/69] mm/sparse-vmemmap: Introduce vmemmap_nr_struct_pages() Date: Wed, 13 May 2026 21:20:16 +0800 Message-ID: <20260513132044.41690-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org compound_nr_pages() exposes sparse vmemmap optimization details to the core memory initialization code. Introduce vmemmap_nr_struct_pages() to report how many struct pages are actually allocated and need initialization for an optimized vmemmap mapping. This gives memmap_init_zone_device() the information it needs without depending on sparse-vmemmap internals. With this helper in place, drop compound_nr_pages() and keep the vmemmap-specific logic inside sparse-vmemmap code. Signed-off-by: Muchun Song --- mm/internal.h | 11 ++++++++++- mm/mm_init.c | 21 +-------------------- mm/sparse.c | 13 ++++++------- 3 files changed, 17 insertions(+), 28 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 06022074ebcb..9597a703bc73 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -997,7 +997,16 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } -int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages); +int vmemmap_nr_struct_pages(unsigned long pfn, unsigned long nr_pages); + +static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages) +{ + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); + VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); + + return DIV_ROUND_UP(vmemmap_nr_struct_pages(pfn, nr_pages) * sizeof(struct page), + PAGE_SIZE); +} #else static inline void sparse_memblocks_present(void) {} static inline void sparse_init(void) {} diff --git a/mm/mm_init.c b/mm/mm_init.c index 9ff118e35641..4ea39392993b 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1062,25 +1062,6 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } -/* - * With compound page geometry and when struct pages are stored in ram most - * tail pages are reused. Consequently, the amount of unique struct pages to - * initialize is a lot smaller that the total amount of struct pages being - * mapped. This is a paired / mild layering violation with explicit knowledge - * of how the sparse_vmemmap internals handle compound pages in the lack - * of an altmap. - */ -static inline unsigned long compound_nr_pages(unsigned long pfn, - struct dev_pagemap *pgmap) -{ - const struct mem_section *ms = __pfn_to_section(pfn); - - if (!section_vmemmap_optimizable(ms)) - return pgmap_vmemmap_nr(pgmap); - - return VMEMMAP_RESERVE_NR * (PAGE_SIZE / sizeof(struct page)); -} - static void __ref memmap_init_compound(struct page *head, unsigned long head_pfn, unsigned long zone_idx, int nid, @@ -1145,7 +1126,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(pfn, pgmap)); + vmemmap_nr_struct_pages(pfn, pfns_per_compound)); } pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); diff --git a/mm/sparse.c b/mm/sparse.c index 3390cb82f114..f314b9babc4a 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -237,26 +237,25 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -int __meminit section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages) +int __meminit vmemmap_nr_struct_pages(unsigned long pfn, unsigned long nr_pages) { const unsigned int order = pfn_to_section_order(pfn); const unsigned long pages_per_compound = 1UL << order; - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); - VM_WARN_ON_ONCE(nr_pages > PAGES_PER_SECTION); - if (!order_vmemmap_optimizable(order)) - return DIV_ROUND_UP(nr_pages * sizeof(struct page), PAGE_SIZE); + return nr_pages; if (order < PFN_SECTION_SHIFT) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return OPTIMIZED_FOLIO_VMEMMAP_PAGES * nr_pages / pages_per_compound; + return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES * nr_pages / pages_per_compound; } VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + /* Ensure the requested range does not cross a compound page boundary. */ + VM_WARN_ON_ONCE((pfn % pages_per_compound) + nr_pages > pages_per_compound); if (IS_ALIGNED(pfn, pages_per_compound)) - return OPTIMIZED_FOLIO_VMEMMAP_PAGES; + return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES; return 0; } From patchwork Wed May 13 13:20:17 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237866 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nix2Yt6+; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20901-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHf6Z0Hz1y5L for ; Wed, 13 May 2026 23:21:34 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHb6fNfz2ypw; Wed, 13 May 2026 23:21:31 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102b" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678491; cv=none; b=MYHDiKR6+mZCPqPkncabnb8CYy5GyZ0So2Wcx0FZiM4kct6+WsH9pzPjP1pqrgienRbp7wouAYdH4p499+G/XZHvg5Dpeyt5bkfdm6+RwPtsiYh/9jl62m/ceOY6zgglutYasn9hZPKbDdKL+3ggGJe6NjZJIc1v1IYqPU73WHYytB0RaNCY/b8hPUAID//EudBHJibL4mqdDAmTm9dS8T5yY7cVaOF9A/osSobcDwmoSvVe9ATqg1n6/EHBiF6nmjNv2fIQgQbFVYM1SYBoLpsTVJDSe9+P96JnQtvCkOkAU4o4pDWN9ph9nsKIJY0z9yzO/Cu5s2u7XyIH8apffA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678491; c=relaxed/relaxed; bh=3PXFCj0/ECPhuZwScmmiKfqjkOg4F11/Y6QM4pfwtxE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ivRN70u34QO0NQ4TWf/s1xDdw9l5nO7HeBZSDoL7a8t0E2mMA3gXj0JlFGlasJaF17IPx04fGDmMbaWGP1hcQfTXtzU2yFHEQJuSoB2PEepNl1U2HE0daoFCA599oxSQRHmrdoPsenT5+hMbI2aO1sD7kA3Y9UX7YfdwL4+nMy7ERxkvCnoE2dPKQHAhtEu4w+0pXp29zvP1S/wrqTl2uOvSp9HTKn3dMssYegxNkx3W3hWU7IfoTaxYvjaTg+k7yj8Fq9DQctZwYChFZNtjtNdOrBXHdZ6zfX6lTa3d21LR/yCFppVkml/jRFG4DvHSmwxPir1MIu5cgNgj3vaDjA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nix2Yt6+; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102b; helo=mail-pj1-x102b.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nix2Yt6+; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102b; helo=mail-pj1-x102b.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHb0t4vz2yTP for ; Wed, 13 May 2026 23:21:31 +1000 (AEST) Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-369002b26f4so351270a91.3 for ; Wed, 13 May 2026 06:21:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678489; x=1779283289; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3PXFCj0/ECPhuZwScmmiKfqjkOg4F11/Y6QM4pfwtxE=; b=Nix2Yt6+r244dpZbqj0jQQ0+srKOwql8+ENS9MCALGmEJhzQwsUSE94MMlThTdj4tE KeFWGvbeC1HtC1EeB3dOgrwmQZ/8JrvJa7dTjzWl1ZFXc268jPCbga6h+hFQOng12pr3 K1nFQ/50rPxoXVlPS55Ir785Z0d4zmAoGl3zILvnpbd6Yw9ealEMT63QyvP4H0td6T/s X9U71TkNJ5dIAI0NfNWT5PDAtbP8vaSt7gVRNd1ph3ZyneP/bsN/MvHoDmjI3S8nWDdw /Z6u2VWJKGQHJAR5gZkJPsnfqtlIO30Q4rD3TuklXW+2AKGjCXLyt8RwnlS2pzKN/Mea E7mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678489; x=1779283289; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3PXFCj0/ECPhuZwScmmiKfqjkOg4F11/Y6QM4pfwtxE=; b=sCH2sb0ePedzKAxkp9B9dcwJQ0KQGMtnHDbZxJWBkNC52uo5NsNytrxphn/rSPX2qW 4ryjppDtdahTP4P3xgpS8wrg9bTM99qyQoRGpt0eknhfFcKhDyVJ7v3++96jI9VOD5fr qNENqOf4fSW0C8aiMfbhCuKUdyqmSVoDeaQSujt7VCgPZQtCsn9YVhzWedi8Mr+7m4Zi 6gLhMShsNjW+/gBCoKO1OUIvaJjs87WCoew0pWP8oatvpsUJ8AVK7IX/KYYKgKU2POnq 0a2BUN/vh/dcscCm1rsjME7aUVLv09DR2o7axWL1RKOdevYAmVxRLLXHIa1T3gMHhIIR H66g== X-Forwarded-Encrypted: i=1; AFNElJ+tQYsw2pGKHOzQzKtZh+TGQKu6dNmVL1v8iUZtOrXnI5frbAeUtO3eAnFXg7x00CpId2pl1f1ZAYbbC+Q=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yy2uKxW0RFCuyD5tkYj2yQly2Ch3QIIGni+yYVNdtAE4O9ZSEL4 r0tRlyizL4FTIJZ5HyYcrizKJnBc1J/8rYhdEKwNrnkEMVNJvdEGvwqnN7J0na9zVeU= X-Gm-Gg: Acq92OE6qQWkry5EXlw/iHko8vCa339KjLkVJcAYC+4A7VSih+anyGhFar3DuG40yJN qCkXljogfBLfk5TUZBjJz6kZVwBLez7hZP7gK/GCQjWAlSsUGrArm+BiSKaHyAvtthzXH16hYGx R0GD+Y/GykOPPjP3bMFzorUeaEvW3qHWlFg3XwNiDQOOtMXtypNI2zVKacEZvpM4kFafJvGLODj i49C5ytBSF9lW9BlA13oOhk+jWNicP79u74Czjguug5gBteKOl9B6Sme6e8xOv9uUNybGmtF/5K pUxM9xya81X7qMU/jXCTebgYclLEerl5sPRFyCJIibleFmrpU2aTvMW3u15HF3ScMXmuMxnPOCq wxIwoqRXOV+ACilt/GlawKw7SBBcp2hby/SqctsNPHpCY0hLe/2zI9ky/erk+DreLNcaKacPWTa QfPlL+j937AaPTQoK8vLs9b4gHhobpWONk/7sQ23jXkYCQihXzFjw5D6zy2JME X-Received: by 2002:a17:90b:1b0d:b0:35f:b5df:453 with SMTP id 98e67ed59e1d1-368f79cd1dfmr2775567a91.22.1778678489114; Wed, 13 May 2026 06:21:29 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:28 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 52/69] powerpc/mm: Drop powerpc vmemmap_can_optimize() Date: Wed, 13 May 2026 21:20:17 +0800 Message-ID: <20260513132044.41690-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org PowerPC no longer needs an architecture-specific vmemmap_can_optimize() override for DAX vmemmap optimization. Whether the optimized mapping can be used is now decided in the architecture-specific vmemmap_populate() path. When PowerPC has to fall back, such as on Hash MMU, it can simply clear the section order there and disable the optimization for that section. Drop the radix-specific vmemmap_can_optimize() override and rely on the generic checks instead. Signed-off-by: Muchun Song --- arch/powerpc/include/asm/book3s/64/radix.h | 5 ----- arch/powerpc/mm/book3s64/radix_pgtable.c | 10 ---------- arch/powerpc/mm/init_64.c | 1 + 3 files changed, 1 insertion(+), 15 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index 8452a2714cb1..df67209b0c5b 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -351,10 +351,5 @@ int radix__create_section_mapping(unsigned long start, unsigned long end, int nid, pgprot_t prot); int radix__remove_section_mapping(unsigned long start, unsigned long end); #endif /* CONFIG_MEMORY_HOTPLUG */ - -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP -#define vmemmap_can_optimize vmemmap_can_optimize -bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap); -#endif #endif /* __ASSEMBLER__ */ #endif diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 4c3d027c823c..2f8783b3f678 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -977,16 +977,6 @@ int __meminit radix__vmemmap_create_mapping(unsigned long start, return 0; } -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP -bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) -{ - if (radix_enabled()) - return __vmemmap_can_optimize(altmap, pgmap); - - return false; -} -#endif - int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, unsigned long addr, unsigned long next) { diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index b6f3ae03ca9e..8e18ed427fdd 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -283,6 +283,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, return radix__vmemmap_populate(start, end, node, altmap); #endif + section_set_order(__pfn_to_section(page_to_pfn((struct page *)start)), 0); return __vmemmap_populate(start, end, node, altmap); } From patchwork Wed May 13 13:20:18 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IyUnCMMP; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20902-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHn5qz1z1y5L for ; Wed, 13 May 2026 23:21:41 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHk4G3pz2yr9; Wed, 13 May 2026 23:21:38 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1030" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678498; cv=none; b=ElnVKG+nj6FEiAbAiu3/G6vyT7Dn91Pb77T1on46jDxj5YyUA+GVMu5qg0asCSwYXjtN1zL9a5T7+xm4zO9d+UpvAGdLOAsHphO0ZjuX0diu14vRye5IeaotwAht0yWpFe94x1WIZYbhc2u2YJSUXvTXKITInM5xMVB8x24wS+4HeXVVj8gHCbJh/dehPEOCiPZts1H0tF0bZ3sVlTFHeFDmWPT50n2eCAb5sAArCVugaOeb3DdRjG9AktkckwrAKGlAY2FXVt3m7z3NPwxD0QJZWfPuVYrbNKRVmwP/kzccnCtlSL9Gp9lrl3xEQzYgKyOXoBKF/SUQoV4qrSxiAQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678498; c=relaxed/relaxed; bh=Wk2DSdjqKxbCAHCUFPSIsYyI95cGv7pmmX0gweVd3ds=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DUKr7B6hXr661HsPtB22P0GLVHmmmITj3m4xjSP6kzo6AyeJt1rxcjZkehIuYehIOFYYoM44Un8dlzjV7MBMT4lz670ELYVodb47hLPP8VmXPwUyh2Q6ocVOiQPix57yINWzJpxmA65rPxP5Uptdsf/4AVqE05RRnXhvOnt/ntq1X4EVrDvH/o0NgweBF2wmjaAA29xhqeQaYH0xfWlu+HSO9SRtvGdKXImW2yBVW2y2Pm58wOlVRWmBIArwMVllRqJqqBaPt8jak/qFHSKJ0EmLCRimZ14qOUfOPpKe+Mupi5czlfyfRraas2vljmKQnQiSPSACBCLgXCTUTydoHw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IyUnCMMP; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IyUnCMMP; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHj4THtz2yTP for ; Wed, 13 May 2026 23:21:37 +1000 (AEST) Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-3660ab73adbso4490488a91.1 for ; Wed, 13 May 2026 06:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678496; x=1779283296; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wk2DSdjqKxbCAHCUFPSIsYyI95cGv7pmmX0gweVd3ds=; b=IyUnCMMPxaTNsnMMaoR9shgHpEcKw90Ppj2lh7JzN0VkceB/umKkBT7c1jOkMBhQVo t1Wtt5U67xzFcuycdNHeWjGz6jX4CPXHF2KHeslLSOU4QOWwzguRd2YnuEuIl1l0ve50 VEUav7CF6XGdL3D5Gru/eHjkqAvE54Hqiv4Y7QAtV/WsmN/n+JOCdA+z9OVgtPmAv4tK 2+31qVnQVqybSiFx5c0fdF/CHmSd8Cv4YKf2i23TtYJbBnoVrx6Z4cCAEJiG4jcLMU4m tygvP7brevElbB0TN06OopdH8eiELodCJS4zSAfJYAGuOvQuouOPlnvR4QbRjTFclzo+ Bwcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678496; x=1779283296; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Wk2DSdjqKxbCAHCUFPSIsYyI95cGv7pmmX0gweVd3ds=; b=OJnnII6SnBJSFfF+oo5+A1P+GViD+GuNIKEeRa3pEIM8LGSVojf+ym2bst1syOESYN IQ03RLgelYMz9ay3ggOcnYq2jG51KfhbiH4rt7ZR/AFu15w4ojiAN1l1k6+HgJQwGYoh igmxrbHP34RiHnGoP/RU+/vnEFt42HcSqoluXJXCVFaqp7ke0zwOKm8N5QwzIwYDPD0e OAgB8vUHrEaiwyVJdISFo9vxKab+tKpH0NALdrHGiIQYHsmpGskxHrWpUTTdqILvJmZ7 4ZZvYh7fmF51NF8oJ9OpqjyPADWJ2PsQ/03n6fWJppySlcCUvjpVKJll0oav7UiNSq3f Y2Og== X-Forwarded-Encrypted: i=1; AFNElJ/73p52qim1SCb2vWd6gDpnjqeGltmMNeZ75cRApZCUkLp544HWb4mVfWuQ/Vw9iWki52uEobxO6ptRfh0=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyzBol9QvwcMDJjaKWzjzJdoXM89fXid+QbC7xD2f4kGvaZTRu1 8VBWfR7ECJzCuwA06MbJPiudE5ANsCcB1KINgmOvj2vMFAv+e4JVhoE1AWXWgmIYU5A= X-Gm-Gg: Acq92OG5EtH6zV/HBYF85R16s3b6rOqBdpEKYgDZzrX96Xd+D8Qj5+K5qrsYykUETvo swzNUJknJxhc4ey5c/wqKEyvU1UCjgg8BTBOHUy3tdzFZ9x9bFkSrueIr4S0WzNWjJGZ0HnbsPN l1dcfrU7V4LB60RVBq8+89ue0BXT8G1ahjzVU24Jc2uBISCfyE7Hi17ltftXXzPErXiP4FVIOne eFAxUR8hXhaSbGs4ZtNxROGO2EkwoQY+Gv616+xijErwq9/AykpxeSGpleWW/fy2yypGyW9XPYh mPPmb5chcVB7VOydtehEXkZsuN9/YeiwK2nCF20GHfPauWpUtNOwZyZ+3IlvVtU1E6rzEi6iYyi OUguX56SptAdvLbtYCOLdltODcQex85TQo0HA5zi6HEFcqEAhed4HVX7NtOMbjbs05poULfUbb9 bgX5ap0LPR710NZ6raNl2wnSZbFdMZB/pNpk1BcadTrZEKYFzAEUOUTma0twEf X-Received: by 2002:a17:90b:2550:b0:368:fd8b:7da9 with SMTP id 98e67ed59e1d1-368fd8b8195mr2143454a91.0.1778678495451; Wed, 13 May 2026 06:21:35 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.29 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:34 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 53/69] mm/sparse-vmemmap: Drop vmemmap_can_optimize() Date: Wed, 13 May 2026 21:20:18 +0800 Message-ID: <20260513132044.41690-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org vmemmap_can_optimize() no longer needs to gate section activation. section_activate() can use pgmap->vmemmap_shift directly to record the requested section order and leave support checks to the vmemmap population path. That keeps the policy local to the code that actually instantiates the mapping, instead of requiring callers to pre-filter unsupported cases. In particular, altmap-backed memmap allocation cannot support HVO, so __populate_section_memmap() clears any inherited optimized section order for full-section adds and rejects subsection re-adds. Unsupported optimized mappings are therefore rejected where the vmemmap backing is set up, and callers no longer have to care about that restriction. With that handling in place, vmemmap_can_optimize() becomes redundant and can be removed. Signed-off-by: Muchun Song --- include/linux/mm.h | 34 ---------------------------------- mm/sparse-vmemmap.c | 14 +++++++++++++- 2 files changed, 13 insertions(+), 35 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 5e38c9a16a0a..5f45de90972d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4896,40 +4896,6 @@ static inline void vmem_altmap_free(struct vmem_altmap *altmap, } #endif -#define VMEMMAP_RESERVE_NR OPTIMIZED_FOLIO_VMEMMAP_PAGES -#ifdef CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP -static inline bool __vmemmap_can_optimize(struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) -{ - unsigned long nr_pages; - unsigned long nr_vmemmap_pages; - - if (!pgmap || !is_power_of_2(sizeof(struct page))) - return false; - - nr_pages = pgmap_vmemmap_nr(pgmap); - nr_vmemmap_pages = ((nr_pages * sizeof(struct page)) >> PAGE_SHIFT); - /* - * For vmemmap optimization with DAX we need minimum 2 vmemmap - * pages. See layout diagram in Documentation/mm/vmemmap_dedup.rst - */ - return !altmap && (nr_vmemmap_pages > VMEMMAP_RESERVE_NR); -} -/* - * If we don't have an architecture override, use the generic rule - */ -#ifndef vmemmap_can_optimize -#define vmemmap_can_optimize __vmemmap_can_optimize -#endif - -#else -static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) -{ - return false; -} -#endif - enum mf_flags { MF_COUNT_INCREASED = 1 << 0, MF_ACTION_REQUIRED = 1 << 1, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 793fd4ce1393..549be01d90f8 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -384,11 +384,23 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + struct mem_section *ms = __pfn_to_section(pfn); if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; + /* HVO is not supported now when memmap pages are backed by an altmap. */ + if (altmap && section_vmemmap_optimizable(ms)) { + /* + * A subsection re-add can inherit order left by a partial + * remove after full add. + */ + if (nr_pages < PAGES_PER_SECTION) + return NULL; + section_set_order(ms, 0); + } + return vmemmap_populate(start, end, nid, altmap) ? NULL : (void *)start; } @@ -613,7 +625,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, unsigned int order; int rc; - order = vmemmap_can_optimize(altmap, pgmap) ? pgmap->vmemmap_shift : 0; + order = pgmap ? pgmap->vmemmap_shift : 0; /* All sub-sections within a section must share the same order. */ if (nr_pages < PAGES_PER_SECTION && section_order(ms) && section_order(ms) != order) return ERR_PTR(-ENOTSUPP); From patchwork Wed May 13 13:20:19 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237868 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=V+43+HF+; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20903-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvHx01Xfz1y5L for ; Wed, 13 May 2026 23:21:48 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHs1tLcz2ysb; Wed, 13 May 2026 23:21:45 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1030" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678505; cv=none; b=PKjryebJBcoNdoksrSTlrCG1BDxfXCIzQkoZLVm+6JS+kmx1MrSKYOfGOumk9pH7JZu/WMS98DgLPl4O6qAdg5v3rBnzoC/lc/cDMLghsTnRKZ6BDtBYjabaxLn2Gt/8XtcCNkI8EvzKaXTs0UAfPRdaHgJJdZMqcwDHzyobQvWPJBgJGnp6FOBWKO2GKcwleNWGVBsS33CpoPVRjubpX2R9ExgsBMRDoczOYrTZVW2/j3HxoWTGf6Mf3Vl70KoO/xLh316F+jOmFAEkcOO9Ia4f+4gd8kwvtedmyztGH2endsHH5/xGV13wi4hudMw31JLe8xdpl/+dKFbDTwcpqg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678505; c=relaxed/relaxed; bh=oFdfq+cF3LQdgAm48Y9XRloCk1IGuh3dGA8j2DIRoK4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZCMalnedRV1hyrMjNZgWOUfBGuP4JlwZ1Bn/FIYGfVmmq9TrreT+wFNoBNhsZLdmjsR/8zqaIFYNSV3h0Rt/tMJequBtuI/A53hmHBfYiT+NgDYiG1O/WHYnblBGf7vJBROcwfh9KgjkY9OhPUwOn52stlOOmJ6TDanLquxPpjKu1UZoHMsujG/sJY0xl+8R/p5QU5jJDdXzCqKzdlnyoShBagUNTjm6DIGfp0moX/oUQafDMLwy+0eJZQJ9oHf8xTK7beyLuc+0vhJPEWN4x9I5j2LNrgTvpCMK0pW5YLDQz6xnXzo+jX53po863dZNTKDL2yvpBiVs1HxrdBN23Q== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=V+43+HF+; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=V+43+HF+; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1030; helo=mail-pj1-x1030.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHr0Cg1z2yTP for ; Wed, 13 May 2026 23:21:44 +1000 (AEST) Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-367d88b9940so3258358a91.1 for ; Wed, 13 May 2026 06:21:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678502; x=1779283302; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oFdfq+cF3LQdgAm48Y9XRloCk1IGuh3dGA8j2DIRoK4=; b=V+43+HF+gOpaOHW0mobq77tCchj/K9M0I49PXoyLshPYdVgVkBM2o7JTYgJsJIs6ZI km2ZP81mDTJnmVssxWPr//+HHutVZm3tcEm3lk1xp9ieEX0ILyyY4lvAKt9ZzHTkKDT4 NvhZLI6D6KYl7fZJ04uhRbo2PQD/gjFw1QCxzuPTeZ96g47rhDpDEaqNhR0xG8pmA4fl X9Pyu1EvqUIEOEnrgI3mJ23TZ2n4bth/8kFe46a7Cfp1/PKL6sAbtGYgXWxpp1qO6tq6 vIEK0dE2yizk6tYl6K08mgP+N4m4H3fWT5trFdvsgSjydUviLvEnGml7bfrqZlbnwzM0 rgNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678502; x=1779283302; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=oFdfq+cF3LQdgAm48Y9XRloCk1IGuh3dGA8j2DIRoK4=; b=Zfi5cQBVBMNNpWmtxvJ+3k03GqCJ4nrbn7ML3OFWrZAZrM7gWEx6Nv+71dVsp/Enbo A3a27Q4tncC422nItdrji1AZRQC0Vcnzv0xzSOvSViyF/GG5mvOb7iOhm/QbWhpu7O6S zQsz7rNHzaRkoyEUrHOj1rmMTyriXn+dL2Ty9+39PtEl/55QHGDrdBeA9VCNA9kRXEVd AJqCOUtCgEzbw7N83Bhntx089EpdqGtzWTK0Onz8P4I+9vX7QIx5ZXsN4xKr3Wj1DxlK fTa9Rlzcw0HIpksvq7RFUPQS2Bg7VX41E2MTzjAzqoiAj+AM09faDSmPc/whTrbRaiOK Ishw== X-Forwarded-Encrypted: i=1; AFNElJ8C1dzwT+UUJnXXfobi7Tj/N+meywO+0c8sc2RU8ugyYpECZvuQDzgkLJTl0sS4O/1xvYUqINuwumYCRWk=@lists.ozlabs.org X-Gm-Message-State: AOJu0YywyHjl9Q7mnIyOoYuFb8ao111M2/T11hd/BjZPX6RCjDoXzFbL HNyf9uhgTL3l3RtS62z+RFf19DvAdx2Cs9k7RlpySRcjrexrQUjnwR4SqotR/O1mtGk= X-Gm-Gg: Acq92OGN9/aAY9xjgg8vyk9glkJp2QGUmVG9BoPhaQrrQxekYw6GjPi6cntECpdgR8o TJbJYAmwThwshjb4BB3fzpOBgiYZF/mSmXkOzdKLOmOhFZ2uSF4Cm906SyIyBPPZoPgJkuxodnF Z1DcMBKTZCHZH5InPG6KgwI+ko2vzDyV/1V9pfYvroCGesn6oosPTpWTjaS5b7dkMkauyfpxukX tH3Hq/fRy/94DPB11a2iRQK9fRbsY0ciVgFzOS0i8nUlbQd2AddbXq6d7R3EIr6qr8EX4+B/f1B zuGDjToNO8aYg+O8+Psgm5zFaZfANoru6AhN7BkRd05vJArBbjsyZLhUkbtLmNCcltcM0ey726b cuGYh9JRUKUy2EfOTqTSYVi+kGTAvsWIBW91iEL9nz052KwQVdDOFf0dexQY8FwWCLECI/Kgtf0 oHG/21TLSW+PEG9ex8NwLqEwbepkER6N+kFpYvgjG7as46SZWSQK0snbkH1NAEJZUOvPE6k9g= X-Received: by 2002:a17:90a:d647:b0:35f:c729:de9f with SMTP id 98e67ed59e1d1-368f40ac9dbmr4195481a91.27.1778678501698; Wed, 13 May 2026 06:21:41 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.35 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:41 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 54/69] mm/sparse-vmemmap: Drop @pgmap from vmemmap population APIs Date: Wed, 13 May 2026 21:20:19 +0800 Message-ID: <20260513132044.41690-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The vmemmap population and memory hotplug paths no longer need @pgmap to decide whether a mapping can be optimized. That state is now carried in mem_section, and the architecture-specific population code can make the remaining decisions internally. Drop the @pgmap parameter from the vmemmap population helpers and the related memory hotplug interfaces, and remove the remaining dev_pagemap-specific coupling from those call chains. Signed-off-by: Muchun Song --- arch/arm64/mm/mmu.c | 5 ++--- arch/loongarch/mm/init.c | 5 ++--- arch/powerpc/include/asm/book3s/64/radix.h | 1 - arch/powerpc/mm/mem.c | 5 ++--- arch/riscv/mm/init.c | 5 ++--- arch/s390/mm/init.c | 5 ++--- arch/x86/mm/init_64.c | 5 ++--- include/linux/memory_hotplug.h | 8 +++----- include/linux/mm.h | 3 +-- mm/memory_hotplug.c | 13 ++++++------ mm/memremap.c | 4 ++-- mm/sparse-vmemmap.c | 23 ++++++++++------------ mm/sparse.c | 6 ++---- 13 files changed, 36 insertions(+), 52 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e5a42b7a0160..dd85e093ffdb 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -2024,13 +2024,12 @@ int arch_add_memory(int nid, u64 start, u64 size, return ret; } -void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { unsigned long start_pfn = start >> PAGE_SHIFT; unsigned long nr_pages = size >> PAGE_SHIFT; - __remove_pages(start_pfn, nr_pages, altmap, pgmap); + __remove_pages(start_pfn, nr_pages, altmap); __remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size); } diff --git a/arch/loongarch/mm/init.c b/arch/loongarch/mm/init.c index 055ecd2c8fd9..3f9ab54114c5 100644 --- a/arch/loongarch/mm/init.c +++ b/arch/loongarch/mm/init.c @@ -119,8 +119,7 @@ int arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params) return ret; } -void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { unsigned long start_pfn = start >> PAGE_SHIFT; unsigned long nr_pages = size >> PAGE_SHIFT; @@ -129,7 +128,7 @@ void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, /* With altmap the first mapped page is offset from @start */ if (altmap) page += vmem_altmap_offset(altmap); - __remove_pages(start_pfn, nr_pages, altmap, pgmap); + __remove_pages(start_pfn, nr_pages, altmap); } #endif diff --git a/arch/powerpc/include/asm/book3s/64/radix.h b/arch/powerpc/include/asm/book3s/64/radix.h index df67209b0c5b..0c9195dd50c9 100644 --- a/arch/powerpc/include/asm/book3s/64/radix.h +++ b/arch/powerpc/include/asm/book3s/64/radix.h @@ -316,7 +316,6 @@ static inline int radix__has_transparent_pud_hugepage(void) #endif struct vmem_altmap; -struct dev_pagemap; extern int __meminit radix__vmemmap_create_mapping(unsigned long start, unsigned long page_size, unsigned long phys); diff --git a/arch/powerpc/mm/mem.c b/arch/powerpc/mm/mem.c index 4c1afab91996..648d0c5602ec 100644 --- a/arch/powerpc/mm/mem.c +++ b/arch/powerpc/mm/mem.c @@ -158,13 +158,12 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, return rc; } -void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { unsigned long start_pfn = start >> PAGE_SHIFT; unsigned long nr_pages = size >> PAGE_SHIFT; - __remove_pages(start_pfn, nr_pages, altmap, pgmap); + __remove_pages(start_pfn, nr_pages, altmap); arch_remove_linear_mapping(start, size); } #endif diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 885f1db4e9bf..fa8d2f6f554b 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1742,10 +1742,9 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *param return ret; } -void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { - __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, pgmap); + __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap); remove_linear_mapping(start, size); flush_tlb_all(); } diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index 11a689423440..1f72efc2a579 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -276,13 +276,12 @@ int arch_add_memory(int nid, u64 start, u64 size, return rc; } -void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { unsigned long start_pfn = start >> PAGE_SHIFT; unsigned long nr_pages = size >> PAGE_SHIFT; - __remove_pages(start_pfn, nr_pages, altmap, pgmap); + __remove_pages(start_pfn, nr_pages, altmap); vmem_remove_mapping(start, size); } #endif /* CONFIG_MEMORY_HOTPLUG */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 77b889b71cf3..df2261fa4f98 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1288,13 +1288,12 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end) remove_pagetable(start, end, true, NULL); } -void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) +void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap) { unsigned long start_pfn = start >> PAGE_SHIFT; unsigned long nr_pages = size >> PAGE_SHIFT; - __remove_pages(start_pfn, nr_pages, altmap, pgmap); + __remove_pages(start_pfn, nr_pages, altmap); kernel_physical_mapping_remove(start, start + size); } #endif /* CONFIG_MEMORY_HOTPLUG */ diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 7c9d66729c60..815e908c4135 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -135,10 +135,9 @@ static inline bool movable_node_is_enabled(void) return movable_node_enabled; } -extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap); +extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap); extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap); + struct vmem_altmap *altmap); /* reasonably generic interface to expand the physical pages */ extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages, @@ -308,8 +307,7 @@ extern int sparse_add_section(int nid, unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, - struct dev_pagemap *pgmap); + struct vmem_altmap *altmap); extern struct zone *zone_for_pfn_range(enum mmop online_type, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages); diff --git a/include/linux/mm.h b/include/linux/mm.h index 5f45de90972d..87e98bdb0417 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4846,8 +4846,7 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) #endif struct page * __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap); + unsigned long nr_pages, int nid, struct vmem_altmap *altmap); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index c9c69f827efa..5c60533677a1 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -577,7 +577,6 @@ void remove_pfn_range_from_zone(struct zone *zone, * @pfn: starting pageframe (must be aligned to start of a section) * @nr_pages: number of pages to remove (must be multiple of section size) * @altmap: alternative device page map or %NULL if default memmap is used - * @pgmap: device page map or %NULL if not ZONE_DEVICE * * Generic helper function to remove section mappings and sysfs entries * for the section of the memory we are removing. Caller needs to make @@ -585,7 +584,7 @@ void remove_pfn_range_from_zone(struct zone *zone, * calling offline_pages(). */ void __remove_pages(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) + struct vmem_altmap *altmap) { const unsigned long end_pfn = pfn + nr_pages; unsigned long cur_nr_pages; @@ -600,7 +599,7 @@ void __remove_pages(unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - sparse_remove_section(pfn, cur_nr_pages, altmap, pgmap); + sparse_remove_section(pfn, cur_nr_pages, altmap); } } @@ -1429,7 +1428,7 @@ static void remove_memory_blocks_and_altmaps(u64 start, u64 size) remove_memory_block_devices(cur_start, memblock_size); - arch_remove_memory(cur_start, memblock_size, altmap, NULL); + arch_remove_memory(cur_start, memblock_size, altmap); /* Verify that all vmemmap pages have actually been freed. */ WARN(altmap->alloc, "Altmap not fully unmapped"); @@ -1472,7 +1471,7 @@ static int create_altmaps_and_memory_blocks(int nid, struct memory_group *group, ret = create_memory_block_devices(cur_start, memblock_size, nid, params.altmap, group); if (ret) { - arch_remove_memory(cur_start, memblock_size, params.altmap, NULL); + arch_remove_memory(cur_start, memblock_size, params.altmap); kfree(params.altmap); goto out; } @@ -1558,7 +1557,7 @@ int add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags) /* create memory block devices after memory was added */ ret = create_memory_block_devices(start, size, nid, NULL, group); if (ret) { - arch_remove_memory(start, size, params.altmap, NULL); + arch_remove_memory(start, size, params.altmap); goto error; } } @@ -2270,7 +2269,7 @@ static int try_remove_memory(u64 start, u64 size) * No altmaps present, do the removal directly */ remove_memory_block_devices(start, size); - arch_remove_memory(start, size, NULL, NULL); + arch_remove_memory(start, size, NULL); } else { /* all memblocks in the range have altmaps */ remove_memory_blocks_and_altmaps(start, size); diff --git a/mm/memremap.c b/mm/memremap.c index 81766d822400..053842d45cb1 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -97,10 +97,10 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) PHYS_PFN(range_len(range))); if (pgmap->type == MEMORY_DEVICE_PRIVATE) { __remove_pages(PHYS_PFN(range->start), - PHYS_PFN(range_len(range)), NULL, pgmap); + PHYS_PFN(range_len(range)), NULL); } else { arch_remove_memory(range->start, range_len(range), - pgmap_altmap(pgmap), pgmap); + pgmap_altmap(pgmap)); kasan_remove_zero_shadow(__va(range->start), range_len(range)); } mem_hotplug_done(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 549be01d90f8..a807210fe9e1 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -379,8 +379,7 @@ int __meminit vmemmap_populate_hugepages(unsigned long start, unsigned long end, } struct page * __meminit __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); @@ -474,11 +473,9 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) } static struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap) { - struct page *page = __populate_section_memmap(pfn, nr_pages, nid, altmap, - pgmap); + struct page *page = __populate_section_memmap(pfn, nr_pages, nid, altmap); memmap_pages_add(section_nr_vmemmap_pages(pfn, nr_pages)); @@ -486,7 +483,7 @@ static struct page * __meminit populate_section_memmap(unsigned long pfn, } static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) + struct vmem_altmap *altmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); @@ -567,7 +564,7 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) * usage map, but still need to free the vmemmap range. */ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) + struct vmem_altmap *altmap) { struct mem_section *ms = __pfn_to_section(pfn); bool section_is_early = early_section(ms); @@ -605,7 +602,7 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, * section_activate() and pfn_valid() . */ if (!section_is_early) - depopulate_section_memmap(pfn, nr_pages, altmap, pgmap); + depopulate_section_memmap(pfn, nr_pages, altmap); else if (memmap) free_map_bootmem(memmap); @@ -656,9 +653,9 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, return pfn_to_page(pfn); section_set_order_range(pfn, nr_pages, order); - memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); + memmap = populate_section_memmap(pfn, nr_pages, nid, altmap); if (!memmap) { - section_deactivate(pfn, nr_pages, altmap, pgmap); + section_deactivate(pfn, nr_pages, altmap); return ERR_PTR(-ENOMEM); } @@ -720,13 +717,13 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, } void sparse_remove_section(unsigned long pfn, unsigned long nr_pages, - struct vmem_altmap *altmap, struct dev_pagemap *pgmap) + struct vmem_altmap *altmap) { struct mem_section *ms = __pfn_to_section(pfn); if (WARN_ON_ONCE(!valid_section(ms))) return; - section_deactivate(pfn, nr_pages, altmap, pgmap); + section_deactivate(pfn, nr_pages, altmap); } #endif /* CONFIG_MEMORY_HOTPLUG */ diff --git a/mm/sparse.c b/mm/sparse.c index f314b9babc4a..bdf23709a1c7 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -224,8 +224,7 @@ size_t mem_section_usage_size(void) #ifndef CONFIG_SPARSEMEM_VMEMMAP struct page __init *__populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap) { unsigned long size = PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); @@ -283,8 +282,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, if (pnum >= pnum_end) break; - map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); + map = __populate_section_memmap(pfn, PAGES_PER_SECTION, nid, NULL); if (!map) panic("Failed to allocate memmap for section %lu\n", pnum); memmap_boot_pages_add(section_nr_vmemmap_pages(pfn, PAGES_PER_SECTION)); From patchwork Wed May 13 13:20:20 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237869 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LH1X93Z9; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20904-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJ372vMz1y5L for ; Wed, 13 May 2026 23:21:55 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvHz1YDqz2yvC; Wed, 13 May 2026 23:21:51 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1031" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678511; cv=none; b=mMBAfj6AKDHuKoeNuPo1cE9YEGyumimP1++94F2Dan+NupGBy7fLSWCPItjA4bnjR55F8o/yHwhQ+eLiHsshqU3zMhs/O1fWOBSz5TWAkpUXXjL6YIC1+ZBCbJMaF4uOP7tTa38jz91UJNoUqHDQYbbkOzu5xln7cVUThbBWn+lgFEvZv6+5ggEI6ZiDej8C7bAVSB3l7RS5Fy4tTzQyzKlTjp4VHKyoLUuYQrEuv4gStoi3khi+E17zHhGLt2rjCdh0GeNFm/8E9i3PmzQT3DiqXqzvhXiJG8aS8CeQkteWdaQ2an6l8xRxmuj543IzptY8ehwO73yPL1XvMqmYqQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678511; c=relaxed/relaxed; bh=bX0eh9yAs1Ger4W6kkJzAbg+VRL/E8mjDpD877qpJLc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eiinqDUIXiBswwfYWb6aCyeAbq0u3ALK7CWYxH5w0K2BGK9CSVB5f4CfSjL4ER0F9Z7Q+SuG2E3csEr2+25rv9UXZHmVOweJBUH+1QMIR98kJEoR+3iZx31FaEeBcwe7B/WrSbip+inI7Gqi0tdvffv4ki/vy0eehQaV8dMcgGf6fJBlnbfuJtmxDa3Tk37AS0tBppZ2OTEwen/bN7pz9XJZ65jQXDIN9ETmvL/xBOhXcImd0a9dOr2j22FiJkMhDPZpyMuGRYvH8R3q1BLyo0ksO5E4PYRNCr4VyEAA4hstjSAfExD47kUnpfj2kLF2aSBYBZwvNq6aEdQpkfwGGA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LH1X93Z9; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=LH1X93Z9; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvHy2kFsz2yTP for ; Wed, 13 May 2026 23:21:50 +1000 (AEST) Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-3664df32e91so5493821a91.3 for ; Wed, 13 May 2026 06:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678508; x=1779283308; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bX0eh9yAs1Ger4W6kkJzAbg+VRL/E8mjDpD877qpJLc=; b=LH1X93Z9nBmjUxWgAi1Gxt0xY5OCY40WHqbQv8NEIxzqBb+HiL8S522atloG7a7bFF wWtk2E+ViLlLspUiORsZlbN6fIYZ228m8z7CE1J0IYRC8IDoVfQpJBkXsIx5K4MW8wUr Ra3FAIkcisNZGLVtUw9hieHCQ6ZpS+8koch3TQ8ITiRb7Hz9OFpeHHW6FGtZLjKMcrid HmvRUyqRpX8/GGFBskVUAL2aSvo9NshskMNNBDuaostbcek279Uc1CLt08YYaNCx6P/t gFHZb6svfizXqNEJYIFEqcOgtVi0AlRhVd8dQwAneBwxYb0H+afZjcT/wZLitKs1Tc2d 98Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678508; x=1779283308; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bX0eh9yAs1Ger4W6kkJzAbg+VRL/E8mjDpD877qpJLc=; b=OoaVDaj2NlZp6wA12iYF9kWxCTevjn9FQ7oJdPMGKE4iwBynlcqmzfYDcqxTwSWEIO 9l4UsyyVlHGeaL73z5ZlKEiZxxC1Ha8EQyADPaefPW/laOsKoK1Gx+DvWB5IST2Ty/+b 8wqJSHwdkrSoj9z0UOEfMPW+hpFrehuiV1roCi2KDgdQi1nkrsmMyWV53zlWiBOVoc4y KzXyo1lmk1UtcQVDqZB/zFyvTpKMLXDpXt5KnYuvswHPgnA5gmDu8rDITzMb1fllLc+c cgfUW7UooO0BkJc2LiC3b7bIVfRpN8DekcEtEb0Q/yvCHwTq5IrR2f5NynYbw3ZrR0h6 s5Rg== X-Forwarded-Encrypted: i=1; AFNElJ/rgkawkyiwtanxd623pfkMdD4QPuH/Eh+3+tjCcDx5UACfNThZvRn3zm6WZepTIsTrps9mQxeOBUM7Q9g=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyAIhN8GZu+ap1BFs7YpYe8LsUTPXn+wD+IdC5EfLRJ3Znc2Kc8 SpE9+6oXCO9IwZ5aEiYx4uZm8DomaOqJmNFY4jvF1wzwEQ9xynzj6cO/m1HqPeTahdo= X-Gm-Gg: Acq92OEg743BT0RoXvoHoVsIf6ho30JpmxBYVZiFuIxp3jyYfrxzFoHXat2NYe/tSpF WCsz3I63pcclOxSV+QkI3GsuI0IPTVEaRsQhyEgbFL2xd8cg0uynTePKdjYTRaMkD4yprwlGGM+ JgYWL9QQ3QcpnxILFeTFpbNpbedto+RhvXZfoUYmYWuK7YIIm2kHMYqoqXmaRdovesFuqgOIl6R p45IMPuogkA105b7sAlHHGygb0z8AvgVfTihUu+GN1RRrjiyMBaLOo9gDlzm0yiYjfM5+UgmjyR 3QvcNnlMoNg3sTdCylNp9EwtdMwAUhNfbFEKYSa/7xcV3XTtHt/6FmhcytNzrWT1gxFbTcQuD/x HZfilPMiPhtlGwrIRQu1i457y4jFNG3jmqaTKQli+gr8Lk4XoisGypj7uo2bJ+/zGzTVP9b3nd2 lfxjPsu7JkXmkt7Ku48DBM4kTQF7CtZRqD0NRcMpdz908xL84yD4IL5X7SIiqCHZ7XHy7YbO4= X-Received: by 2002:a17:90b:3f86:b0:366:33a6:9921 with SMTP id 98e67ed59e1d1-368f3a909b4mr3551470a91.4.1778678508386; Wed, 13 May 2026 06:21:48 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:47 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 55/69] mm/sparse: Decouple section activation from ZONE_DEVICE Date: Wed, 13 May 2026 21:20:20 +0800 Message-ID: <20260513132044.41690-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org sparse_add_section()/section_activate() currently takes struct dev_pagemap only to obtain the compound page order. Pass the order explicitly instead of routing it through a ZONE_DEVICE specific structure. This removes the dev_pagemap dependency from the generic sparse memory population path and keeps the interface usable for other callers (if possible). Signed-off-by: Muchun Song --- include/linux/memory_hotplug.h | 4 ++-- mm/memory_hotplug.c | 4 ++-- mm/sparse-vmemmap.c | 14 ++++++-------- 3 files changed, 10 insertions(+), 12 deletions(-) diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 815e908c4135..083f0abea62d 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -304,8 +304,8 @@ extern void remove_pfn_range_from_zone(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages); extern int sparse_add_section(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap); + unsigned long nr_pages, unsigned int order, + struct vmem_altmap *altmap); extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap); extern struct zone *zone_for_pfn_range(enum mmop online_type, diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 5c60533677a1..ef1595bdfd3a 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -385,6 +385,7 @@ int __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, unsigned long cur_nr_pages; int err; struct vmem_altmap *altmap = params->altmap; + unsigned int order = params->pgmap ? params->pgmap->vmemmap_shift : 0; if (WARN_ON_ONCE(!pgprot_val(params->pgprot))) return -EINVAL; @@ -412,8 +413,7 @@ int __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - err = sparse_add_section(nid, pfn, cur_nr_pages, altmap, - params->pgmap); + err = sparse_add_section(nid, pfn, cur_nr_pages, order, altmap); if (err) break; cond_resched(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a807210fe9e1..667424aadd6b 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -613,16 +613,14 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, } static struct page * __meminit section_activate(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) + unsigned long nr_pages, unsigned int order, + struct vmem_altmap *altmap) { struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; struct page *memmap; - unsigned int order; int rc; - order = pgmap ? pgmap->vmemmap_shift : 0; /* All sub-sections within a section must share the same order. */ if (nr_pages < PAGES_PER_SECTION && section_order(ms) && section_order(ms) != order) return ERR_PTR(-ENOTSUPP); @@ -667,8 +665,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * @nid: The node to add section on * @start_pfn: start pfn of the memory range * @nr_pages: number of pfns to add in the section + * @order: section order * @altmap: alternate pfns to allocate the memmap backing store - * @pgmap: alternate compound page geometry for devmap mappings * * This is only intended for hotplug. * @@ -682,8 +680,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * * -ENOMEM - Out of memory. */ int __meminit sparse_add_section(int nid, unsigned long start_pfn, - unsigned long nr_pages, struct vmem_altmap *altmap, - struct dev_pagemap *pgmap) + unsigned long nr_pages, unsigned int order, + struct vmem_altmap *altmap) { unsigned long section_nr = pfn_to_section_nr(start_pfn); struct mem_section *ms; @@ -694,7 +692,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (ret < 0) return ret; - memmap = section_activate(nid, start_pfn, nr_pages, altmap, pgmap); + memmap = section_activate(nid, start_pfn, nr_pages, order, altmap); if (IS_ERR(memmap)) return PTR_ERR(memmap); From patchwork Wed May 13 13:20:21 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237870 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OU8nKm+c; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20905-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJC1Jtzz1y5L for ; Wed, 13 May 2026 23:22:03 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJ56bm9z2yvY; Wed, 13 May 2026 23:21:57 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678517; cv=none; b=kn2ppuWQDGER9rXewk5SuXnpPEkqGf1QmWkQkjYrv4cUkTQgzwqtNa/C30VPz6CwQQnLocnfWPr3TGLg5TSmZaq9TqbiP1i/MubgR+hoxj3VeA4s9ATCUxmivCpDEKa7ZDcJJL1MXh+pFC/fW3Agj7dl8W2kfnGITdqb51fT51a2rM41sFN81+BAdfHIZh6JfsXqVvjM70OPUa85dQkjrGPrgIYap8DU3g8+O3s4iSvXe52SGE0+oX4GITS9lB33DLr0GagS7+VlH++jVf94jQdxnmDtWm50Gfd0hHH5G0yTGmR5Fbx++ulRTBCubhaKAKeJJVqjlOfqMpCx4mdkhA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678517; c=relaxed/relaxed; bh=09ZAgIruVIYnDt/DEHrKUKC/462Uq37TlLUpE9pDl/8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mlwcc2iTMVCVHoZaLj7TQ0QArym0DlLOIOyAK6gJy3rogSCIUVgq0K6EBEUUs2jtp/XbwhpkLh7a+6GDP0pSlogZSOVgX/DrYhKevu+nYSY7HpQpjISULFiGSIoyz/dipRpGdq1m7WSeKOaPhA9+LdEEPFWs2xGIKQTWy+zziGYPwioHCFUCg6la1LUiMl6LQTRkjUkpXlnRDvATvtvDQPeYjEhXc6ipaST21mJmQ45WPMIZwMGF8rq7T1lDYw8sxA9j217bxuZgvZmlqk9rcc1J0oqqARBeBATZ3JfgSlk8/gVnFPipwfi/rA0tl/Rj7KmP2MGRr/ZMc0hAFA48Ag== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OU8nKm+c; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102f; helo=mail-pj1-x102f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=OU8nKm+c; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102f; helo=mail-pj1-x102f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJ50bDCz2yVP for ; Wed, 13 May 2026 23:21:56 +1000 (AEST) Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-367d88b9940so3258795a91.1 for ; Wed, 13 May 2026 06:21:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678515; x=1779283315; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=09ZAgIruVIYnDt/DEHrKUKC/462Uq37TlLUpE9pDl/8=; b=OU8nKm+cfLuDrJC6fS8HctDqtTNh6Q3ZTntB+mnhaVFVH9JoVozKQ9n0aAYhY/ceZ5 wfRvyQ3J8pXUYDOyP5npDAcgB3VvXW4EccF377ytJ6IBKskYYrCJC1nwSlGz4mwI5lGi eqCAZBR2dL444/KsFsWKKTp7uqVNG2UqnlsLiXQoGVVeDEgX022/ftC31shW9zr3hgvH Bb+JC+rYxOYxBwJtNCCCisXTVnIiBVO5hRJl7w1tnNnMz/6orPeZfy+aPuY423jNikBE gc6Id2PPQdX5wFKhtmJIK1JYM2XN4Wfvw1BlFRP1WMVuKbQRw6HqNHQv3gNsTBIMtAWL jUoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678515; x=1779283315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=09ZAgIruVIYnDt/DEHrKUKC/462Uq37TlLUpE9pDl/8=; b=CxLPP4h5VxQB2W/IDRKLsUweNPCCOWLnzSQk5tStOTBTOd5dGDK6p9KLTnu9jnBdZy 88yPZS3+darxRsF/78lEahmzPrdtrk1X1dtnVWt6SX1UYJaSNNqW/dVl7WKoHnOe8+bg Yrz1rIKfX+FIrnF7j6Zxp4M5qKlvnkdcVYkaMOiswuDH3Ji4tgvBkQxWkcHL6ENX5whi Mj1CiHXqpo5SQ5d/2Z4r1iVhBaS+xYQdL2d7HCj6HsvHQyJjijpxuhqQviHp5Kmfqj0i rWpLU4iCBOTTet9lNx0bhW2h/1SKxlO72exMNrdv9ye6/2PNSc717ixH04eALEPy/MoX tgEg== X-Forwarded-Encrypted: i=1; AFNElJ80IMK/e+7GZw2Buhaxeg1BUuD2/zAbEVNux0CmV0jqv4K2xHpb6MM+TyvDppS13lqKlzUuiaK1s8WhU7M=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxElkBZm8aQS0Cg6RfCoxv95xzw8zUprWU9f7t5SrxPpME8ZSKV hDk3k0K9ky4KxYth1iQ2RoNc8bVcE3sSrhwJYojuMWQeeGL4lc9S25sO+PxJjnCB3TI= X-Gm-Gg: Acq92OHefrrhpLCDDz43ePzmVhG0Sp4RaNUUTeYWOy4RI3p2W342Q9U6ihM/XDYhHDA IvbdukSgfOvXXit8/x/1mJxs/NtmEKFUlED9sjgbEn8F4TggLB5kOIw8S7ExEJ6M2LHOz54Xdvx 1oKuui38SwUnrj56/+5ZdrDZ9D2YJ0wB+auaUNzKusJdSO9AIcUxz6kx9s0AF973MBfuILpNEjz /f1lhdXAR0RlsWwtNGyL42x+MdvSPEde9URbvf5zxH8gndIAycPqSPzhE+vLWLtD1CqjANE6o68 6zCyM5ZV/5ksYxiv5gy04h6NqqpLhPAHus/J1MZVaZT8rjKVMxlwReEV4Sdm7+P82dOsMf758yf Bn7hFaWxL1A/roS2YHybbBJ0ycSZJqtyH7VgTAQOm7KnZtfde+SZ7vxIk5LqzOi+/FE4JBpVp30 dhtr3QSHIsEDVT6Hjk0ETqdkXr3+HfVD6nIzJV8E3JlVM3PVKx4NOzUVFTNmEO X-Received: by 2002:a17:90b:1d4e:b0:368:2289:42cc with SMTP id 98e67ed59e1d1-368f4092542mr3912735a91.24.1778678514599; Wed, 13 May 2026 06:21:54 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:21:54 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 56/69] mm: Redefine HVO as Hugepage Vmemmap Optimization Date: Wed, 13 May 2026 21:20:21 +0800 Message-ID: <20260513132044.41690-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HVO no longer refers only to HugeTLB vmemmap optimization. The same optimization is now used more broadly for large compound-page mappings, so the old expansion is too narrow. Redefine HVO as Hugepage Vmemmap Optimization and update the generic documentation, Kconfig text, and comments accordingly. Signed-off-by: Muchun Song --- Documentation/admin-guide/kernel-parameters.txt | 2 +- Documentation/admin-guide/mm/hugetlbpage.rst | 4 ++-- Documentation/admin-guide/mm/memory-hotplug.rst | 2 +- Documentation/admin-guide/sysctl/vm.rst | 3 ++- Documentation/mm/vmemmap_dedup.rst | 2 +- fs/Kconfig | 4 ++-- include/linux/mmzone.h | 2 +- mm/Kconfig | 2 +- mm/hugetlb_vmemmap.c | 2 +- mm/hugetlb_vmemmap.h | 2 +- mm/memory-failure.c | 6 +++--- 11 files changed, 16 insertions(+), 15 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 0eb64aab3685..2d4cfdcb7535 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2114,7 +2114,7 @@ Kernel parameters hugetlb_free_vmemmap= [KNL] Requires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP enabled. - Control if HugeTLB Vmemmap Optimization (HVO) is enabled. + Control if Hugepage Vmemmap Optimization (HVO) for HugeTLB is enabled. Allows heavy hugetlb users to free up some more memory (7 * PAGE_SIZE for each 2MB hugetlb page). Format: { on | off (default) } diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 67a941903fd2..3f98ca1d7ce1 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -172,8 +172,8 @@ default_hugepagesz will all result in 256 2M huge pages being allocated. Valid default huge page size is architecture dependent. hugetlb_free_vmemmap - When CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP is set, this enables HugeTLB - Vmemmap Optimization (HVO). + When CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP is set, this enables Hugepage + Vmemmap Optimization (HVO) for HugeTLB. When multiple huge page sizes are supported, ``/proc/sys/vm/nr_hugepages`` indicates the current number of pre-allocated huge pages of the default size. diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst index 0207f8725142..d5e350607baa 100644 --- a/Documentation/admin-guide/mm/memory-hotplug.rst +++ b/Documentation/admin-guide/mm/memory-hotplug.rst @@ -682,7 +682,7 @@ block might fail: ZONE_MOVABLE for increasing the reliability of gigantic page allocation against the potential loss of hot-unplug reliability. -- Out of memory when dissolving huge pages, especially when HugeTLB Vmemmap +- Out of memory when dissolving huge pages, especially when Hugepage Vmemmap Optimization (HVO) is enabled. Offlining code may be able to migrate huge page contents, but may not be able diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 97e12359775c..9f333970fdb2 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -665,7 +665,8 @@ This knob is not available when the size of 'struct page' (a structure defined in include/linux/mm_types.h) is not power of two (an unusual system config could result in this). -Enable (set to 1) or disable (set to 0) HugeTLB Vmemmap Optimization (HVO). +Enable (set to 1) or disable (set to 0) Hugepage Vmemmap Optimization (HVO) for +HugeTLB. Once enabled, the vmemmap pages of subsequent allocation of HugeTLB pages from buddy allocator will be optimized (7 pages per 2MB HugeTLB page and 4095 pages diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index 9fa8642ded48..44e80bd2e398 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -8,7 +8,7 @@ A vmemmap diet for HugeTLB and Device DAX HugeTLB ======= -This section is to explain how HugeTLB Vmemmap Optimization (HVO) works. +This section is to explain how Hugepage Vmemmap Optimization (HVO) for HugeTLB works. The ``struct page`` structures are used to describe a physical page frame. By default, there is a one-to-one mapping from a page frame to its corresponding diff --git a/fs/Kconfig b/fs/Kconfig index f6cee1bbb1fc..496cfa2379e5 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -261,11 +261,11 @@ menuconfig HUGETLBFS if HUGETLBFS config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON - bool "HugeTLB Vmemmap Optimization (HVO) defaults to on" + bool "Hugepage Vmemmap Optimization (HVO) for HugeTLB defaults to on" default n depends on HUGETLB_PAGE_OPTIMIZE_VMEMMAP help - The HugeTLB Vmemmap Optimization (HVO) defaults to off. Say Y here to + The Hugepage Vmemmap Optimization (HVO) for HugeTLB defaults to off. Say Y here to enable HVO by default. It can be disabled via hugetlb_free_vmemmap=off (boot command line) or hugetlb_optimize_vmemmap (sysctl). endif # HUGETLBFS diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 7484e7be7b6d..efb37f2ffec4 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -97,7 +97,7 @@ #define MAX_FOLIO_NR_PAGES (1UL << MAX_FOLIO_ORDER) /* - * HugeTLB Vmemmap Optimization (HVO) requires struct pages of the head page to + * Hugepage Vmemmap Optimization (HVO) requires struct pages of the head page to * be naturally aligned with regard to the folio size. * * HVO which is only active if the size of struct page is a power of 2. diff --git a/mm/Kconfig b/mm/Kconfig index ddd10cb4d0a3..c85ed7d7f37d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -416,7 +416,7 @@ config SPARSEMEM_VMEMMAP_OPTIMIZATION # # Select this config option from the architecture Kconfig, if it is preferred -# to enable the feature of HugeTLB/dev_dax vmemmap optimization. +# to enable the feature of HVO. # config ARCH_WANT_OPTIMIZE_DAX_VMEMMAP bool diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index fce772e95adc..6f6f1740f540 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * HugeTLB Vmemmap Optimization (HVO) + * Hugepage Vmemmap Optimization (HVO) for HugeTLB * * Copyright (c) 2020, ByteDance. All rights reserved. * diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 2b0a85e09602..b4d0ba27b42c 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* - * HugeTLB Vmemmap Optimization (HVO) + * Hugepage Vmemmap Optimization (HVO) for HugeTLB * * Copyright (c) 2020, ByteDance. All rights reserved. * diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 866c4428ac7e..ad6416145667 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -175,9 +175,9 @@ static int __page_handle_poison(struct page *page) /* * zone_pcp_disable() can't be used here. It will * hold pcp_batch_high_lock and dissolve_free_hugetlb_folio() might hold - * cpu_hotplug_lock via static_key_slow_dec() when hugetlb vmemmap - * optimization is enabled. This will break current lock dependency - * chain and leads to deadlock. + * cpu_hotplug_lock via static_key_slow_dec() when HVO for HugeTLB + * is enabled. This will break current lock dependency chain and leads + * to deadlock. * Disabling pcp before dissolving the page was a deterministic * approach because we made sure that those pages cannot end up in any * PCP list. Draining PCP lists expels those pages to the buddy system, From patchwork Wed May 13 13:20:22 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237871 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nw1Tax7A; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20906-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJL3VN9z1y5L for ; Wed, 13 May 2026 23:22:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJD2Bzqz2yx5; Wed, 13 May 2026 23:22:04 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102e" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678524; cv=none; b=QotYAlxSlqLP6uTe4hSUmLICmcBoir81flizb5Rk4aqu9lHye4aHuM7qk5QDgYeUE+Fku0iUajs4z7pNHPq172mSF6FZupHbKItrjMCDqMzYrUwd5DdV0Ss98YES2tlG9NV9C08Jzvw47aBXw3ln2f9NlEZfRTE9AwbiHM9JJWMjkr7AHdTmnektpDCBpVj71Tk6R9enGYnvlkIDdSN1qe1xt0ApizyVvu0p7vopQ+EB2kFKRhchx5j6WAfQGuQwwycirZoKDpBwYmMKcWqsiy/CxPQaGZe6g1MR9SaPCMpAkO9YZgArD4fjzLGSwCySdrRWu1L4CO6ZyTItzKctHw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678524; c=relaxed/relaxed; bh=8xAcJFzw5l8sNYVc/df08CWPD7oki7YaxoFEOHbnbjI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=llXwFqf7KIB7cXLpRPk+B7jAHrVdDwzNdhDbxQYE9wNt5+2PX/Cqdi9yS6NIsk5FsPS5AWyJs7i7b/F8V8ARAx3bPfk33wgwlyxCohN9HJuTqFDPn1XttezrugFgiC3vuWe2ZHqrOJ8JqgIiRuUF02s1JXRTJOLlTvHLhUx9BsDbRmT8UcJ45bxAx28QlDIEQfCUXDQU2WMEA83w+bhUyTeEr04/hljRcHZILAmv2jOgkQvgE/Czxh+u1Ccncn2H44mw0fXtqrYNePEtqNqPOY8LktKU8zEupqXL49OB+rzx/o0VMkFeqQy8ChM6Rh+BtSS+pwD82L22MGHY4uMvTA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nw1Tax7A; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=Nw1Tax7A; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102e; helo=mail-pj1-x102e.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJC1xM6z2yVP for ; Wed, 13 May 2026 23:22:03 +1000 (AEST) Received: by mail-pj1-x102e.google.com with SMTP id 98e67ed59e1d1-365e20fe3b8so3905613a91.3 for ; Wed, 13 May 2026 06:22:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678521; x=1779283321; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8xAcJFzw5l8sNYVc/df08CWPD7oki7YaxoFEOHbnbjI=; b=Nw1Tax7AVpNcEHKg9IBb2HWTqSzXtYhSx4wKLA6NDK9RXuBUV4CaB23mXuqnKvrpUs C1xAK83k5alY3xHPEpybZsDC7C73bj7TCqtYFXmO8ZbDceC7+im+8xd/xCtnsEdyryw0 94oIv2/MHo1YBnBp6TEihESqgd8pA940f7BaclL/G8d1kLuLHZObHFqRqy3ZcEHqQU/6 V9ZCBs3Ym5EBsTTWnZe8acxGwaKYw0fP1meGqS6PAr/Kvpo30N0rVoYFxw9xaGjzl2Tr lCle321t8is/FFVxa0LVuIIMU0/vWoA9RjKqMz5TpD4KV5f4Fh+Ev9t0mzMJABzZ6ImN GuNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678521; x=1779283321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=8xAcJFzw5l8sNYVc/df08CWPD7oki7YaxoFEOHbnbjI=; b=qn9OvkhSOe+2Ydmtr0GztTbq6tBVgzRxTLliftuMCkE3SPQa1QcFTopj/vxS00l8DE 8ww92kjgIYtqRtnvuQQT62m18pGYAysn3ZyKPq4ieKwCS13RvpWFoLhxGlMdW0xnVw7W bKf545TfWJ8dkZjpadn6RsN9ZpfUs7X/4rQycafCbpdrhkN1LSXEdi/uWYL2VL7OcIqt p5af71HMmLzrIKfsTRHlPPWYmueaCZEptjDK6JlQ0H19JTMrTBBHYzWGJHlp6S90AMWJ 2STndVrvWI/K+/KasfMYZCohm/uDfy/P2hnOoPSJaS2U9MGDFvoN2HPKWcD4aYxz77j8 WIew== X-Forwarded-Encrypted: i=1; AFNElJ/eLDO5xwVkckPvibS77Gr7KzhS1qwbjeXRCx5vS4ljboZ4CXVm18BSC2UhyBSnitQjZWJj1Taf1zGejyQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0YyQWO4ZC3sd0Lt8Zh/2gXMn0Y54LuPIslwhH/z9T3Gv5NrDQlhH RzTl5z9ekeu0mbTG6unLKsIIZ3u2ZmzVnzt1b20Lj9LQ/txzVCggA+QD6VNixdkhm0M= X-Gm-Gg: Acq92OEIhvY9sISO3JzGNCBlp+xLO4B648IPVCuwbGirFomeExb/cYU3AnhzkSjqsEQ Y48oRqxry6BtcseLyGnjlJQJOWj7DcZ241hDfPry1UgREEfR2+haEhiUuUmIxnQLPpLFixqqwjg dNL5ATlm27oTpI1zoxf3n0mZLHiop9pYCoUETULgyUIaTFxPKiDKG3yWPKMgn8HUEoTSHZikejl 1aO5KsmS4apbzbHHO/iN4aQuB15boxjz+2YqyfvCSdr0kjQdLWRnmqqxTKyiYqP00+pEhVpBlYS vxfo3OyxMubm+yq8SwIjOFQZJ2RHMP8C8kOd8Spe6kqwfaSFFi1zwlueLuAArva9o6s3FLYhIko f2sAbKOtMRkd24XLH88VMzsDRVlU+HhyrpNedqpeW3Gx4Op39ZL7ase9odnCd1k6vPK95xrb3EX YDasF6beA47CQoH11kgeM25m4teR+Kf8yq8mvz0muA1+9phywmyTNh5hIWUVkzD9bg1OLwe1I= X-Received: by 2002:a17:90a:617:b0:368:f7a7:daee with SMTP id 98e67ed59e1d1-368f7a7df06mr1739782a91.8.1778678520875; Wed, 13 May 2026 06:22:00 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.21.55 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:00 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 57/69] mm/sparse-vmemmap: Consolidate HVO enable checks Date: Wed, 13 May 2026 21:20:22 +0800 Message-ID: <20260513132044.41690-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HVO depends on build-time conditions that are not fully expressible in Kconfig, including whether sizeof(struct page) is a power of two and whether the supported folio order range can use the optimized layout. Those checks are currently duplicated in several places. Define SPARSEMEM_VMEMMAP_OPTIMIZATION in bounds.c when the build-time requirements are met, and use that generated constant to guard the generic HVO code. This centralizes the build-time checks instead of repeating them throughout the HVO paths. Signed-off-by: Muchun Song --- arch/x86/entry/vdso/vdso32/fake_32bit_build.h | 1 - drivers/dax/Kconfig | 2 +- fs/Kconfig | 2 +- include/linux/mm_types.h | 3 +- include/linux/mmzone.h | 38 ++++++++----------- include/linux/page-flags-layout.h | 2 + include/linux/page-flags.h | 28 ++------------ kernel/bounds.c | 5 +++ mm/Kconfig | 2 +- mm/hugetlb_vmemmap.c | 2 + mm/hugetlb_vmemmap.h | 4 +- mm/internal.h | 3 -- mm/sparse.c | 6 +-- mm/util.c | 2 +- 14 files changed, 38 insertions(+), 62 deletions(-) diff --git a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h index 5f8424eade2b..db1b15f686e3 100644 --- a/arch/x86/entry/vdso/vdso32/fake_32bit_build.h +++ b/arch/x86/entry/vdso/vdso32/fake_32bit_build.h @@ -11,7 +11,6 @@ #undef CONFIG_PGTABLE_LEVELS #undef CONFIG_ILLEGAL_POINTER_VALUE #undef CONFIG_SPARSEMEM_VMEMMAP -#undef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION #undef CONFIG_NR_CPUS #undef CONFIG_PARAVIRT_XXL diff --git a/drivers/dax/Kconfig b/drivers/dax/Kconfig index 60cb05dce53d..cb7710c29885 100644 --- a/drivers/dax/Kconfig +++ b/drivers/dax/Kconfig @@ -8,7 +8,7 @@ if DAX config DEV_DAX tristate "Device DAX: direct access mapping device" depends on TRANSPARENT_HUGEPAGE - select SPARSEMEM_VMEMMAP_OPTIMIZATION if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP && SPARSEMEM_VMEMMAP + select SPARSEMEM_VMEMMAP_OPTIMIZATION_ENABLE if ARCH_WANT_OPTIMIZE_DAX_VMEMMAP && SPARSEMEM_VMEMMAP help Support raw access to differentiated (persistence, bandwidth, latency...) memory via an mmap(2) capable character diff --git a/fs/Kconfig b/fs/Kconfig index 496cfa2379e5..ab3937abe07f 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -278,7 +278,7 @@ config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE depends on ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP depends on SPARSEMEM_VMEMMAP - select SPARSEMEM_VMEMMAP_OPTIMIZATION + select SPARSEMEM_VMEMMAP_OPTIMIZATION_ENABLE config HUGETLB_PMD_PAGE_TABLE_SHARING def_bool HUGETLB_PAGE diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a308e2c23b82..9a7cd7575f3a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -546,6 +546,7 @@ FOLIO_MATCH(flags, _flags_3); FOLIO_MATCH(compound_info, _head_3); #undef FOLIO_MATCH +#ifndef __GENERATING_BOUNDS_H /** * struct ptdesc - Memory descriptor for page tables. * @pt_flags: enum pt_flags plus zone/node/section. @@ -1990,5 +1991,5 @@ static inline unsigned long mmf_init_legacy_flags(unsigned long flags) (1UL << MMF_HAS_MDWE_NO_INHERIT)); return flags & MMF_INIT_LEGACY_MASK; } - +#endif /* __GENERATING_BOUNDS_H */ #endif /* _LINUX_MM_TYPES_H */ diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index efb37f2ffec4..0d49d6e163ff 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -3,8 +3,6 @@ #define _LINUX_MMZONE_H #ifndef __ASSEMBLY__ -#ifndef __GENERATING_BOUNDS_H - #include #include #include @@ -96,33 +94,32 @@ #define MAX_FOLIO_NR_PAGES (1UL << MAX_FOLIO_ORDER) -/* - * Hugepage Vmemmap Optimization (HVO) requires struct pages of the head page to - * be naturally aligned with regard to the folio size. - * - * HVO which is only active if the size of struct page is a power of 2. - */ -#define MAX_FOLIO_VMEMMAP_ALIGN \ - (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION) && \ - is_power_of_2(sizeof(struct page)) ? \ - MAX_FOLIO_NR_PAGES * sizeof(struct page) : 0) - /* The number of vmemmap pages required by a vmemmap-optimized folio. */ #define OPTIMIZED_FOLIO_VMEMMAP_PAGES 1 #define OPTIMIZED_FOLIO_VMEMMAP_SIZE (OPTIMIZED_FOLIO_VMEMMAP_PAGES * PAGE_SIZE) #define OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES (OPTIMIZED_FOLIO_VMEMMAP_SIZE / sizeof(struct page)) #define OPTIMIZABLE_FOLIO_MIN_ORDER (ilog2(OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES) + 1) -#define __NR_OPTIMIZABLE_FOLIO_ORDERS (MAX_FOLIO_ORDER - OPTIMIZABLE_FOLIO_MIN_ORDER + 1) -#define NR_OPTIMIZABLE_FOLIO_ORDERS \ - ((__NR_OPTIMIZABLE_FOLIO_ORDERS > 0 && \ - IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) ? __NR_OPTIMIZABLE_FOLIO_ORDERS : 0) +#ifdef SPARSEMEM_VMEMMAP_OPTIMIZATION +/* + * Hugepage Vmemmap Optimization (HVO) requires the struct page of the head page + * to be naturally aligned with regard to the vmemmap size of the maximal folio. + */ +#define MAX_FOLIO_VMEMMAP_ALIGN (MAX_FOLIO_NR_PAGES * sizeof(struct page)) +#define NR_OPTIMIZABLE_FOLIO_ORDERS (MAX_FOLIO_ORDER - OPTIMIZABLE_FOLIO_MIN_ORDER + 1) +#else +#define MAX_FOLIO_VMEMMAP_ALIGN 0 +#define NR_OPTIMIZABLE_FOLIO_ORDERS 0 +#endif static inline bool order_vmemmap_optimizable(unsigned int order) { + if (!IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION)) + return false; return order >= OPTIMIZABLE_FOLIO_MIN_ORDER; } +#ifndef __GENERATING_BOUNDS_H enum migratetype { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, @@ -2044,7 +2041,7 @@ struct mem_section { */ struct page_ext *page_ext; #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +#ifdef SPARSEMEM_VMEMMAP_OPTIMIZATION /* * The order of compound pages in this section. Typically, the section * holds compound pages of this order; a larger compound page will span @@ -2236,7 +2233,7 @@ static inline bool pfn_section_first_valid(struct mem_section *ms, unsigned long } #endif -#ifdef CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION +#ifdef SPARSEMEM_VMEMMAP_OPTIMIZATION static inline void section_set_order(struct mem_section *section, unsigned int order) { VM_WARN_ON(section->order && order && section->order != order); @@ -2277,9 +2274,6 @@ static inline unsigned int pfn_to_section_order(unsigned long pfn) static inline bool section_vmemmap_optimizable(const struct mem_section *section) { - if (!is_power_of_2(sizeof(struct page))) - return false; - return order_vmemmap_optimizable(section_order(section)); } diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-layout.h index 760006b1c480..6a7e7f3dbb93 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -2,6 +2,7 @@ #ifndef PAGE_FLAGS_LAYOUT_H #define PAGE_FLAGS_LAYOUT_H +#ifndef __GENERATING_BOUNDS_H #include #include @@ -121,4 +122,5 @@ (NR_NON_PAGEFLAG_BITS + NR_PAGEFLAGS)) #endif +#endif /* __GENERATING_BOUNDS_H */ #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 12665b34586c..df7f6dea2e5b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -198,32 +198,12 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H -/* - * For tail pages, if the size of struct page is power-of-2 ->compound_info - * encodes the mask that converts the address of the tail page address to - * the head page address. - * - * Otherwise, ->compound_info has direct pointer to head pages. - */ -static __always_inline bool compound_info_has_mask(void) -{ - /* - * The approach with mask would work in the wider set of conditions, - * but it requires validating that struct pages are naturally aligned - * for all orders up to the MAX_FOLIO_ORDER, which can be tricky. - */ - if (!IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION)) - return false; - - return is_power_of_2(sizeof(struct page)); -} - static __always_inline unsigned long _compound_head(const struct page *page) { unsigned long info = READ_ONCE(page->compound_info); unsigned long mask; - if (!compound_info_has_mask()) { + if (!IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION)) { /* Bit 0 encodes PageTail() */ if (info & 1) return info - 1; @@ -232,8 +212,8 @@ static __always_inline unsigned long _compound_head(const struct page *page) } /* - * If compound_info_has_mask() is true the rest of the info encodes - * the mask that converts the address of the tail page to the head page. + * If HVO is enabled the rest of the info encodes the mask that converts + * the address of the tail page to the head page. * * No need to clear bit 0 in the mask as 'page' always has it clear. * @@ -257,7 +237,7 @@ static __always_inline void set_compound_head(struct page *tail, unsigned int shift; unsigned long mask; - if (!compound_info_has_mask()) { + if (!IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION)) { WRITE_ONCE(tail->compound_info, (unsigned long)head | 1); return; } diff --git a/kernel/bounds.c b/kernel/bounds.c index 02b619eb6106..9638260d67f8 100644 --- a/kernel/bounds.c +++ b/kernel/bounds.c @@ -8,6 +8,7 @@ #define __GENERATING_BOUNDS_H #define COMPILE_OFFSETS /* Include headers that define the enum constants of interest */ +#include #include #include #include @@ -30,6 +31,10 @@ int main(void) DEFINE(LRU_GEN_WIDTH, 0); DEFINE(__LRU_REFS_WIDTH, 0); #endif + if (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP_OPTIMIZATION_ENABLE) && + is_power_of_2(sizeof(struct page)) && + MAX_FOLIO_ORDER >= OPTIMIZABLE_FOLIO_MIN_ORDER) + DEFINE(SPARSEMEM_VMEMMAP_OPTIMIZATION, 1); /* End of constants */ return 0; diff --git a/mm/Kconfig b/mm/Kconfig index c85ed7d7f37d..52d9d69a95ff 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -410,7 +410,7 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. -config SPARSEMEM_VMEMMAP_OPTIMIZATION +config SPARSEMEM_VMEMMAP_OPTIMIZATION_ENABLE bool depends on SPARSEMEM_VMEMMAP diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 6f6f1740f540..1305bee1195a 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -22,6 +22,7 @@ #include "hugetlb_vmemmap.h" #include "internal.h" +#ifdef SPARSEMEM_VMEMMAP_OPTIMIZATION /** * struct vmemmap_remap_walk - walk vmemmap page table * @@ -693,3 +694,4 @@ static int __init hugetlb_vmemmap_init(void) return 0; } late_initcall(hugetlb_vmemmap_init); +#endif diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index b4d0ba27b42c..dfd48be6b231 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,7 +10,7 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include -#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +#if defined(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) && defined(SPARSEMEM_VMEMMAP_OPTIMIZATION) int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio); long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list, @@ -32,8 +32,6 @@ static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate { int size = hugetlb_vmemmap_size(h) - OPTIMIZED_FOLIO_VMEMMAP_SIZE; - if (!is_power_of_2(sizeof(struct page))) - return 0; return size > 0 ? size : 0; } #else diff --git a/mm/internal.h b/mm/internal.h index 9597a703bc73..afdae79640b5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1023,9 +1023,6 @@ static inline bool vmemmap_page_optimizable(const struct page *page) unsigned long pfn = page_to_pfn(page); unsigned long nr_pages = 1UL << pfn_to_section_order(pfn); - if (!is_power_of_2(sizeof(struct page))) - return false; - return (pfn & (nr_pages - 1)) >= OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES; } #else diff --git a/mm/sparse.c b/mm/sparse.c index bdf23709a1c7..598da1651e49 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -301,10 +301,8 @@ void __init sparse_init(void) unsigned long pnum_end, pnum_begin, map_count = 1; int nid_begin; - if (compound_info_has_mask()) { - VM_WARN_ON_ONCE(!IS_ALIGNED((unsigned long) pfn_to_page(0), - MAX_FOLIO_VMEMMAP_ALIGN)); - } + VM_WARN_ON(IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION) && + !IS_ALIGNED((unsigned long)pfn_to_page(0), MAX_FOLIO_VMEMMAP_ALIGN)); pnum_begin = first_present_section_nr(); nid_begin = sparse_early_nid(__nr_to_section(pnum_begin)); diff --git a/mm/util.c b/mm/util.c index 3cc949a0b7ed..4543f2b6ffa1 100644 --- a/mm/util.c +++ b/mm/util.c @@ -1338,7 +1338,7 @@ void snapshot_page(struct page_snapshot *ps, const struct page *page) foliop = (struct folio *)page; } else { /* See compound_head() */ - if (compound_info_has_mask()) { + if (IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION)) { unsigned long p = (unsigned long)page; foliop = (struct folio *)(p & info); From patchwork Wed May 13 13:20:23 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237872 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SlhE6UVY; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20907-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJT3jV5z1y5L for ; Wed, 13 May 2026 23:22:17 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJL2Bbwz301K; Wed, 13 May 2026 23:22:10 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102b" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678530; cv=none; b=R5DNJayKojRz8q8wCX4siVlvlaOTTHIln3/h0OofEDoE4l6n4paFCd7nrPYtEP809G5fZIRymZWrQyOZAYvD7jJakNsymHOD/e4cO+OC8/SQZ5eEOguol1BIBxQgKCEgJr9aEeHAoVx1gwiPX6mhh8DnrWL63jGOC429s+zkmdtPcU0lqhkf3q5GGZDsF6VZpPvf2it5TR9YPvNzdrQgsZhtXKcKwvgfN1YxP9Eymxkgt0x67CMPNsHQdAgozv5JT8dAqblhMMY8/A2ex5XQ8YumE8hpIIplRUqr6mpzxPE0lYasNWgz/yrF5OuPKOrG03J7BWcZZv1QWUsBvNMCkQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678530; c=relaxed/relaxed; bh=gyY9SCyAZIGNdueTrW/y87O7+cp/ViobtWpZedyU1UI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RB4P6sT708REcN8QmV7NbXbpohnEKBtS0a4hpnL98rCzZyRT4+1ReQqycKL3VlWsYi5ZY6f/KRI5OH9CzsoPP4od1TBjfXTuPuXioq/Bv7kp+pCMuTiBsAOEpAhN7EjlsyDJ7Y2b9jpresfLDn0vOQ4iv1B4K0i45Nanxd21Xz9BowCg8DKFPg3Ixb5ddeN/kVxUTUyBAanLOe3qYCeWzZq/lWN+AKARdNeszOo2UldgNXpDsxWIQh4323zu9pjFBs9fw8qsd3kc3JwYxbMSNTOGSqQLlakzg/1sEuwptvqnDdhOa8g5Jew+FXlcuEUsPXiGutK6/Xum+7hewitIbg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SlhE6UVY; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102b; helo=mail-pj1-x102b.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SlhE6UVY; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102b; helo=mail-pj1-x102b.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJK4Slpz2yVP for ; Wed, 13 May 2026 23:22:09 +1000 (AEST) Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-3653cb9c6f8so5431536a91.0 for ; Wed, 13 May 2026 06:22:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678528; x=1779283328; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gyY9SCyAZIGNdueTrW/y87O7+cp/ViobtWpZedyU1UI=; b=SlhE6UVYLE7iiIygKqRDr//e4SXrxMwnvOP51psB9vhe5uZdi5Plvu2LYtsdvTcw51 iNSTVcD9Yn37pF4ZslwSMb/Pnw8fYGG+szrcWl/FlkIEKYBCsgHH9r/7rFmu8VEMuz8L 7DpJUn3feFZS6UKXJh8dXPpdlcsHhJ98cQ38ZPWYXTQ5/g4BBtyUtohAzPAox4Q3YeX2 XwGZMbEaD6rFnvT+ltp4U8l60oDzGnnb7jHLibkUteCUFlj2JEr0nVnUhhSPfuGz/akX SAz7xOoeAD/xbTSfFVlRwQ1e2+CXhDUQSKqUPptyWH5rP9c/z+8sNFPL8KcItOQrPg61 cdAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678528; x=1779283328; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=gyY9SCyAZIGNdueTrW/y87O7+cp/ViobtWpZedyU1UI=; b=IuWGglv26Wnlx2i9WOkTbI949ovpp4U9jlz0/o9P2OUXhPkD+X7tcSODdkSQj38Q/G pke4JYfVY9RyNovZbz+t6wYcy1/tuXpyEphbnpKyDBHMZqvxXTm6b7f8fxqt6O6VKkBT DII79pUIL23aXRAqrW00mMqgwsF31oGRlvQFsiBLpBy68A253A9sZ4q9XY1lH5jeRKvr 2G0P6bOW1VTJ+wUEYVjpBPK2xhHr7aEMauseQg7qdfTbt3bB5Wnv3U6+o2u8ah4Mek4m y9PcusNC7f1oWtYj5uIw/IuWpQh9bJgAl2H1qX8+0P+2RSCFd0I0A5HAI4hGLma+tCni gytA== X-Forwarded-Encrypted: i=1; AFNElJ/YsUFlHm0b978zHjdAPqKhlMfb+9j1YbEprh2NL7FRbQ9EVk3xoOk2V/9vyGf5D44QFJuh9ahupilSj1I=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxTUXJUx2q8rA+iZ3QeYGF+2qpzXYIW210aJukvlSB3lnsdc0a7 L1+HHmNH1VCx3EzICmiS1BinR9de/Mak0ooJOQPyEuGg+L0QA+MfpFgebS+usy/oBbk= X-Gm-Gg: Acq92OFTPStnEL1ysir9EnX2pTUgQ+nzRaDi9whFKO7kNYvaX5wODkCc0zRwNHqASeM Ket7MPgKLjjdDO0W6gc9+UHuO12dQKFCWuJTWD2JxOX6JTpPC2qdrAM5l34VGk92bUOp+vR8QzZ YHBfa/YzsfjYquXVEyi4ZUVntkviNXuWnaSxScp7XUCoU2DeIfE8LmAiysQqXN+yciSpx6nqy6N qmVkWFShKx76LwXo1Lqu8zHuR9JOyd5RZGa5osQE1xIUjvEX9mOKxh5wEM7us/E3kbtdzBNOm9/ IgnPaoGlHhO6Q0wdj9TLJj/Av0qDZea4TEGgsxC7DFafWBnM0AA1jI6nMEkFoh32Y9NK7hJDBam AzETtEAusEp/akIZ1iB2OrTN13WtKbKwGLlvRg3kiB10LxIcp+CsbX5QkzVb852wcoX02c4BSG2 5a/mX68mrTkZ+8WAchuuJmkO5/mGQykMaWdnlj+e+gs7o/05Ii0Wb7Flqn7Ngs X-Received: by 2002:a17:90b:3bce:b0:35e:3b08:8fc4 with SMTP id 98e67ed59e1d1-368f398c9d1mr3681030a91.5.1778678527413; Wed, 13 May 2026 06:22:07 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.01 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:06 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 58/69] mm/hugetlb: Make HVO optimizable checks depend on generic logic Date: Wed, 13 May 2026 21:20:23 +0800 Message-ID: <20260513132044.41690-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Make hugetlb_vmemmap_optimizable() reuse the generic order_vmemmap_optimizable() logic, and switch hugetlb boolean call sites to use the dedicated helper directly. This keeps HugeTLB-specific optimizable checks aligned with the generic vmemmap optimization rules and avoids open-coding the size-based test. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 2 +- mm/hugetlb.c | 4 ++-- mm/hugetlb_vmemmap.h | 43 ++++++++++++++++++++--------------------- 3 files changed, 24 insertions(+), 25 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 82dbb9ebead8..2383adb22ce1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -778,7 +778,7 @@ static inline unsigned long huge_page_mask(struct hstate *h) return h->mask; } -static inline unsigned int huge_page_order(struct hstate *h) +static inline unsigned int huge_page_order(const struct hstate *h) { return h->order; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 54ef7d12c585..bd136fc6aec0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3351,7 +3351,7 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, &node_states[N_MEMORY], NULL); if (!folio && !list_empty(&folio_list) && - hugetlb_vmemmap_optimizable_size(h)) { + hugetlb_vmemmap_optimizable(h)) { prep_and_add_allocated_folios(h, &folio_list); INIT_LIST_HEAD(&folio_list); folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, nid, @@ -3420,7 +3420,7 @@ static void __init hugetlb_pages_alloc_boot_node(unsigned long start, unsigned l for (i = 0; i < num; ++i) { struct folio *folio; - if (hugetlb_vmemmap_optimizable_size(h) && + if (hugetlb_vmemmap_optimizable(h) && (si_mem_available() == 0) && !list_empty(&folio_list)) { prep_and_add_allocated_folios(h, &folio_list); INIT_LIST_HEAD(&folio_list); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index dfd48be6b231..1765f8274220 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -18,22 +18,6 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m); - -static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) -{ - return pages_per_huge_page(h) * sizeof(struct page); -} - -/* - * Return how many vmemmap size associated with a HugeTLB page that can be - * optimized and can be freed to the buddy allocator. - */ -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) -{ - int size = hugetlb_vmemmap_size(h) - OPTIMIZED_FOLIO_VMEMMAP_SIZE; - - return size > 0 ? size : 0; -} #else static inline int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio) { @@ -56,11 +40,6 @@ static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list { } -static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) -{ - return 0; -} - static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_page *m) { } @@ -68,6 +47,26 @@ static inline void hugetlb_vmemmap_optimize_bootmem_page(struct huge_bootmem_pag static inline bool hugetlb_vmemmap_optimizable(const struct hstate *h) { - return hugetlb_vmemmap_optimizable_size(h) != 0; + if (!IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP)) + return false; + + return order_vmemmap_optimizable(huge_page_order(h)); +} + +static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) +{ + return pages_per_huge_page(h) * sizeof(struct page); +} + +/* + * Return the size of the vmemmap area associated with a HugeTLB page + * that can be optimized. + */ +static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) +{ + if (!hugetlb_vmemmap_optimizable(h)) + return 0; + + return hugetlb_vmemmap_size(h) - OPTIMIZED_FOLIO_VMEMMAP_SIZE; } #endif /* _LINUX_HUGETLB_VMEMMAP_H */ From patchwork Wed May 13 13:20:24 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237873 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=K0NpTwA4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20908-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJd0h0hz1y5L for ; Wed, 13 May 2026 23:22:25 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJS4F07z3069; Wed, 13 May 2026 23:22:16 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102a" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678536; cv=none; b=TKIHHL+yxUC1TZT1nihBFf0fj9c2cA//wV4AxKBAwdkCQNApoUbY2nDIxzM4FkOOWeZ8aQs6gIG5BemAu7bBfU5c3dQp/nJWShgUeuRO1WTFnbt9gw+yKURQbA36dBZt2AVQ3zSaORvW8U9Tr0393bzLLRgqfzZlRLbWd7g8TYXTWvuBYXQ2PxX+9s/2XzuXYH3bri9yeV/uMTYBDNek2d+alIFq0mjh74tOs2htCaC8f8ZMGCrYmKI4CFgGPmthzdVn8y4AL/Pqc82yKZKKSZt6diM3Y3aTFSBQadbjuu4f5xPXUwfM8lej8+4/kTOmdDDFCI5FpVJboL/eypU6xQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678536; c=relaxed/relaxed; bh=VZDeIIjrnNpMULL+G/A/MulBde5nkuD9M1+mQAv5l6w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ju/SQxEGsjWNYWbTr64cCm4LvDqzbW16N0bzYojBo+oBfTUmdpQwWX/hdwghDkWx6uOrWFgEJPAcQLGcdDtdpEaJVkVW+Ba+0gn3NVZ70Rl22JaOQ6bJkhw2qgQLmz3WIieOl8MnWad+2GYc0R8BRv7Fc8oNc7u84wSL09a0BuVjnEA44swdSoR90e+A7/8wMB0KNnAijUH00buzRSViEIjT6Z1NDN6VH/zeGoD30miGlI5PLK5QgxftqcA8wxlHSuAYIlk3pCvX77UyADm9KayIm46bUERb06BrY0qGHI3ucEKRPdinBlsF3u/dubnDUVZOFnehGNYJzG7CEkivfw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=K0NpTwA4; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=K0NpTwA4; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJR6M2Bz2yVP for ; Wed, 13 May 2026 23:22:15 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-367c2a39fcfso2512271a91.3 for ; Wed, 13 May 2026 06:22:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678534; x=1779283334; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VZDeIIjrnNpMULL+G/A/MulBde5nkuD9M1+mQAv5l6w=; b=K0NpTwA4At24Ix47hm3dvI/JXZDZ64efpU9C5M9LVP5x4D6G02t1LylAv4Q+uYY//t l9m2dIg2kl3JLE2z3Jw0Uib9mKQDzWYKnUHlVnjGwLGvaagcR2sWe/sHJhJrqAtPa4+h YYDFHWDKVRfl+bwycLcUAgpvoWE19GfotO7VTuudOIXJ20rTxrMqrMBfkfj1pyGPbRGy MZ/IW0MDuXVQ9VQBy5ixzTGU0aATGe7R2jv7FWzquxEB0jrl9hYDNtpUeqG0v+VYVTJp KLslnUPWSvX7FkwjTscipw56t1j9FH8lxCcysqJu8vYqJ69wc7ciFxC6tX79AAQXPvBf jtYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678534; x=1779283334; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=VZDeIIjrnNpMULL+G/A/MulBde5nkuD9M1+mQAv5l6w=; b=mzpSADGTPubfWQD80bX0uOTwzlvY2V6lNgLAj/pjRqgFT+fzmDBEk21a4zCKCHsOyT MNVMODh5GDxRHEKAfUbT3DT/UCNJIdXcw3uzufc6NnSwR0lNjJTqIpkSqQjzvQVvovxF 70rWOvUw3Z3ftDwQ+SHa19SUg22U+jn4cmE71RoVMatUaSaoYri/VcHDDnkrwLk3nTh2 0YdZnz+YOWzmUCEljcuEIuwyMH3P/UrPVAKQR+Yb0AT/kDfHnshjd5pd1k7irRtYHABa bXFEenaZkSzOZpwE9ganhFHIycaIo72y62qhPpcpblzW2tu+Gf0VuoxLIfKP18fczteI sKKA== X-Forwarded-Encrypted: i=1; AFNElJ83X64b+62zVHEEKxDM/DTXCtUAAdBcmS+Dw8hWS/+fjV2J6D1sz6vmgafWm+UIzfQ8CiAXQvdvi+pzf9Q=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yxb24ikApJC0weyXA+Za/AtvL5DfMwPKJ/BsycSY+khajNCjmz8 RrXSkyKIUFobBdze5Y3Ujmms5rQeySa/HYpwoB+W60eDWhtgN2YQi23XPPHCNLhz0dg= X-Gm-Gg: Acq92OFeq6vbNTobx1FWgSivWJiFi7ZeBy2akr9kMMHo1jhqoqYmkbrhvy/0QMYrJNH L3E3Hyq5Frz0b1bCWTpSGI50ZtwZ+POHXwGA+ifoS+OCIbdg7R2H6QBrxmQAgu+MB5IIcdwWNlP 18Gwmqx0Q38e0Q4ul3OfJiHwlc1P/lmvDQ9bH1StrMQ9fzr7x6dBIjD9MzLI9DGT7SWynC2yA1k AEUzNZow5jHG3/0fYW39BOc5iCvMrMBn8eIjzoydGjLXQJksIctJZSjSEo5oT4Y8qVp1ikTt55d fMdVRUXhXLcAdzu4V5aOkK3ZKGhHJPW6oxe3/tQgXAZTRczl58Y6JJ25pSvJwMe/Yaz9sPAsyrm XNn8uiL4FlIjlSqaIIw+xVQOF+UFwM5fg6/i068NqlPjj9B7n34Aaer/vTfw5T2/K0vSVnsYS4R 7iHqHiPeXjtv9wpKXt5GW3HCeZDNCK2q1hV22VGS/AyqJq/iFVxU/avFKVowYo X-Received: by 2002:a17:90b:43:b0:368:1064:62f7 with SMTP id 98e67ed59e1d1-368f77f6824mr3031213a91.6.1778678533741; Wed, 13 May 2026 06:22:13 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:13 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 59/69] mm/sparse-vmemmap: Localize init_compound_tail() Date: Wed, 13 May 2026 21:20:24 +0800 Message-ID: <20260513132044.41690-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org init_compound_tail() is only used in mm/sparse-vmemmap.c, so there is no need to keep it in mm/internal.h. The helper is only used for SPARSEMEM_VMEMMAP_OPTIMIZATION, where passing NULL as the compound head is valid. Keeping it visible outside makes that usage look more generally applicable than it really is, which increases the chance of misuse. Move it into mm/sparse-vmemmap.c so the helper stays tied to the only context where its NULL head argument is valid. No functional change intended. Signed-off-by: Muchun Song --- mm/internal.h | 9 --------- mm/sparse-vmemmap.c | 12 +++++++++++- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index afdae79640b5..aff7cebb1da4 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -907,15 +907,6 @@ static inline void prep_compound_tail(struct page *tail, set_page_private(tail, 0); } -static inline void init_compound_tail(struct page *tail, - const struct page *head, unsigned int order, struct zone *zone) -{ - atomic_set(&tail->_mapcount, -1); - set_page_node(tail, zone_to_nid(zone)); - set_page_zone(tail, zone_idx(zone)); - prep_compound_tail(tail, head, order); -} - void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern bool free_pages_prepare(struct page *page, unsigned int order); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 667424aadd6b..38777e4952e1 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -265,6 +265,16 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, return 0; } +static void init_compound_tail(struct page *page, unsigned int order, struct zone *zone) +{ + BUILD_BUG_ON(!IS_ENABLED(SPARSEMEM_VMEMMAP_OPTIMIZATION)); + + atomic_set(&page->_mapcount, -1); + set_page_node(page, zone_to_nid(zone)); + set_page_zone(page, zone_idx(zone)); + prep_compound_tail(page, NULL, order); +} + struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zone) { void *addr; @@ -286,7 +296,7 @@ struct page __ref *vmemmap_shared_tail_page(unsigned int order, struct zone *zon page = (struct page *)addr + i; if (zone_is_zone_device(zone)) __SetPageReserved(page); - init_compound_tail(page, NULL, order, zone); + init_compound_tail(page, order, zone); } page = virt_to_page(addr); From patchwork Wed May 13 13:20:25 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237874 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lt+a5pMq; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20909-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJm3m9jz1y5L for ; Wed, 13 May 2026 23:22:32 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJZ5xCRz30GV; Wed, 13 May 2026 23:22:22 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1036" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678542; cv=none; b=kTiXjjyIuRodQuf+f39yGKgIdeC2jc9fUp+kWpKXm4yTfoMoCW66o4a+TOyBCr57LeHrEMJdXPkNjBq5z4x6oMlx0zp28nFbt5arXXcP8Z2DQLqfBo8+dFPCigyGHGJ2DGwViw6VexE/DidD7Dztf3qRBR8lo+G/oSNajgC9EYiXe7Qww/ZQbmWpAbgDexVz6br6W+aF7E+KOFxD+FD0g/uDatgfYf1/UlMS96bRDnuQUmJjfvb0ACcHDSFOqz/32AWwgoitefM/HeMTeIPZrOsrzWlrj2n0gKWjeEK8N5oAjXlwrifH97hDFhqfz0Ib/ZKd1KLwIQkVM8L/yq7l6A== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678542; c=relaxed/relaxed; bh=CiUCqo7mN96xuEU3BEXI0sREbdNGZzXWZJQTkejfapo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N2XXXiTyJpJSo+8mzP4QDkMGA5GsrguzXk45R2ddrzkxkZXSwUKeQOb3dvlpq2KRrQF+QaFnCA7e0Py+WtiXvryg9UUl6Jn+qrbGpa8HLsUxUJ5Hux/TwqqB1L7VQCn7ePxquaFRStCpjN0CTAgduiv1wXmkB71S/ixbMWRLULp43oArzJi5KfS4Fc/kEFT+3y0ymO1GoBkg7VOFiR/FZhUrlNLigf31UL66Q22xjAei9a9qhZopAVC/MvCRNCAkEkDT3BXi+tjJBv47WcpPyJ9HasmJMFKQD4KDAzecdk8tJlMuz7h0N5orHgZ+rsk+8DPt9b3y/o5NwwON8tRvoA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lt+a5pMq; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=lt+a5pMq; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJZ0wPRz30DV for ; Wed, 13 May 2026 23:22:22 +1000 (AEST) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-365cae89bf5so2958675a91.3 for ; Wed, 13 May 2026 06:22:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678540; x=1779283340; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CiUCqo7mN96xuEU3BEXI0sREbdNGZzXWZJQTkejfapo=; b=lt+a5pMqUimAO45sd5U1AeGYdL43Qd67QDMDkVPY7VhA4TQ66+vRwdIy7U97t5/ZYZ rrHRVmqA5HSgrCTGpnZ7Z1HLBj7OEubcxEpGe7fwN0Pvx68kQJt/ZpLQCisjV3uQynoB 4ge8VO+f/Gv/uUdQxCcPKGRbqO7aOxIuA6H2DIGwqeGDzIFmbohbbstUTnFXqh5B/yNS rFr/bIMaNAkeZ6f0bE3Gjfe0mkdLpbAOjo4LsoZiuGoFPVDFj5r4dwN9sOzAJ40Q8wxN yrTWAEu5ZsLirKIGq7BqD1sfYXNctEwtfNiGeAdtTAjyb3hjPLt4mN5ZHe5e1nHTRq/2 O40Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678540; x=1779283340; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=CiUCqo7mN96xuEU3BEXI0sREbdNGZzXWZJQTkejfapo=; b=U8EHlNkVWCjfJs2hR+gPpx9gAWuL8i2angjVPTna+6exTQEh553CsxY9TI2DSXjoqy DHmiSGJdhBYPUph7ud1we7kUGA4+vkknzpzt8+J3a84opfTD6YVEFQJoEOOl6ODgc398 FcbixnmleSVIyYBU0lYDOTsogJVO8EG17EHUWnaJvX+yqRUgYJj97TFQqUIIKLO5WnWA IqcvhQhrurQKYIwtWJ4mdqT4azV31LHxsCQx5U6vnJQfUTkGJqNdBstdPRGbCsZO7hbB nM21NKECOsM64at7zI0VlGqj4ZNgVJoRjCxxOO+Ej3kerCV8LQHOxs9CptnyJ0LFsrH0 vagg== X-Forwarded-Encrypted: i=1; AFNElJ/8n8q2Al9kPi99hV2LTtYithZWWcL68TisEMIy52doaZk4aCgBAMp6bAOEiOB2FwWlUdYWq+Mh366lx2k=@lists.ozlabs.org X-Gm-Message-State: AOJu0YwxsznX0M4WPYYvqRm++aVIKjQFVTJ7ArUipeeBSFu4+bhUJT/6 x/mQIG6LJXYnNaTobeYbyddvcfFOcNVVjVWYOZnNALoQtEE3t0zc3n4KMNK0heKloFA= X-Gm-Gg: Acq92OGMcxFb0UGPKUtPgpZCPmvTsOo+zwX/1sKOHJzu9pro+Z3Hw7JAjcsinjCgStN wkbbwozDYUWm5uiLtD/g5iUWEj7PHf8ksMAsPAkNVhyYaq1tzM38Hgi3YNqCzcq/qTl0f+OIqjJ fu/MfzGLtTeeNNIGoU4SBM+9Pr3mdL6IXbH8C6m8guRlOKlKMQ4M9M5gaiwfkU6W3pFjJGkIons /uHI0IX2exrWjdQtutJbGExanbbN3azAxsGRclOiYvIGXpiu9QIA2UfOyNYYi6qCPDk7w5ItMp3 Nn/w8pe1IYrzfwqq7YMNqpIhxCE6XIo756+USUTvnEfHGKo5s5M4KBXWI9EBOUCBDbVjdUfJ4Lk GTp6PdWDhFVxhJ0Ch86+2w49MCSnXdKE17SdFi3B3yq12WGi1pESVkSrwhbWNDFHU5j90qx11ML FhUxlOxzq/uQ6MuIMWRoohrc67TZRqF5Wh3ncxuwRtPm6cClNBzvZpwuafPRUo X-Received: by 2002:a17:90b:4f92:b0:368:f179:ba07 with SMTP id 98e67ed59e1d1-368f3aa9f3cmr3656850a91.9.1778678540011; Wed, 13 May 2026 06:22:20 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:19 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 60/69] mm/mm_init: Check zone consistency on optimized vmemmap sections Date: Wed, 13 May 2026 21:20:25 +0800 Message-ID: <20260513132044.41690-14-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org For vmemmap-optimized sections, the shared tail struct pages are reused across compound pages and should already carry the expected zone and node. Warn in __init_single_page() if such a shared tail page is seen with a different zone or node, which would indicate inconsistent initialization. Signed-off-by: Muchun Song --- mm/mm_init.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mm_init.c b/mm/mm_init.c index 4ea39392993b..95422e92ede8 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -605,6 +605,9 @@ void __meminit __init_single_page(struct page *page, unsigned long pfn, if (!is_highmem_idx(zone)) set_page_address(page, __va(pfn << PAGE_SHIFT)); #endif + VM_WARN_ON_ONCE(order_vmemmap_optimizable(pfn_to_section_order(pfn)) && + page_zone_id(page + OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES) != + page_zone_id(page)); } #ifdef CONFIG_NUMA From patchwork Wed May 13 13:20:26 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237875 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=gy0RaltD; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20910-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvJw04kGz1y5L for ; Wed, 13 May 2026 23:22:40 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJj39cDz30TM; Wed, 13 May 2026 23:22:29 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102c" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678549; cv=none; b=V3XDQ//fpI/0vC6F3bxE/tHVnwmuGvjMw9palwJnk6igoCwUzYfZL85TJJTRj2ONJJMrqCjCRzB7jKQ6wciWRm8PBPmQawekkXgqWhMxfcN+arSbdgoQO4UdaiFs330Alj5kBNYISmlhyVVtUXgU2e/eiqvljsecVEtilqk55TRhCBB49I35zo3xfAk+AiE1NQ3PPoOA97c6B4XMSlSft1S6IjfdqePH5OjwFQ5AN+5rfmmAMTyuK1HiJi9/Iaqnws+YKAwKdCiNtskbAEddymZemJ3bdTP9RCj3DURfpmGj1LWmc0Gu4D52RVDGVt5LrTq0jkw4AjAA0FVSzq2ufA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678549; c=relaxed/relaxed; bh=5LFbqiG/RD0CCzo7+/AP1zWQqzbysWKSZNji4RwzSyY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iGsacAI+SoFu7OaCPJ1Gqr/P0zmxLUgTm4UdU1tWjAjTkZaMrkjU2lzLcKSkMC1M6l/iP1/p1bjGVrZzTLUXtz+FFluBPnHuifw0rBFuxjqw5iXcnhjjEbUpHoFH34eAca/I2zXB0Wx55wSjUT5WomgimfbRw7rRoHx+/rFPz6g9Vuh/L2yEEcDGAPw3BFCEVeFIvLgTR/DWGHTkqX/iIuN/xe6n+v8pbi6p0ScuiBE9ayzxXWq3W0yKw0FYuqyiw3r+rvLTjSD0H6i3ea+VLbxkoRLrqdVCSc0zgTEjS+69KelZ0Ciq1FAR1M8TAk3UvXHW271JuYwqlu4CNdZteQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=gy0RaltD; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102c; helo=mail-pj1-x102c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=gy0RaltD; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102c; helo=mail-pj1-x102c.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJh4ZPKz30DV for ; Wed, 13 May 2026 23:22:28 +1000 (AEST) Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-367c2a39fcfso2512613a91.3 for ; Wed, 13 May 2026 06:22:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678547; x=1779283347; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5LFbqiG/RD0CCzo7+/AP1zWQqzbysWKSZNji4RwzSyY=; b=gy0RaltDCqMJ/8gNmEu/INTkeYEMluy1FM1CckKIkN7mKjHSdOnzosMkmpbo37TEWp d2J7C3OPiLb6Sykjogtj7LlUO0PFCdYmNRTivxV49x6kMh3z5HPtI/DoWopxb4vq1UKn LzWZNHGkP53DZWGMHUFxWCsueAkxUcwkoMvmCSsTcI6bVV1ezwr0EqJfUzmxVEW/2I8T K1GseGcIOgSirfyYTZP8s2Ax9oOyIPrH+mBO/sH3s1IjYIKCcjRPFj6WJbygeo+1phCm bp4BmQc5QT95AHKi0NW1CQQLodMQ2sWNtn7YTbl4ei+3O6qFYRnXPBsTArXmI0glBSzW 1psA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678547; x=1779283347; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5LFbqiG/RD0CCzo7+/AP1zWQqzbysWKSZNji4RwzSyY=; b=ZyJ/oL3nsS7KFbbgc+P3kWUIepypj+SW777vcIWldnkjFeE3SlH/rkPIP8Kbqk4Qsy 1UrXA0haIWFC9jBQfH2gqRyMmDe4cRmOisaHXzvFIo+z23YqOcbdrYmVwJEuOpbxfigA DgEpCnRq6l2jiVE3tyHcZ3YR8M6JYhLrNCwS6Pq0SNPFFzooDwlaWdcDo9aI1i6BEbdu I1LZihsN4/O83s/r8WYAmScA7ZLu9VrjSp7x+VSbaZGN3bDYm23jE4PZ0AE4cJzkE9+J 4p7I0JbH52Qm2aMk7uQ0wGlP2yWo/0LBKcrqi4sjrJL5oNwc2gmL3f1R9+zPDKtKGJ94 ftfw== X-Forwarded-Encrypted: i=1; AFNElJ9U5/HaYEqgH3gxoFzct2JJ2MPmT8dRrclHT9/K3/Y371oJmaaQA/mkVibbN80Hj9oQ9s0a16K+0xg0Cto=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yxr98vhupoOnnPd60SnaTNSU+85a/q1h+vU0PdHir5U29s7NWbJ IUjX1st4QG4OcclzxbXYsgDa2oM5d+NSPhHmyTfoyJ+/C1Yi2qANt/pvycRWB7/8FFw= X-Gm-Gg: Acq92OE23ldSmg6xgeTvaRF6FiGZD9VXi1FKjUF5XDGUzE7oYLbTmrhLyJJcFMgyZ2H M7g5V1k1oCMPLNZuDOUFiWrco+gERTkDv/FCwcSt3MUbtPcR1lfk7f2bUU9E38TLl7b+EEgiAVU hfuRVsiACyzrZ1QKAHmto1jda7r4Yn1pgMeMVDB9bRjVNJ3t9biUw2T7Klq4qURMafruxTLN4EC kaKde3q8k4woy280DOmgufUwAkK4yI9jWupNpwgr8Rkgg8vEnc2lyYnuqlxnf1OtMqy464+Lh0i JWrQI6WiMhRG0mc2E+KY4ra2WU1DXA1ND0+4CqAXXLwjmq8XVUQ3z+C3dTz5pyjC9AuhSXF1THO GfknJ6TZksoO7IK7w4vNyYFCMBpOGCfzjjrwFYbx/7sZ1djGBb06fzPBi9Dc/bjNWfG9iEwvByj TBi4pJamxtyChiGhnKncr+W/Y/F6kPc19kxLybjHXAgJRmgZWXvYo5bwCKhTwnJO89grl+Koo= X-Received: by 2002:a17:90a:da8b:b0:368:ddd7:abca with SMTP id 98e67ed59e1d1-368f79e8cfbmr3065574a91.26.1778678546422; Wed, 13 May 2026 06:22:26 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:25 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 61/69] mm/hugetlb: Drop boot-time HVO handling for gigantic folios Date: Wed, 13 May 2026 21:20:26 +0800 Message-ID: <20260513132044.41690-15-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HugeTLB HVO is currently supported on x86-64, riscv64, and LoongArch. On x86-64 and riscv64, gigantic HugeTLB pages are larger than the section size, so the existing section-based vmemmap optimization infrastructure is already sufficient to cover the whole folio. On LoongArch, HugeTLB HVO is supported without gigantic HugeTLB pages. Therefore, boot-time HugeTLB HVO folios can rely on the section-based vmemmap optimization infrastructure directly, without the extra bulk optimization and fallback handling. Signed-off-by: Muchun Song --- mm/hugetlb.c | 25 ++++++------------------- mm/hugetlb_vmemmap.c | 21 ++++++--------------- mm/internal.h | 25 +++++++++++++++++++++++-- mm/sparse.c | 23 ----------------------- 4 files changed, 35 insertions(+), 59 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bd136fc6aec0..3cb8fffb9e3e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3201,21 +3201,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, unsigned long flags; struct folio *folio, *tmp_f; - /* Send list for bulk vmemmap optimization processing */ - hugetlb_vmemmap_optimize_folios(h, folio_list); - list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { - if (!folio_test_hugetlb_vmemmap_optimized(folio)) { - /* - * If HVO fails, initialize all tail struct pages - * We do not worry about potential long lock hold - * time as this is early in boot and there should - * be no contention. - */ - hugetlb_folio_init_tail_vmemmap(folio, h, - OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES, - pages_per_huge_page(h)); - } hugetlb_bootmem_init_migratetype(folio, h); /* Subdivide locks to achieve better parallel performance */ spin_lock_irqsave(&hugetlb_lock, flags); @@ -3238,6 +3224,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; + unsigned long pfn = PHYS_PFN(__pa(m)); + unsigned long nr_pages = pages_per_huge_page(m->hstate); h = m->hstate; /* @@ -3251,13 +3239,12 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - hugetlb_folio_init_vmemmap(folio, h, - OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES); + hugetlb_folio_init_vmemmap(folio, h, vmemmap_nr_struct_pages(pfn, nr_pages)); init_new_hugetlb_folio(folio); - if (order_vmemmap_optimizable(pfn_to_section_order(folio_pfn(folio)))) { + if (order_vmemmap_optimizable(pfn_to_section_order(pfn))) { folio_set_hugetlb_vmemmap_optimized(folio); - section_set_order_range(folio_pfn(folio), folio_nr_pages(folio), 0); + section_set_order_range(pfn, nr_pages, 0); } if (hugetlb_early_cma(h)) @@ -3274,7 +3261,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) * (via hugetlb_bootmem_init_migratetype), so skip it here. */ if (!folio_test_hugetlb_cma(folio)) - adjust_managed_page_count(page, pages_per_huge_page(h)); + adjust_managed_page_count(page, nr_pages); cond_resched(); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 1305bee1195a..d20d2ce13906 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -599,23 +599,17 @@ static int hugetlb_vmemmap_split_folio(const struct hstate *h, struct folio *fol void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; - unsigned long nr_to_optimize = 0; LIST_HEAD(vmemmap_pages); unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH; - list_for_each_entry(folio, folio_list, lru) { - int ret; - - /* - * Bootmem gigantic folios may already be marked optimized when - * their vmemmap layout was prepared earlier, so skip them here. - */ - if (folio_test_hugetlb_vmemmap_optimized(folio)) - continue; + if (!vmemmap_should_optimize(h)) + return; - nr_to_optimize++; + if (list_empty(folio_list)) + return; - ret = hugetlb_vmemmap_split_folio(h, folio); + list_for_each_entry(folio, folio_list, lru) { + int ret = hugetlb_vmemmap_split_folio(h, folio); /* * Splitting the PMD requires allocating a page, thus let's fail @@ -627,9 +621,6 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - if (!nr_to_optimize) - return; - flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { diff --git a/mm/internal.h b/mm/internal.h index aff7cebb1da4..416afdf7b2ec 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -949,6 +949,29 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long, unsigned long, enum meminit_context, struct vmem_altmap *, int, bool); +static inline int vmemmap_nr_struct_pages(unsigned long pfn, unsigned long nr_pages) +{ + const unsigned int order = pfn_to_section_order(pfn); + const unsigned long pages_per_compound = 1UL << order; + + if (!order_vmemmap_optimizable(order)) + return nr_pages; + + if (order < PFN_SECTION_SHIFT) { + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); + return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES * nr_pages / pages_per_compound; + } + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); + /* Ensure the requested range does not cross a compound page boundary. */ + VM_WARN_ON_ONCE((pfn % pages_per_compound) + nr_pages > pages_per_compound); + + if (IS_ALIGNED(pfn, pages_per_compound)) + return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES; + + return 0; +} + /* * mm/sparse.c */ @@ -988,8 +1011,6 @@ static inline void __section_mark_present(struct mem_section *ms, ms->section_mem_map |= SECTION_MARKED_PRESENT; } -int vmemmap_nr_struct_pages(unsigned long pfn, unsigned long nr_pages); - static inline int section_nr_vmemmap_pages(unsigned long pfn, unsigned long nr_pages) { VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SUBSECTION)); diff --git a/mm/sparse.c b/mm/sparse.c index 598da1651e49..21a0eb636fea 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -236,29 +236,6 @@ void __weak __meminit vmemmap_populate_print_last(void) { } -int __meminit vmemmap_nr_struct_pages(unsigned long pfn, unsigned long nr_pages) -{ - const unsigned int order = pfn_to_section_order(pfn); - const unsigned long pages_per_compound = 1UL << order; - - if (!order_vmemmap_optimizable(order)) - return nr_pages; - - if (order < PFN_SECTION_SHIFT) { - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, pages_per_compound)); - return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES * nr_pages / pages_per_compound; - } - - VM_WARN_ON_ONCE(!IS_ALIGNED(pfn | nr_pages, PAGES_PER_SECTION)); - /* Ensure the requested range does not cross a compound page boundary. */ - VM_WARN_ON_ONCE((pfn % pages_per_compound) + nr_pages > pages_per_compound); - - if (IS_ALIGNED(pfn, pages_per_compound)) - return OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES; - - return 0; -} - /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_end) * And number of present sections in this node is map_count. From patchwork Wed May 13 13:20:27 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237876 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VHrax/ao; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20911-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvK40Lp7z1y5L for ; Wed, 13 May 2026 23:22:47 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJq4Nrwz30DV; Wed, 13 May 2026 23:22:35 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1033" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678555; cv=none; b=BlPL8MTojKoqAhuY9yZJvITGUGsOQDU7L9rLfLn2AlGlrcY2QZNHR2eug+Pek9ZZHCVKnL5x51cTS4uc4Y2TxoiDqJhXLbAiROk6YjX4MZNpeHeEU2I6j8BpO/iGdfYCQEffWZZO8icxinEtlMRPuYw5M67x4ADKFp8dtcvDLvqKTh3z7+xhpCRj8YPIl/eyqhcHbJEG0N0a1SHwgrUxfYX5UDONOXBoI9vmgbRIXB/GOuCtjnLCyhsQgF4SIOwl+66FzOoZf7YWFKLypwSOQcX77AvCmjqLKwo00eNsDCzt9rbkN6r6eGogD1FsLf01415QmzHFwtarSe5Ua7xYfQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678555; c=relaxed/relaxed; bh=x2kKgj5K23Ztq0NiYkfPLR6EY3A9OUYR9zP+JIm9wZc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fBJar5HJekIX7gZ0SpxYN8HQ2bcGpBGY6TsA5uN4Q2zUwSQ7tbr5JXncmygNerkeIuSxqQWvIqUclN1eKV5MubxO7Q8mhvLzfVX+VgSeXowEkkC/GGfmaN0Cei6DuATYvxwdW3KddHM+x5gDN5H+Swq0zF8PC4P4BJZCkxYh8P30n2NHXwaVGgqWMe+eM+oK1GeHAkLedvFHzcwqy/PiDvQuonp97EGOaEdo8NXMKw0efS89gcFVG6pJNcaaktWBzRXqKsDwmNtumB9xa1eNahpF20UzToOOjyEvbv9SoQVZ6s51xTjISfwwF00p52NZQ/IAC66KY3ohBwrm0sizbA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VHrax/ao; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1033; helo=mail-pj1-x1033.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VHrax/ao; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1033; helo=mail-pj1-x1033.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJp5gGYz2yVv for ; Wed, 13 May 2026 23:22:34 +1000 (AEST) Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-366070f71adso6068680a91.2 for ; Wed, 13 May 2026 06:22:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678553; x=1779283353; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x2kKgj5K23Ztq0NiYkfPLR6EY3A9OUYR9zP+JIm9wZc=; b=VHrax/aoCwZJuHAEMqQVkljszPmv90OscpTobabuwAE2ZI8CMjVOuvNd1nABawTbiV YRkd7v/U/MpfnoIRZy5YcnX9DHGgQ/fqJxc8YV40acJKj9c0kw2YC2U9NCtHOLmUC2jy HpfF6Td+ai7e/pMLV6X92aIaCGT1GPHzXo2UKFQUZ8oyyCCO8WvWJNY+JUQNJEGfdiEJ nbFCtZrMKAMyLYpMtP0YWszuyZLKN2nihEbz7+sZjyuRi7ZSuNZTcUJMGV5ZpfDh8zD0 AdoWLLBqh65ktq/2lULPAs4Ndy1Hna0qYiuqPwLRTtk9tZOWTgQXG8eA/tk0AKWLuKj6 PyBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678553; x=1779283353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=x2kKgj5K23Ztq0NiYkfPLR6EY3A9OUYR9zP+JIm9wZc=; b=CnKkKdUj+4CIe6+TJVufbh+2Qt2bOZgh90jR4oTkepPJNAAG7/EcgzJhoH2o0CVqMZ UatEQBpNgrH/J55kUJALSQmOmIqeIS4ay18HmeBkW8FNRQ3zlKvSR3w7g77sx82drx5t TxxkDqQ5ka26itLy2gUBl8hva5qRTEj8wWkRDdFQsaNg1cT5SrlhrBnxuUWrt7spKW5A ymZPwhVECfzwnXkSBExL7SZrYWL9TDLfXjdt0cUlPdYqhxwmcmU/HhK9bQDNov/Pt7YZ 2O+VXOJdFq0tb2/CzFPGRYs/q61EdugD00Z3wEUE7AT2+V07SnmUufFJNjiu6YEREPmt Wdmw== X-Forwarded-Encrypted: i=1; AFNElJ/a2LnkWQRLa3OjwpJijGNhPXkgKhBof/anmDLhxUbjG9OL07m6E1+Rx6RwJlCzIMhPP18IPr3duwhh0ZQ=@lists.ozlabs.org X-Gm-Message-State: AOJu0YzWooik9rD/ITrIItqyWbF6X5Mf5LvTumppjNXM5W9GamjSWLwS GHUwiX0KNrGuPC/0z+lmjd85UeX8D0pdDjALa7GiMdTZEQMwMuao1PhK5bMYmIPXXAA= X-Gm-Gg: Acq92OEIMl84n3Yyn6Od4nI1TjoQQAdsrspX1+/GLpyvduyyPQ7Q9V7i1OOdZduoPqN LQnHjRNwZSymXNbzn/bqOTumkaPoY8YwUIonPcgziIuwFfLeW5pKZzpwy9ssfRlZcd/z7s7IFP4 xgmXgGPO6QcHcHxdySTFDn7OxsC6A+N00OZeGKbocrI9xxrB8qKdMARytkE4CrWvzkWo0jvYrlk MG/5UH0/yjWOTVW0hOv0SFaggw9wTrFoH59hHkMY6sMCN8HHKqYJHSfVBprswIYlgUiC7QA5kvH qG7Md3G0XVMwx42hTZh+VxwKSpZCJTbeOIPeB8TxFaSmjhtCZktmnlJfNKt+JNuXZiD4CA587az bYMeTy13iw/lYTb5bzxYoLZPHB4sb64M0BQZS9KAfewfsLIDtHc745IA18yQSWDWoQTJdZSySub inoUxY/B3isHYvHLJH21CEC5mwPlT8ZfptKr8sp4caDSYjIDq7vG7i2uhJM193 X-Received: by 2002:a17:90b:3d44:b0:365:7e4d:bcb8 with SMTP id 98e67ed59e1d1-368f77f1013mr2804387a91.1.1778678552706; Wed, 13 May 2026 06:22:32 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:32 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 62/69] mm/hugetlb: Simplify hugetlb_folio_init_vmemmap() Date: Wed, 13 May 2026 21:20:27 +0800 Message-ID: <20260513132044.41690-16-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org hugetlb_folio_init_vmemmap() currently splits the open-coded compound-page setup across two helpers even though the tail-page initialization is only used here. Fold the tail-page initialization into the main helper and pass the precomputed page metadata in from the caller. This makes the initialization flow easier to follow. Signed-off-by: Muchun Song --- mm/hugetlb.c | 50 +++++++++++++++++--------------------------------- 1 file changed, 17 insertions(+), 33 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3cb8fffb9e3e..950b0fa3bc27 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3126,33 +3126,8 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) return true; } -/* Initialize [start_page:end_page_number] tail struct pages of a hugepage */ -static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, - struct hstate *h, - unsigned long start_page_number, - unsigned long end_page_number) -{ - enum zone_type zone = folio_zonenum(folio); - int nid = folio_nid(folio); - struct page *page = folio_page(folio, start_page_number); - unsigned long head_pfn = folio_pfn(folio); - unsigned long pfn, end_pfn = head_pfn + end_page_number; - unsigned int order = huge_page_order(h); - - /* - * As we marked all tail pages with memblock_reserved_mark_noinit(), - * we must initialize them ourselves here. - */ - for (pfn = head_pfn + start_page_number; pfn < end_pfn; page++, pfn++) { - __init_single_page(page, pfn, zone, nid); - prep_compound_tail(page, &folio->page, order); - set_page_count(page, 0); - } -} - -static void __init hugetlb_folio_init_vmemmap(struct folio *folio, - struct hstate *h, - unsigned long nr_pages) +static void __init hugetlb_folio_init_vmemmap(struct page *head, unsigned long pfn, + enum zone_type zone, int nid, unsigned int order, unsigned int nr_pages) { int ret; @@ -3161,12 +3136,19 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, * walking pages twice by initializing/preparing+freezing them in the * same go. */ - __folio_clear_reserved(folio); - __folio_set_head(folio); - ret = folio_ref_freeze(folio, 1); + __ClearPageReserved(head); + ret = page_ref_freeze(head, 1); VM_BUG_ON(!ret); - hugetlb_folio_init_tail_vmemmap(folio, h, 1, nr_pages); - prep_compound_head(&folio->page, huge_page_order(h)); + + __SetPageHead(head); + for (int i = 1; i < nr_pages; i++) { + struct page *page = head + i; + + __init_single_page(page, pfn + i, zone, nid); + prep_compound_tail(page, head, order); + set_page_count(page, 0); + } + prep_compound_head(head, order); } /* @@ -3226,6 +3208,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) struct folio *folio = (void *)page; unsigned long pfn = PHYS_PFN(__pa(m)); unsigned long nr_pages = pages_per_huge_page(m->hstate); + enum zone_type zone = folio_zonenum(folio); h = m->hstate; /* @@ -3239,7 +3222,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); - hugetlb_folio_init_vmemmap(folio, h, vmemmap_nr_struct_pages(pfn, nr_pages)); + hugetlb_folio_init_vmemmap(page, pfn, zone, nid, huge_page_order(h), + vmemmap_nr_struct_pages(pfn, nr_pages)); init_new_hugetlb_folio(folio); if (order_vmemmap_optimizable(pfn_to_section_order(pfn))) { From patchwork Wed May 13 13:20:28 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237877 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=EV7KFMo5; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20912-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKC0z58z1y5L for ; Wed, 13 May 2026 23:22:55 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvJx5LW0z30gX; Wed, 13 May 2026 23:22:41 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::635" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678561; cv=none; b=Oju1Tqt7CSMS7unDRmZkBhVZ1xrG9L1ZXTN0WpokjOXIRNYBqbMVrTXI8Gyrs/MGxfr9c3fvHebXnU0LPt8KPJ3/6em7jgeU/x7PzCrLC0qCMlwG91GY8iBiBlaqfflMaO9G+CFvut2B27owonO5ca1tTpyE8mGTk2L2syFTsVvzkRJo1qqlf/E4hXN8QEO6J0YvniTdHk/UZKNfz0IBDjvccRR2SM8H/GeUFH5L+P3jlJjqf6DF5edcqWFBRYURnMOQ91tBuicPAzpKykOlnzvYpgV9PlyGHQXUb3qmyBb2/imK3azuzAA+0sKLCthCrHjua6Q03kU796BCCTkuYg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678561; c=relaxed/relaxed; bh=JweoZhL4wX+SrFX5pL1+i9XjuzkUT4z611TUJKtW0N4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fe4XkiQgE4kAEghLCSZZhJSM1d+yQOt479Hd1f0HQOnbWP4qpfT/anNc5mtUvMyvVsLPRHVo/TWN2Bz1D7hvEirhd4KT1EqisUuOiwdRMNKOHIOpCiIrPGTsJ9fFtKYvvuN6+dQXKojMQV+jj+jK/R/4vWRbWxv3JaISi8n5Ycn2PH2IDkYSo/VWBMvU5c8/2pI/N1npOWthAiOg6m4XEUf1VfTgARq07JB0YcRCWBhlWsxs2EJ7O4kLAzvD2s6M2Iro186hGe3BWXYFSX0qxzW06wWposbrpAG2k6JgV6sa4P2I6htv5f1DZBwZUXEAra/zssnZ/N7xmkdPNtgPfw== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=EV7KFMo5; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::635; helo=mail-pl1-x635.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=EV7KFMo5; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::635; helo=mail-pl1-x635.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvJw6Phdz2yVv for ; Wed, 13 May 2026 23:22:40 +1000 (AEST) Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-2babfd18435so34605705ad.1 for ; Wed, 13 May 2026 06:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678559; x=1779283359; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JweoZhL4wX+SrFX5pL1+i9XjuzkUT4z611TUJKtW0N4=; b=EV7KFMo5Uj1d89tDXHkUXFAQfyW/vq50luPlHxLemYEO2Xc88b7814utjO1PDv4Zi5 9OPubCsJciWsiYyId/Mm5I0kCbc5HYYQCeHQ0O9XcurqAtWJbLMa3UqrGWJege/lCi/3 j3ZS71+QruVPkD+nJTmCIZpdg0ZqjVvQy0UMwgQQjlCJvwdtQpcw6FbUsMfv0OTEpb3S OFAzAHJghy4DQCPn4B/UXa6rgWijWqDzl7RQMwUOeaOz5HXIilpYkQ8mIyR6HvK469+9 /3QyZ7c6y9TO8buKc6FmORTgjU3ggFvTqqMAScbapgHCznvuZ/n4VidiqhgyTg81wI6/ bObw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678559; x=1779283359; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=JweoZhL4wX+SrFX5pL1+i9XjuzkUT4z611TUJKtW0N4=; b=ljxyxjtXkbbbowDjkbuYh6j5as5aP+9pDo5DHCC6+xHpvUgYzps0E7i0sir12O1aPx MwwEKhHGuBHTWBNd0V1tNrkCsM0vNBY8fAo8SYHpYcaAhALrADynPmkN8f9C5GSEVuQR D8QA7WwDRw8P6l+wtFJpKMwB6rDKL7gC5/jsTSDh10zOMhpdB5TSa4CPsKegQjigUS4D v+t3yQC75od4oosyYNrJ6gsLSuoBmjdQVo2sf9BS1bq+QfIx8Yk1LpnLz6xAY3wPQabl 71SleHv6vbi5uO7htMKyOv/yib2A5iTjsGbuGvaaYqBbq4OpawHk2NjkgUnNnExmVbx4 +PiQ== X-Forwarded-Encrypted: i=1; AFNElJ/YRQNg13k1Td9LqBBkSBo3ZR9IkHUGvA4zGKa9IuejXy+qvbE1EUFvHdo2RbbpEMi3XciaQs+Z86cdl4g=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxmBvFcDrMy5DhBsMBLPrJs5bT8rQNtv148eQwM0OaKi9duHzf6 Ih3GfrZ7jrgqlQAFKyOcuBzD6ITh2yKPTK1rPPHexR07W5qoilv+BbOCvflB0kmyy7c= X-Gm-Gg: Acq92OFOeubJ540NHct4U+YSFH8866tIYGKnHd0vw2xkSVSw+Bq+Irfqo6+GA5P/ocx pIYo3pWLN7Dq30MVaT9BTtTi3cy5bQPjIqMgD90gAmK8GcchR5jVz6PFUvsY/wqJDS4O6vIbmAz 1jnrTgzxy0R0uMvOt8mnUq5k3OFbs8Ph3K32klw9GVz/A1NjwYUGLZMhhrLPGCc30405pynILfz nVeMDccC1ZPOHJ6QCaDJh5QcJVxPISddS3jzD1sXXAOwjU9834kekhLLVUrUkyaJsonjkxtr+Gl flLB6GBik5OsYbLgELaVcYhBC0sfLYzgUbKL+HRwFe+HYuM2zREE8116GPaLSvjkvgvktrAIaQT w11VBlRx6c15dIP+9Scphh0N94GC0tr8RUMy0NZKjv3DOPCu9y+R7qiavQmPXz2eB/TrYvlMLYC qAm64egomFT144vh1xh9+vuDtXh7qaocL57j+tgdK7wAlu8UW/hgtf+UQ7Zbjs X-Received: by 2002:a17:90a:fc44:b0:366:aba:4c86 with SMTP id 98e67ed59e1d1-368f3e62d47mr4012510a91.27.1778678558803; Wed, 13 May 2026 06:22:38 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.33 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:38 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 63/69] mm/hugetlb: Initialize the full bootmem hugepage in hugetlb code Date: Wed, 13 May 2026 21:20:28 +0800 Message-ID: <20260513132044.41690-17-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Boot-time gigantic hugepages currently leave the head struct page to the generic memmap initialization path while the HugeTLB code initializes the remaining struct pages itself. Mark the full hugepage noinit and initialize the head struct page in hugetlb_folio_init_vmemmap() as well, so the whole compound-page setup is handled in one place. This can also reduce memblock metadata overhead when many boot-time HugeTLB pages are reserved, because physically contiguous hugepages can be covered by fewer noinit regions. Signed-off-by: Muchun Song --- mm/hugetlb.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 950b0fa3bc27..10f04fa95d43 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3112,15 +3112,7 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) list_add_tail(&m->list, &huge_boot_pages[nid]); m->flags |= HUGE_BOOTMEM_ZONES_VALID; hugetlb_vmemmap_optimize_bootmem_page(m); - /* - * Only initialize the head struct page in memmap_init_reserved_pages, - * rest of the struct pages will be initialized by the HugeTLB - * subsystem itself. - * The head struct page is used to get folio information by the HugeTLB - * subsystem like zone id and node id. - */ - memblock_reserved_mark_noinit(__pa((void *)m + PAGE_SIZE), - huge_page_size(h) - PAGE_SIZE); + memblock_reserved_mark_noinit(__pa(m), huge_page_size(h)); } return true; @@ -3129,16 +3121,13 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) static void __init hugetlb_folio_init_vmemmap(struct page *head, unsigned long pfn, enum zone_type zone, int nid, unsigned int order, unsigned int nr_pages) { - int ret; - /* * This is an open-coded prep_compound_page() whereby we avoid * walking pages twice by initializing/preparing+freezing them in the * same go. */ - __ClearPageReserved(head); - ret = page_ref_freeze(head, 1); - VM_BUG_ON(!ret); + __init_single_page(head, pfn, zone, nid); + set_page_count(head, 0); __SetPageHead(head); for (int i = 1; i < nr_pages; i++) { @@ -3208,7 +3197,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) struct folio *folio = (void *)page; unsigned long pfn = PHYS_PFN(__pa(m)); unsigned long nr_pages = pages_per_huge_page(m->hstate); - enum zone_type zone = folio_zonenum(folio); + enum zone_type zone = zone_idx(pfn_to_zone(pfn, nid)); h = m->hstate; /* @@ -3220,7 +3209,6 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); - WARN_ON(folio_ref_count(folio) != 1); hugetlb_folio_init_vmemmap(page, pfn, zone, nid, huge_page_order(h), vmemmap_nr_struct_pages(pfn, nr_pages)); From patchwork Wed May 13 13:20:29 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237878 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YT4h8y2b; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20913-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKM08Zrz1y5L for ; Wed, 13 May 2026 23:23:03 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvK36ldTz3cNB; Wed, 13 May 2026 23:22:47 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102f" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678567; cv=none; b=ONczu84TVd4okF/xeNV2pIqne+8IIRxMjxA5vXUE3WIH8iSRSIemflLHTeFRcMzIiFMIOavV/HWpP6s5sshWza5EcVug/Hu3ipeaPeiEmnrCzB77SCE93vUy/K04iDM9x8GO0V29dmQZdA3Qi3IRDwPippbRE7w0wHwUP7EdHzlWHCQ/9ya/prONO/ZLrd6RG85d6AKtHkaRe+Ig6RRVMD05kRxLeT/1vMN5xchNly/FXBpoAtllHqGp/EoMY9t9GHdCUC58wpGrxR6HM4Yy6fkpQvyG9oe41yQpdWCgvbThqRN3GBtYfc/9ef7+nftJ+Y6JuVCdGGauAgBk8YeNbA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678567; c=relaxed/relaxed; bh=flkX+tolfoJMepk6tCll9FGS1Em3GZPOVlthey3SFjQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h5smcyX7tIZVI1C4XsDNIB8tqZJw31As10/G8B8OIv8/IP6+0hEPtWlLQukm5OiisUKtVMKqdwdfeIPlYfdFv/j/vNLDdIMOBB5tzLEuW3+uIvZNFiA5y5nRrwhMejsknVRVAj6H7rWWCDAMqsnBlrg+L/nckSG9Qris/ldy/8arzOlWkuZprYbKrsce7Q0o1q5zdJGzowkb2GA4oiMpfCJXmJVLp19KPBah2LdDk74LVZE0sSIMvPBglP68IDeQ4avVbbVllJUrDxcvzKOYjcbz+q1yScT+qVUPvKkNqcVTzfcy0p0RPNrXAlpIzZCp55AInal5runqYEy9aCVv9A== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YT4h8y2b; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102f; helo=mail-pj1-x102f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=YT4h8y2b; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102f; helo=mail-pj1-x102f.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvK31FKXz2yXj for ; Wed, 13 May 2026 23:22:47 +1000 (AEST) Received: by mail-pj1-x102f.google.com with SMTP id 98e67ed59e1d1-366375c43c2so3972348a91.2 for ; Wed, 13 May 2026 06:22:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678565; x=1779283365; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=flkX+tolfoJMepk6tCll9FGS1Em3GZPOVlthey3SFjQ=; b=YT4h8y2btbEiOpw1NHTDHy+IpoB+sJfCp1Xt1inAWDUsV3iLz0VpFm8CZmhH4WnBpP xlHQS1N64xHlQchojbpo5bgLTJx9gwBq2VrpmK715rs+VmGeDQuyw7ucmfwn3w5YcHpT KB34DxFXHoaUSwuSl/B+9HPgAyW7PfU3pEs11P3XG26hG15uFBBFUR5vJM0m6jJzlh52 vC15SeU1OM/Ol8gQ1oTh28T+jZSUUx/cvMnFYouXfxYBIOA3JPxNlSrzCZ+YCYRms6f/ v2FRSyKY+4GOX2E2ykQHE9qQpqEU9Mg8H6bfW8/ebXaySTuUdrKIkrrcppK/Yx75qa3/ O49Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678565; x=1779283365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=flkX+tolfoJMepk6tCll9FGS1Em3GZPOVlthey3SFjQ=; b=DFPUVN15tYtRSdoMOU6N1Qk/ehyV7g7SFXop3VCAraM8IMRNgGXWazMpX1vZjaXHH5 deXhy6a4oa1g2Ny0DGWFIFM0VT1nmVDQnD6Vh2vWcOhN9NLp9jGoPartpiMoik3O+Pv0 bChlxQHrZjZwJsZdEyfXxBHUI/wMSMBmqW5hD32vbCDb19xXiorxtWleYP495lFMcHHi MwFMP3wkeX5XJ2s+Vz/0icffAVHm2MkQcoYGKUE8Wf9xn5waRSarnTUMWwDVAFsfGLKJ z7vMeryKLKg7mB456q95JrEIGCBV1xLCKUsmJ/Eevta+624vuPLpC8YC5CNllz02m/p9 XWRg== X-Forwarded-Encrypted: i=1; AFNElJ/i/yVeNeDwaghyI/Wmd1NeFLimwtcfqeKzHhAlq+fc8i99HMFgtRl/AGZwdyuWlg+W362swVwfqaRQfEI=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yxp8b1/6PsbdBWYtXvHx5LBhLo/9VqTbUDERMf0FEQSPp5+qpGv uHLlQ5MZK8+KtN7CHsEXT/I3YrWBE3yGm03+iwcXdDi7Nq28E9gp9T3krMIWQh6uGdM= X-Gm-Gg: Acq92OGB5osViyJxIuqaMO6CVeEcu6oRZHySBpssyaVDx15Y4ONIWwmnmM1XDVX78ER BFAowhBWIi0JZASJw8Zc/3MjCHl29oOOKliqh7Mz+oW1FTJPjcGB09OBluzd+0ppz3RMu1w7uTP aqifO8foEeucTtljGx7PFrE3snQq7ECxlqFzMF++Gy0PTXCeZPD4hquO5xRsL2ER+obrmrGrv8F LYNt9wcGTgb/o8pqBXJYLY9tMvCLqIEoeylJC8zquqA8stArczNivyRP4oj/Bd4FGK711rDdp90 R5yWzaFe+8g80B877i37uxZmh+d18GwsZpeqKkAtGvYFKrWEjelRehcC7e3L98dqyUSMQm4L4r0 Vgi6PWsZImtkj99uXAsXNL8gGQmPEV0CEFqFJHVKPAn19CQJb/3u9Rgl+Cq9QGMfFl1tJ2xaAOa 0iW28GGlzrKfdwPJHEMikNGDz5Wz5MCwlauJJVTynmygxutOqOW2pQanm1PfL3hC4bcr9TCq0= X-Received: by 2002:a17:90b:53cf:b0:35e:3103:d4d8 with SMTP id 98e67ed59e1d1-368f3ad4a1emr3575522a91.6.1778678565011; Wed, 13 May 2026 06:22:45 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.39 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:44 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 64/69] mm/mm_init: Factor out compound page initialization Date: Wed, 13 May 2026 21:20:29 +0800 Message-ID: <20260513132044.41690-18-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org The compound struct page initialization needed by boot-time gigantic hugetlb folios is currently open-coded in hugetlb code, while ZONE_DEVICE has its own separate initialization path in mm_init.c. Factor the common compound memmap setup into memmap_init_compound_page_frozen() so both paths can share the same frozen page initialization logic. This removes duplicated open-coded compound page setup and keeps the initialization rules in one place. Signed-off-by: Muchun Song --- mm/hugetlb.c | 25 +----------- mm/internal.h | 2 + mm/mm_init.c | 111 +++++++++++++++++++------------------------------- 3 files changed, 45 insertions(+), 93 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 10f04fa95d43..7e9f49882395 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3118,28 +3118,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) return true; } -static void __init hugetlb_folio_init_vmemmap(struct page *head, unsigned long pfn, - enum zone_type zone, int nid, unsigned int order, unsigned int nr_pages) -{ - /* - * This is an open-coded prep_compound_page() whereby we avoid - * walking pages twice by initializing/preparing+freezing them in the - * same go. - */ - __init_single_page(head, pfn, zone, nid); - set_page_count(head, 0); - - __SetPageHead(head); - for (int i = 1; i < nr_pages; i++) { - struct page *page = head + i; - - __init_single_page(page, pfn + i, zone, nid); - prep_compound_tail(page, head, order); - set_page_count(page, 0); - } - prep_compound_head(head, order); -} - /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) @@ -3210,8 +3188,7 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) VM_BUG_ON(!hstate_is_gigantic(h)); - hugetlb_folio_init_vmemmap(page, pfn, zone, nid, huge_page_order(h), - vmemmap_nr_struct_pages(pfn, nr_pages)); + memmap_init_compound_page_frozen(page, pfn, zone, nid, huge_page_order(h)); init_new_hugetlb_folio(folio); if (order_vmemmap_optimizable(pfn_to_section_order(pfn))) { diff --git a/mm/internal.h b/mm/internal.h index 416afdf7b2ec..2c67ae25124b 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1793,6 +1793,8 @@ static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid); +void __meminit memmap_init_compound_page_frozen(struct page *head, unsigned long pfn, + enum zone_type zone, int nid, unsigned int order); /* shrinker related functions */ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, diff --git a/mm/mm_init.c b/mm/mm_init.c index 95422e92ede8..9b23c31db8c6 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1018,79 +1018,46 @@ static void __init memmap_init(void) init_unavailable_range(hole_pfn, end_pfn, zone_id, nid); } -#ifdef CONFIG_ZONE_DEVICE -static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, - unsigned long zone_idx, int nid, - struct dev_pagemap *pgmap) +static void __meminit init_single_page_frozen(struct page *page, unsigned long pfn, + enum zone_type zone, int nid) { + __init_single_page(page, pfn, zone, nid); + if (zone_is_zone_device(&NODE_DATA(nid)->node_zones[zone])) { + /* + * ZONE_DEVICE pages are not managed by the page allocator, mark + * them reserved to prevent them from being touched elsewhere. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page_folio(page)->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released - * directly to the driver page allocator which will set the page count - * to 1 when allocating the page. - * - * MEMORY_TYPE_GENERIC and MEMORY_TYPE_FS_DAX pages automatically have - * their refcount reset to one whenever they are freed (ie. after - * their refcount drops to 0). - */ - switch (pgmap->type) { - case MEMORY_DEVICE_FS_DAX: - case MEMORY_DEVICE_PRIVATE: - case MEMORY_DEVICE_COHERENT: - case MEMORY_DEVICE_PCI_P2PDMA: - set_page_count(page, 0); - break; - - case MEMORY_DEVICE_GENERIC: - break; + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->zone_device_data = NULL; } + set_page_count(page, 0); } -static void __ref memmap_init_compound(struct page *head, - unsigned long head_pfn, - unsigned long zone_idx, int nid, - struct dev_pagemap *pgmap, - unsigned long nr_pages) +void __meminit memmap_init_compound_page_frozen(struct page *head, unsigned long pfn, + enum zone_type zone, int nid, unsigned int order) { - unsigned long pfn, end_pfn = head_pfn + nr_pages; - unsigned int order = pgmap->vmemmap_shift; + int nr_pages = vmemmap_nr_struct_pages(pfn, 1UL << order); - /* - * We have to initialize the pages, including setting up page links. - * prep_compound_page() does not take care of that, so instead we - * open-code prep_compound_page() so we can take care of initializing - * the pages in the same go. - */ - __SetPageHead(head); - for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_page(pfn); + init_single_page_frozen(head, pfn, zone, nid); - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); - prep_compound_tail(page, head, order); - set_page_count(page, 0); + __SetPageHead(head); + for (int i = 1; i < nr_pages; i++) { + init_single_page_frozen(head + i, pfn + i, zone, nid); + prep_compound_tail(head + i, head, order); } prep_compound_head(head, order); } +#ifdef CONFIG_ZONE_DEVICE void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -1118,18 +1085,24 @@ void __ref memmap_init_zone_device(struct zone *zone, } for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { - struct page *page = pfn_to_page(pfn); - - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + struct page *head = pfn_to_page(pfn); if (IS_ALIGNED(pfn, PAGES_PER_SECTION)) cond_resched(); - if (pfns_per_compound == 1) - continue; - - memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - vmemmap_nr_struct_pages(pfn, pfns_per_compound)); + if (pgmap->vmemmap_shift) + memmap_init_compound_page_frozen(head, pfn, zone_idx, nid, + pgmap->vmemmap_shift); + else + init_single_page_frozen(head, pfn, zone_idx, nid); + /* + * ZONE_DEVICE pages other than MEMORY_TYPE_GENERIC are released + * directly to the driver page allocator which will set the page + * count to 1 when allocating the page. + */ + if (pgmap->type == MEMORY_DEVICE_GENERIC) + init_page_count(head); + ((struct folio *)head)->pgmap = pgmap; } pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); From patchwork Wed May 13 13:20:30 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237879 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VZVjICez; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20914-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKV4Wnrz1y5L for ; Wed, 13 May 2026 23:23:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvK96qK4z3cPt; Wed, 13 May 2026 23:22:53 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102a" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678573; cv=none; b=Bfd3yHz3KPw3y6jKHEgTzN0v+4yD8l5KvyKmK2hbxfMcLphG4G7ynPMs8NU//41knA3XZne+54VzUYiU0WyilAhPIhFY9w7b1Xzj1WnVJhknRUumwRrN8QZ0ueR/8B218BpfkUoYanYRH3TQgLNrRO38cV4sVr5zi7a182AdVqEsH9c+QzKuHwlaa7IqXZi9t38yD/zQbepawpa7t6uG87ITJ5pBFsCXcD43LBDardHC2c4pAtKJb7acI4edqL3481ZQ5fY5eKzjAuuL5wzNTdRJNoIML38ghnLs6/WraMG6stSJy6GTQKB8tEOxbNvvf5zZBvyrDofzKipaofGbCA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678573; c=relaxed/relaxed; bh=z+xjNGO48cR5AqDS1JugnvXgVXjO+vzOT6RKvB10/GY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZbVlTaXo5wD5GGuf3Fqe958Vcvs/QvMocUaoTErGQJ0yOsp5z8EMRLm+ZtqYf2eG0hoKrDWgjpqcAWI4nlpUViSHbDoA8LwiiaOd8NlUAJFb/TXiK7Q9NIdALe3ax6Qsop2aatw6BkxmcSHRCVfTXQDNjNi2Iu1YlRFgl++cBvQhdVibVXFmwKqMBBwW3irEWLgswuPZDumVFLGXl/twlRHuq1fN6wD1TCQZk+Nq64jYVy8W1O1NcSt32aEUfi7D9a3tUYC2S0ka/cMZBhmXOShhigzePp+7N0C31k3tD8sVydOiJD98fXgNyPaBDuy0NlC8QYXuRaMIt4CYhREWlQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VZVjICez; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=VZVjICez; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvK91JStz2yXj for ; Wed, 13 May 2026 23:22:53 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-3680540a6efso2536486a91.2 for ; Wed, 13 May 2026 06:22:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678571; x=1779283371; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z+xjNGO48cR5AqDS1JugnvXgVXjO+vzOT6RKvB10/GY=; b=VZVjICezV0YFyjruwGfWwhcI9XKsM5t+FyBESVrKVH9dgb18kCKtyNaG9V6beFaX1+ VRA1zNm9vMnowSkzPOVGjPGzXmmMS8/4oMYRzgE1t9qFQm6Tl5NLwFrrz0tuLXEqYSXM iqRyJc5bN84kwt+XDBYROfBFeS+ZdxtISTh/6NY/8+jYOCOYIfIBQn8JyvwmAvsXUTbL PwghUkLATlybH6uwri32iDU/R+i9jv9H4r1jgL1Bv8Qbw2ZUr7UcI2YPTxY7blkhsjXF BmqPxXIQzkoqPFuq/WjqQhiIDWn6Bz+8k60HH3Mux3CBGgaI8U9iwdSc5lNcd9S//ty+ ftkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678571; x=1779283371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=z+xjNGO48cR5AqDS1JugnvXgVXjO+vzOT6RKvB10/GY=; b=WxsI2+iBEmiUVTBj4I6saBNrWcR767pKIavYWNl/AdXCMe/snhwUxBaKAm7iIzscA6 MF1zOP8eFrGevs0B4zq9RfH6PnuEHJ5EFMw8Jr20XvXsFMV2IhbDPAAvnXe/IZAWXvEJ wfEneJD8OHyKaGDr12tDUN6sPPlcaHJUe/NNTbnFO4As7oKvEeAnJd2BshQVsQWudIrU A5/f+e6kfAKqWUKIRz7YrYZdT+3k9aWWxS4eStDVVOV9Da8jzINa/8eFx9Hrslw+xz8v 4h6y7p7lgpSk27i9yZi7T2rUT+OUcdvVmSG8vQ2xuuNrrgvC8IQ0tB/Dp5uAqZN28qro Mcow== X-Forwarded-Encrypted: i=1; AFNElJ8/IShliPxqj/6+VjfHyAiYZpYTn33P8dY2bdx09ZVDlPxgT8ahCXOMWVbkHF/VJ1qLbb8clNaerUSRR74=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yznkumy2l9J0WNn6MFMjEaXtRmuCxyjFoSOHqvL/3IB2DIPpxuo gqdFWOfNSsW2sE4K/XegqRtGETVrQy35Fz4IAM8rJPqRpR7EApXRJUGFgpn/MMqzwJc= X-Gm-Gg: Acq92OE/PEG5ceAUn6+8c8lYZEAKOZwuAl8l/gSD+fFl56DLtanDPOw2tLFKfb2L1F0 avrH2VVAIuqKaBosaF1PjVGYIFIIh3p3pMHFVNd8iEtC/yHWBCjTH0Sfj2Nk6hczgZghtPZhQye Ih5WWkp+R9gozM8htASr8q52Tv7rGR7vFvLATqmerdoHlxyOwUM62Fo5+7s9WN/65ip5IXkS52E OWqqkwmds3klpg74tHO7Nfh2YNZPeiNSwEK+z8B/T6HDGljHpNYaSFZU0UEbiQxp5BqTTMhGVOz FJuXjOmRj/vsXdtEBkZaT9SMlWprrPSfFCpN+HCuGKjE9LXkswKqp1kXd9C0mgO78KJO5t7qb6y eZQBj0dFrbbkVXSx8OcNJL/pTu1BQDS1AqlKieF8k1JjORL0+sLLRUYhKw3hC6yZ+BhYaTCy6P/ WuCj0LSaQsOg/RGkuPgRAneDRBnTElet7IkxPlx2/SmoyNCo9ej9Ev6bPX2ahI X-Received: by 2002:a17:90b:574f:b0:368:6159:980c with SMTP id 98e67ed59e1d1-368f79b61camr2563736a91.20.1778678571207; Wed, 13 May 2026 06:22:51 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.45 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:50 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 65/69] mm/mm_init: Make __init_single_page() static Date: Wed, 13 May 2026 21:20:30 +0800 Message-ID: <20260513132044.41690-19-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org __init_single_page() is only used within mm/mm_init.c, so make it static. Signed-off-by: Muchun Song --- mm/internal.h | 2 -- mm/mm_init.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 2c67ae25124b..80b9ab594dc5 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1791,8 +1791,6 @@ static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte return vma_soft_dirty_enabled(vma) && !pte_soft_dirty(pte); } -void __meminit __init_single_page(struct page *page, unsigned long pfn, - unsigned long zone, int nid); void __meminit memmap_init_compound_page_frozen(struct page *head, unsigned long pfn, enum zone_type zone, int nid, unsigned int order); diff --git a/mm/mm_init.c b/mm/mm_init.c index 9b23c31db8c6..1e11fd683292 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -589,7 +589,7 @@ static void __init find_zone_movable_pfns_for_nodes(void) node_states[N_MEMORY] = saved_node_state; } -void __meminit __init_single_page(struct page *page, unsigned long pfn, +static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { mm_zero_struct_page(page); From patchwork Wed May 13 13:20:31 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237880 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IFXPdCka; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20915-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKf3LYbz1y5L for ; Wed, 13 May 2026 23:23:18 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvKJ3ppJz3dy3; Wed, 13 May 2026 23:23:00 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1031" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678580; cv=none; b=jFtPuKmtWlb5MT42mhlSFZc+sFKtxNjUm2bbXfGZ6vCeZOZbojEQYa2NAQqfsXJtOAJMjv0gc4h6sBdldW9H6SvAfj+8TqVXSM9q/4IBEK2g+7WoCRRrXybiigH9y9rkTyW+RrehvRsbvv3jmMGdPLPvqtkSF6RFL25/uo9NiNV1fpmMTzIL3IAvDi7YZPz6fuM7AWnSmSpCdWCwv4FEaxH1iAd0EA3gIFtxXLTYle+9hLerjZ0NLzt/oxPKp8O4O/VLp4th3kNNRu+jzGetqPvopwRchApWofahBVGTExQiA5pic9xRhEyMHJQx2Hu08Qg0e0KOdnAgXkOLnrtfdw== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678580; c=relaxed/relaxed; bh=PZ/NHdOll26b6UfKrMWj7vaap4nNj6N/arSyoQPAGBw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=U6OvdPbrYFdJvb2ukKpvRToq7BCJV+vyRSzqOotj2XJimVzBaPkQ3Y1/VS3kBQrV2M7x6IXNefVBPmedKKMOaxwzv8bCVLyVNO8ZamgNzAvypzInkqCHCKjKRhFAktALJJnhyQhs2Y0UI/tE3p8snEXYTVl8qXpTtIZhg1OZ7A4fLcaOuxSFjAemfs9ZCBZoB/i9NJa0ZV1RGFRQl9rgDQckLiXX9MJbuy7pA6ir0QVxRHeH77rGTwK0r3t6Tn589qQnnfiujCMTjFKM2iYXez5PUb7V7xtQOIrDYwRi4l+KVFoQoKIOcEyTXu09AFPPJwgPDUu9bN88HISOr4RgiQ== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IFXPdCka; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=IFXPdCka; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1031; helo=mail-pj1-x1031.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvKH3HM2z2yXj for ; Wed, 13 May 2026 23:22:59 +1000 (AEST) Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-365eecc5885so6669572a91.0 for ; Wed, 13 May 2026 06:22:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678578; x=1779283378; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PZ/NHdOll26b6UfKrMWj7vaap4nNj6N/arSyoQPAGBw=; b=IFXPdCkaaIInP1LJx/4hcwDakVPVdRyPCIQkT4wD0AlnCswOYHMirEV9cv66hWLtpy pJn/2gM5eFqgFAO0twYWq9+JgZraosY+Bxh3RVueDbN2oJg4I5xCT+TWvlO0f7cuEzSj Spd1pyWSLhl/9E5z7mX0Yt4a0b74QNrkqEEZjKwFAb0P9HMMAMeFGQwzcplxlprrH1Aq 5UwzN4vGz1h75tZGKSdPjDuGMAbqzXZsb/OGFO8TLk2Vb2nxm9/Do9Z4kiyyK7gQNLCP GikXWKXusT16FS2B55TQSlVXucYme6sGjmX4UQCJGSiuSOanJPo/uaEZZP57+Mlt5mnM fdQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678578; x=1779283378; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=PZ/NHdOll26b6UfKrMWj7vaap4nNj6N/arSyoQPAGBw=; b=gpgARH83dE0slXhiwQ1Q1qTIjtGtsEvpmGCXlLpwce8bmwN5bAUVyHQUuVc0IP5W+5 W73gzx8m/imvyMq5vRT/iXelq3JAjrhIyozoZrr5V9T4UEQY5XstLbaxsZIbhafNY0rn /LTuZd4wyTXgiHQA/afGH4FSFC+hbTqICgBa49+wgCfMY5tmH4Hj5vhHGbjHahJ7M/xK 2mzUjD2nlqPnzqmuFwu7SW6/XwkdIU/YzURGy2Md2/yDz9woBeajhMbKZ6KAUG/v+kPK CqeO52BwyUDiSCLqZUffrm91aTtth0ANAEmELH6lDqJ4cOA6NyIauqUpqEsYOio7XGiI Bs4A== X-Forwarded-Encrypted: i=1; AFNElJ8oDaAc4yZVyB4sMLdpYQ6dh2gH3rZD3aM06RuN+iIrLLwZ99TJXxKN49nFpzFWAPY8ITVxPc4UUQOO4TI=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxNM2rrCrMbCA3nncwBLPuVxqoiDZnw9zsz0u6PafjLj2ENAhIK Cno4ppfQc/MbIDkcFsFeNLMNnwuWqWjjry4eNDlonXrenrB4PxnZ3PlR3EKVuZJ/97c= X-Gm-Gg: Acq92OFa3sO4HDlYAh/RtvlbomzKmkgIGoKndyb9YXg99V558Ql8BqkJIs9XIgcQTpB nOBB7i55JWwXF5tWvOTAAui3DOfmNlT7niLkSxLVahUBKuWsl0paTFppIw/E+GrKYnognlD674e 4l19R5yfWPwbYAqbFra/y0deOClsnd27cTMBCZISx/Ls13R9OdNTOs/5dmk4n3RLaigRoRQHIhK kWxuY4DnG76YmfgPXqfB+jUry4SMTmwfPvCSKLxky6mdVsNWX08EMtMcLHxE9tPu7BgTZgFUE52 dzwzyb/kMPVitFrSiGcQxuDinFCkHSNbUY9Atzsy/ZaLtZ/2zUX1U+Ed7/mPMCDUFHpFMrANLyt d2WZUVK9Q13svbNFg4sJ0D1dAtjikcSVgg/R3jlMusXIe4a6+LcFGnw0+MOEQYyyaIM92wcUpI/ n6OTSQhcly69ZldhqxxZM3XJEBPeUFztVookEcAcXAnE2yivndn/gy1160/jB8 X-Received: by 2002:a17:90a:dfc4:b0:366:5c38:fd61 with SMTP id 98e67ed59e1d1-368f3ac4a7fmr3960604a91.12.1778678577410; Wed, 13 May 2026 06:22:57 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.51 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:22:56 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 66/69] mm/cma: Move CMA pageblock initialization into cma_activate_area() Date: Wed, 13 May 2026 21:20:31 +0800 Message-ID: <20260513132044.41690-20-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Move CMA pageblock initialization for early-reserved pages into cma_activate_area() so CMA pageblock setup is handled in one place. This keeps init_cma_pageblock() in the CMA core instead of pushing special handling for early CMA allocations into its callers. As a side effect, this also fixes the zone->cma_pages accounting race for early-reserved HugeTLB CMA pages. The accounting is no longer updated from parallel hugetlb_struct_page_init() workers and is instead performed serially from cma_activate_area(). Fixes: d2d786714080 ("mm/hugetlb: enable bootmem allocation from CMA areas") Signed-off-by: Muchun Song --- mm/cma.c | 7 +++++-- mm/hugetlb.c | 8 +++----- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index 0369f04c7ba5..c1896c0db63d 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -162,6 +162,10 @@ static void __init cma_activate_area(struct cma *cma) count = early_pfn[r] - cmr->base_pfn; bitmap_count = cma_bitmap_pages_to_bits(cma, count); bitmap_set(cmr->bitmap, 0, bitmap_count); + + for (pfn = cmr->base_pfn; pfn < early_pfn[r]; + pfn += pageblock_nr_pages) + init_cma_pageblock(pfn_to_page(pfn)); } WARN_ON_ONCE(!pfn_valid(cmr->base_pfn)); @@ -1098,8 +1102,7 @@ bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end) * * The caller is responsible for initializing the page structures * in the area properly, since this just points to memblock-allocated - * memory. The caller should subsequently use init_cma_pageblock to - * set the migrate type and CMA stats the pageblocks that were reserved. + * memory. * * If the CMA area fails to activate later, memory obtained through * this interface is not handed to the page allocator, this is diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7e9f49882395..df798f9386d6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3136,9 +3136,7 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); for (i = 0; i < nr_pages; i += pageblock_nr_pages) { - if (folio_test_hugetlb_cma(folio)) - init_cma_pageblock(folio_page(folio, i)); - else + if (!folio_test_hugetlb_cma(folio)) init_pageblock_migratetype(folio_page(folio, i), MIGRATE_MOVABLE, false); } @@ -3206,8 +3204,8 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) * in order to fix confusing memory reports from free(1) and * other side-effects, like CommitLimit going negative. * - * For CMA pages, this is done in init_cma_pageblock - * (via hugetlb_bootmem_init_migratetype), so skip it here. + * For CMA pages, this is done in cma_activate_area(), so skip + * it here. */ if (!folio_test_hugetlb_cma(folio)) adjust_managed_page_count(page, nr_pages); From patchwork Wed May 13 13:20:32 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237881 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=D5jx5m6k; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20916-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKn1klSz1y5L for ; Wed, 13 May 2026 23:23:25 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvKQ5kTyz3cQT; Wed, 13 May 2026 23:23:06 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::102a" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678586; cv=none; b=OnkteeuPbiCU1d1cJrALlARZEiD8vjgaOl83V4JvthY4WPJ1C5qprBRubwfEbyNx0YoimXefYXAA/ozmhtSvjIwP6LOQ3eMM4GLeCkcxgF70ZQa5+OdmI2AOEYQgN4SBlpy5rEmJCRIsDLWDPEl2ig5FrvJlVFfHu8o31m08MpWLnlfH0vPhC9rB2X6SQA3RWQfpadilYaOtMRIGzwGMmzdogDaabqycRirpvN5+l4Z+2P4/6gqxsofeP80LQKQ/jxEkNnxeolMtwA/42VVc2nNj7k3WWEJaxliZKQAscLjbsi7qiyEhLmqg9LPS9H8OjM25UYKb+tEbnINCy/gTdQ== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678586; c=relaxed/relaxed; bh=KmWs7Z/HY7dnLOeYCdnqk/zbv31syyn+Ltp9gTVRyME=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d9byK46iYYpyT9nfX9iSQGOP3BW6KdY2Vq7g+fMnUnQ2TCiOtklQCCXWuZN46KwjJBhHxCehHaOVXIqt+gXeSos3jt6lgTzUPgytgMS1Wa75kqDcekG73rDOh4PvwIJR7KfgJ1oGF+XFBMuXxp8uTY97SL50z61KSvhQ+oiRLjq2G9TG6o+7h/XJsj4BTT3Op7s8+H/MfSXWH2KTKu664/QK3GlssDbeeiAVZGFPMAsocE7iXWhu1w3d3CHzdQsRekO4u6M/pxqBkA9GhJX8a9tZgvJ63QyKK8VLlfHmmZ0r1QR5lMRY0GFKlLudIyiGyLqrHSt52fiPFUUqFQWbOg== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=D5jx5m6k; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=D5jx5m6k; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvKP5rKNz2yTQ for ; Wed, 13 May 2026 23:23:05 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-3567e2b4159so4153320a91.0 for ; Wed, 13 May 2026 06:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678584; x=1779283384; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KmWs7Z/HY7dnLOeYCdnqk/zbv31syyn+Ltp9gTVRyME=; b=D5jx5m6kK2UCIabgBKpYaCy18srcWLcHhlRaHJORz5rGbJA6eyu+kqKzfXCupMAwQO Z2GX7NwDn/dbRd0GtSZLNdWXwwVdbOo+1J91lIGr0Quux22grD99xUgSbdEMKMU5Xl+g V5KkNKhTqhRYlutPsMxL4NPZwRZE53qiQzVoDQT+GNiolsoi2KQ3J8beIBvkr+CwdHa2 DewN71hl4En8S3iE2Pgnk0LHr9+g+O7iDjChrxd2dbcl3NhFEUt9/p8c5B7ir1M/Y0Vy qEpQVtO/eQ420i4DjYamMSpMpHhI4SgmyBEPaThEZ8CoBQxCmLld9RFn8l6aHQNOu2Sl zzQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678584; x=1779283384; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=KmWs7Z/HY7dnLOeYCdnqk/zbv31syyn+Ltp9gTVRyME=; b=cb7Y+3QqHlnEyGiOTHRqM2k7hpzMKLlzqvd0Bh44lyJsRP3b37uXrlRV6XAsI1JXq0 R7T1SBiL14EW6GeoQrwOfukvsclR/DMjM/1pUhTPHCrn+9INwdzM7iAYPUaNB2XFoy1U wfaboKBj0CnUFf0/FbQvEfm8NQ91yuHQLz3djceiTEuFIFBciycKA10hwE3UCThztwDc TMYO/K0g6r9ehO9UbKFZ6a//nswuea7DaYH/U8L/tCjhrvFZ/QD5PgLfao7yEZgEJ8rJ 2Rnbq6fXRjxTBTyxhgjwfiaI7qyqasj4Z6HqTxe7AjwlvyBznGCZq6pgxe7T9D3j6A2+ vJ8Q== X-Forwarded-Encrypted: i=1; AFNElJ86/F0BtW0EPcmrTLnV6QRU5qLU4IlAncbmT/aFxYy5QLLTSwGU9+wg8gWJT1dbk7m/ZFDySHtB27dvGgA=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxYSq9/GcddZf8YMhBdXhjNwS8XRLhjCWSX8P+r4UvhqEwd1iJs hulQLVs/4QcIEZ/Rc0SnKG1ioPMxKD2S42Uu/aBgWYLKt91rumwhh+CqUWqe68RwNdA= X-Gm-Gg: Acq92OEh9LBstVl+TV1LYpCxqXWIUvAFF1djue1lfSVKWbQKazGd49PSeEjHYqmAj4B 2QzXZ61qUzGgi+8EBG1vQzFSa97RrQg+pLn55rrc5OiDtHj5bIcG2YdNntjOq9OlYCnG9kVb5Tz uy0hFYcGRWQuYDKrcfgsQ+zs6hJMZWTbCtJFyoGwk7PbzK5b4Hngijy7nulYctED7ZRPFFa9li+ Hva5YzY8SnaGSu0TNDXxx/jJTIjIExdZd0uSvqkcjjtN4LWY3EaUMPcme7MY7hBPU297pG9gvSd RFh5tBz/bY4FPDPpJFceHZ/63Vc6c917feRaWDumj9rVopE0YXuBRsJEbGx2SYocUOYCPNURTkd LfbFkXonhuI5sZE0zwsEd141685dybZAM+97jFzN1UBW1zNlORJ1nW09SIS19CF/veh9bxuepEs e6IFnug+6mwGnWmndmSMTdvFcwhOU8pXkvyK07QL+OAGI1n1spj1IGKMc3x//y X-Received: by 2002:a17:90b:38cb:b0:368:ed92:6f6 with SMTP id 98e67ed59e1d1-368f77c8a81mr2774547a91.1.1778678583622; Wed, 13 May 2026 06:23:03 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.22.57 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:23:03 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 67/69] mm/cma: Move init_cma_pageblock() into cma.c Date: Wed, 13 May 2026 21:20:32 +0800 Message-ID: <20260513132044.41690-21-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Move init_cma_pageblock() from mm_init.c into cma.c so it lives alongside the CMA code that uses it. No functional change intended. --- mm/cma.c | 8 ++++++++ mm/internal.h | 4 ---- mm/mm_init.c | 9 --------- 3 files changed, 8 insertions(+), 13 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index c1896c0db63d..2843c4f59c4e 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "cma.h" @@ -137,6 +138,13 @@ bool cma_validate_zones(struct cma *cma) return true; } +static void __init init_cma_pageblock(struct page *page) +{ + init_pageblock_migratetype(page, MIGRATE_CMA, false); + adjust_managed_page_count(page, pageblock_nr_pages); + page_zone(page)->cma_pages += pageblock_nr_pages; +} + static void __init cma_activate_area(struct cma *cma) { unsigned long pfn, end_pfn, early_pfn[CMA_MAX_RANGES]; diff --git a/mm/internal.h b/mm/internal.h index 80b9ab594dc5..25b6e767cea0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1126,7 +1126,6 @@ struct cma; #ifdef CONFIG_CMA bool cma_validate_zones(struct cma *cma); void *cma_reserve_early(struct cma *cma, unsigned long size); -void init_cma_pageblock(struct page *page); #else static inline bool cma_validate_zones(struct cma *cma) { @@ -1136,9 +1135,6 @@ static inline void *cma_reserve_early(struct cma *cma, unsigned long size) { return NULL; } -static inline void init_cma_pageblock(struct page *page) -{ -} #endif diff --git a/mm/mm_init.c b/mm/mm_init.c index 1e11fd683292..ff6e9fb468bd 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2200,15 +2200,6 @@ void __init init_cma_reserved_pageblock(struct page *page) adjust_managed_page_count(page, pageblock_nr_pages); page_zone(page)->cma_pages += pageblock_nr_pages; } -/* - * Similar to above, but only set the migrate type and stats. - */ -void __init init_cma_pageblock(struct page *page) -{ - init_pageblock_migratetype(page, MIGRATE_CMA, false); - adjust_managed_page_count(page, pageblock_nr_pages); - page_zone(page)->cma_pages += pageblock_nr_pages; -} #endif void set_zone_contiguous(struct zone *zone) From patchwork Wed May 13 13:20:33 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237882 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UO3Ceo6D; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:21b9:f100::1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20917-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:21b9:f100::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvKw38WZz1y5L for ; Wed, 13 May 2026 23:23:32 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvKX4YYMz3cQr; Wed, 13 May 2026 23:23:12 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1034" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678592; cv=none; b=M3lQSzzgewAUMChyJWuCLnH4X7u6nuI2Bxbc+Ku+lUbFPDTqQjzQd0qb2P3FFqBV7oNTnfgTrBjoIpGo0/e2DmOH8qarGhKNEygIMqAMUXMGNrF9ft3/OsUW3tcYd5vfK6s4eAzfdHXN7FAliTv3rMXT2pwrDuI+rY/1zqE/iKaolVBSCNn9d4+MOaxws+d5MzxJ5px5NPxI7WjVDD1f0nl8QlZO36WoOP7Inmdblon716nC6pjiBxi9TRcvSNN+cFnIoplho8Ct/ceZ0njXw37jUOwmZUC+cFUNz8xgC18A6zbpD1hCzSWSkiM4LXPxfM6JHSrNJqkW9xHkZJ/QNA== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678592; c=relaxed/relaxed; bh=EHqr9wHSnVyYVNoFiVSVd8oLdOOQgIPkLvJW4qLJj/0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LK0DoRPyrTT0uHMK0lPcBJPva5cZ02xT1lmZbamdttElGgG2ZoSn5YvqD8jqeVpaVoo3O7sXmHYjWO6iSAu+pP+SE3932/GY8bWULYV6F3Zw9+451n4MtJlxsDUhel7ZaIMJtFp3U5eHwzbfcVl3vus6rMJ39vyPETV6vZEDO9PmUriQb8ic7SlCM07Vhaa2ER2b39kEnqNOW54NKjGh06KhbQeZOgrGmHv4DGeosFG43y6uoNnNbW3MdbTwqpGvPtr6pazVDNU/ro8V+tkLs3wFjY8jPR/VwDossiyO7lquCU67+F6tFf1XV9DrcA+//KE1vc5PGQb2UfXOHxaAag== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UO3Ceo6D; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=UO3Ceo6D; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvKW5wbkz2yTQ for ; Wed, 13 May 2026 23:23:11 +1000 (AEST) Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-368f25ff4c4so671871a91.2 for ; Wed, 13 May 2026 06:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678590; x=1779283390; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EHqr9wHSnVyYVNoFiVSVd8oLdOOQgIPkLvJW4qLJj/0=; b=UO3Ceo6Dq/tqnbWz4bVzMOOuNRqkAlp1uqYxULQ+kv0VDFxx5pr4T3sf9DA9ZXN5bL NVWqPJxBfej2mODWghuo7jhuV8kIExV0XNQeMyT0KpNDerp7y0+9WyoN9Kx2sZNkbMC8 vqIshfqqfjjoKx+ICQEkTZ2UOm8FL+qyUEad3sy+lMACU77qmiXBkglmMrXwFAb9hAm+ w5avi/F5ium6RhASnutIovM8p9mWj7tBUaMIgUcHMtqc0eCIb/IanHT/2fPoxaFruNAd jIu22yeQRMXHKisb7sX15/cT5goCaARODbNvHtGiFD0FdDP1X1CdAD0/RQHQOUr/1Nzl rhEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678590; x=1779283390; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=EHqr9wHSnVyYVNoFiVSVd8oLdOOQgIPkLvJW4qLJj/0=; b=ZTU5720B0rN1s+ddW4dFwGV9iowRSJ9fr6NFOiCj1kzOjIfcmgs1KnHCZ+38rKJSL+ nrNyc++qmt+3B4aTvaPbpgJaQF00ILlPK85KaenMTGICnqhQU57EyLNMQeYZUFxdrrnp 0UI3v37Y+35bCETRy/5So666l9yGH/Y4CjdL2n0swseMrxPkcfmFotEJnzoQxGv9FgHo YJ2CqC0RmKp3vdZNT9SIn29YjJbPoQhtMU/3RAqn+TI/rjTuAAAG/wpixc6u1LKdocXF 249EccE4I9+jr/H/PY1HIwgYSqLgFlR+Tfd969IzIWI7ql1zEgPFt3Y2zidnkyuWP5Mi L0HA== X-Forwarded-Encrypted: i=1; AFNElJ+2D9RnUGsPgch3YEEoK91Ca6b58wyfLy+pCojxgBzYRT+SGtXAtDxio5MRx9eGKiJrRDWbol7Q9BKVlYo=@lists.ozlabs.org X-Gm-Message-State: AOJu0YxRXxCuIrH+/ir69wE8pBhoR9MgJbe5dPhzNBpxA5OpWidsrLjd hWTV7sPPawPvJ1nFq0MWoxVKpdDFPrYzUWwhaonEkX8jxvfh6Cje2RCrpthLkOxuoBc= X-Gm-Gg: Acq92OHi/VN7hYoViJ6DC8Qvewp8o6NBm94UnhK+/WR5EAHL02RSPME3+oNjudj+e7C U2FBSEuq8RQHcL3DMF1NZxc9flCUFDVcnskDXKIED3ONCd8iVwTVAtv72Wzp1Mm+fbJAd2qdizK u3AAEffshJet/hgWX64w2uGjM9khYwQchBfITVvBL9YS/3PqDrwlfIKxQhXS5BaEwO9lnSSjgsr h0TEpxOf6Nk6zPPsmxSidE6Nca9z7bda343frvdPm4rn2oEiZFZyoTDbjX6keAdEtZCSIBLzEbS ff+rgUYoEEiSFNDOFMJcVoqe9QWXdG9dXH4jJiOXjSF7hlCayKDYTXvhwmsm4GELVio/mNGSELg xfU4xT1+IhaU6MiUix8+wNVcwxrl/5qfYkl4+rT2m3xD4tjTxNtP+1b38pDkYqavIfOdv7r55Ez GBb4h3Jk7F5JMGd7AbPAioHlzU4eTJHsbGQEd3ghAFL5bhvags+pYCXFnIcLKw X-Received: by 2002:a17:90a:d610:b0:35d:a3b4:2f0d with SMTP id 98e67ed59e1d1-368f3bd73f7mr3554267a91.6.1778678589854; Wed, 13 May 2026 06:23:09 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.23.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:23:09 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 68/69] mm/mm_init: Initialize pageblock migratetype in memmap init helpers Date: Wed, 13 May 2026 21:20:33 +0800 Message-ID: <20260513132044.41690-22-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org Move MIGRATE_MOVABLE pageblock initialization into the memmap init helpers in mm/mm_init.c. Let init_single_page_frozen() initialize the pageblock migratetype for single-page folios, and let memmap_init_compound_page_frozen() handle the whole range for compound pages. With pageblock initialization centralized there, drop the duplicate hugetlb bootmem-specific initialization in mm/hugetlb.c. The old hugetlb_bootmem_init_migratetype() skipped CMA folios (via folio_test_hugetlb_cma()), but the new helpers always set MIGRATE_MOVABLE. This is safe because cma_activate_area() will later override the migratetype for CMA pageblocks, so the initial MIGRATE_MOVABLE setting does not matter for CMA pages. Signed-off-by: Muchun Song --- mm/hugetlb.c | 25 ------------------------- mm/mm_init.c | 20 ++++++++++++-------- 2 files changed, 12 insertions(+), 33 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index df798f9386d6..fa269560f657 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3118,30 +3118,6 @@ static bool __init alloc_bootmem_huge_page(struct hstate *h, int nid) return true; } -/* - * memblock-allocated pageblocks might not have the migrate type set - * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) - * here, or MIGRATE_CMA if this was a page allocated through an early CMA - * reservation. - * - * In case of vmemmap optimized folios, the tail vmemmap pages are mapped - * read-only, but that's ok - for sparse vmemmap this does not write to - * the page structure. - */ -static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, - struct hstate *h) -{ - unsigned long nr_pages = pages_per_huge_page(h), i; - - WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); - - for (i = 0; i < nr_pages; i += pageblock_nr_pages) { - if (!folio_test_hugetlb_cma(folio)) - init_pageblock_migratetype(folio_page(folio, i), - MIGRATE_MOVABLE, false); - } -} - static void __init prep_and_add_bootmem_folios(struct hstate *h, struct list_head *folio_list) { @@ -3149,7 +3125,6 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, struct folio *folio, *tmp_f; list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { - hugetlb_bootmem_init_migratetype(folio, h); /* Subdivide locks to achieve better parallel performance */ spin_lock_irqsave(&hugetlb_lock, flags); account_new_hugetlb_folio(h, folio); diff --git a/mm/mm_init.c b/mm/mm_init.c index ff6e9fb468bd..17ae0eb1ccfb 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1018,7 +1018,7 @@ static void __init memmap_init(void) init_unavailable_range(hole_pfn, end_pfn, zone_id, nid); } -static void __meminit init_single_page_frozen(struct page *page, unsigned long pfn, +static void __meminit __init_single_page_frozen(struct page *page, unsigned long pfn, enum zone_type zone, int nid) { __init_single_page(page, pfn, zone, nid); @@ -1042,19 +1042,28 @@ static void __meminit init_single_page_frozen(struct page *page, unsigned long p set_page_count(page, 0); } +static void __meminit init_single_page_frozen(struct page *page, unsigned long pfn, + enum zone_type zone, int nid) +{ + __init_single_page_frozen(page, pfn, zone, nid); + pageblock_migratetype_init_range(pfn, 1, MIGRATE_MOVABLE, false, false); +} + void __meminit memmap_init_compound_page_frozen(struct page *head, unsigned long pfn, enum zone_type zone, int nid, unsigned int order) { int nr_pages = vmemmap_nr_struct_pages(pfn, 1UL << order); - init_single_page_frozen(head, pfn, zone, nid); + __init_single_page_frozen(head, pfn, zone, nid); __SetPageHead(head); for (int i = 1; i < nr_pages; i++) { - init_single_page_frozen(head + i, pfn + i, zone, nid); + __init_single_page_frozen(head + i, pfn + i, zone, nid); prep_compound_tail(head + i, head, order); } prep_compound_head(head, order); + + pageblock_migratetype_init_range(pfn, 1UL << order, MIGRATE_MOVABLE, false, false); } #ifdef CONFIG_ZONE_DEVICE @@ -1087,9 +1096,6 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { struct page *head = pfn_to_page(pfn); - if (IS_ALIGNED(pfn, PAGES_PER_SECTION)) - cond_resched(); - if (pgmap->vmemmap_shift) memmap_init_compound_page_frozen(head, pfn, zone_idx, nid, pgmap->vmemmap_shift); @@ -1105,8 +1111,6 @@ void __ref memmap_init_zone_device(struct zone *zone, ((struct folio *)head)->pgmap = pgmap; } - pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false, false); - pr_debug("%s initialised %lu pages in %ums\n", __func__, nr_pages, jiffies_to_msecs(jiffies - start)); } From patchwork Wed May 13 13:20:34 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 2237883 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SPo+W5Po; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev+bounces-20918-incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4gFvL34DHhz1y5L for ; Wed, 13 May 2026 23:23:39 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [127.0.0.1]) by lists.ozlabs.org (Postfix) with ESMTP id 4gFvKg72CKz3f1s; Wed, 13 May 2026 23:23:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; arc=none smtp.remote-ip="2607:f8b0:4864:20::1036" ARC-Seal: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678599; cv=none; b=fIsj6lcaAWx78nRdUXkBUKH9jkkBuueAErhtb9/SoceJl9LxPmf6m5dBuDAl1jCaB57MLSCWuQIDw2K4akuH8P7JL9g+C3ueImN+SPjiAVvVpC5NadVqzXApyLuiB+0+aGr+AAuRpata4Xe5SUxqj04ngQWFLkftDCIFoPj+qXK/j/rqYlKhdzJ8fnw327BCOhJxRCelRWS6iMCzpNNz3+9djsQ7ToQTcqlpUrFI/ASRUHsnjuRaViTZFGivSYVxWo7Yo3OBjjgswvTRK/cgsu/YHg/sBOe2VisfOCNwd0GK8yr0YvRRLDa1n7imPsDxyRB60DOHXsBd0XSc4SzmJg== ARC-Message-Signature: i=1; a=rsa-sha256; d=lists.ozlabs.org; s=201707; t=1778678599; c=relaxed/relaxed; bh=+Cnpd+J8+D+7FfVvUC6O4aPMQlYjZ4NT8MOvKV4QYBk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=JPGG+ekamjI/AEXnny7TqBS6dOJLuJGEry63KYDs4I62ciYvVQQxHiMoRffesiZj6KkaSi3/3ihbjHDsnPiUt224cG4GIBfHpA2PEkavXQsciFM+UcGL5kTQ4RdN79fKdTaM0E2pT5S3/OJ+ZZBspfevAo0xt1GBsN3o3rgpGLe1y5RooUm5ISIVEKHucZFJfmwpav51nkgW+7/fmL4bnJaC/X38iEVnCmQhTBMS5PUuq6ZQhVexhT1XZurz7WYWvJKSbcVvBADIND57V5DOW8gHUPa5uKqQlPiY29ATr5MGDqO2Jo4ZPbThoW8ieXwlFpIrWpvudNWcCyVMSJ+wvA== ARC-Authentication-Results: i=1; lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SPo+W5Po; dkim-atps=neutral; spf=pass (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) smtp.mailfrom=bytedance.com Authentication-Results: lists.ozlabs.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=bytedance.com header.i=@bytedance.com header.a=rsa-sha256 header.s=google header.b=SPo+W5Po; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=bytedance.com (client-ip=2607:f8b0:4864:20::1036; helo=mail-pj1-x1036.google.com; envelope-from=songmuchun@bytedance.com; receiver=lists.ozlabs.org) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4gFvKf550Rz3cR8 for ; Wed, 13 May 2026 23:23:18 +1000 (AEST) Received: by mail-pj1-x1036.google.com with SMTP id 98e67ed59e1d1-3684a6f3b0bso1558313a91.1 for ; Wed, 13 May 2026 06:23:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1778678597; x=1779283397; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Cnpd+J8+D+7FfVvUC6O4aPMQlYjZ4NT8MOvKV4QYBk=; b=SPo+W5Po5Mo9UGSXVx1kP3z/k60zi2Es8DqxTbU8myrlD/4HPEosKksS8NwIe1derr LpF/NjTC9viPyHU6SyXETEPVniqd/K71kDfQnMR5FpWSvs3q2Bi17x19C7JhHN2HNYBj 3z1CQpMIlAUQUYL5XQWLG7qAmPrWC2HqshkFGJtVW9WZzUu9kiWqw+hwGiH9QmjjoVbm hjJP5UPDT2nWsjDCyfklPSJrlr/ZNM9RdzLwnai8/OWkCXWVeP1lya6fYj3b0mflUK2g dTrLe07TvDHm6uamXDheci3TH6eOvNLCBGG1saF/pTHyB90Iv3MFPlNPG9Cf3eR3JsPs wC1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778678597; x=1779283397; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=+Cnpd+J8+D+7FfVvUC6O4aPMQlYjZ4NT8MOvKV4QYBk=; b=Li5aaIgBYRDkSOO3S9RyXvAcJDS1ol2JEpuCPzfCy3ZuF7+jbQjLthg1/1uEBqXTgK M6rECB2qn8fsda9C2EsFJBq4qiCHXf/tcYdO/bUR6n0s3gRSqxJb8gt2uZufSglC9XfA Uw2FEqOZGWMDg6uAjjqXa4LxEpDuj56hu0MUFneF8fmxQi5bM8nlubcgy8t92sIXpD64 rK4c4BBHRx1D0pBmkleJzWnIQnIsvdUO2JqOT5UBKxhXzV6CgUIq8WtqTA+5M7049Xw8 pvwmh6SnT8S3KJJWHlGQ/utx5IwxB2e116J4/zPGf6aLBNgojICEQuOqOQfaKcVVWS7v qTXg== X-Forwarded-Encrypted: i=1; AFNElJ8biNiBTpZoB17ap7N8K49nwu6QtDVpFNwYBPNE7r6sLQ6+DUquH53gaEDG/Mqx9399LYGvd3+Vbm9qKnI=@lists.ozlabs.org X-Gm-Message-State: AOJu0Yy0pIgBIFoXCSIef+DLgMxwY4XSLc4jfa2gcMJrgsBo78Aw83gx O3q9SPY9/FXdLlYFaHcb4bml8T34bZz3jzGbOPNepZL4UGhkxC5YCPuZ8xSqWwMyme8= X-Gm-Gg: Acq92OGCWc1Klzj4Q11/Ssy/oC0wZemAF69JmqVSuSQLzyig3eJjB4ZlS6BNi4A/XG8 wlvyKIqD7ZFcd+ily60G8/BAOcDu/QT1I1Nya8EntWLIoo3itf8s6BCjPcQyiD1WVMVJ945PIOH OKORmTx8kFF2Q/n4NC/dwDa00Yp8PdZB2C8WNfBOZRh5TM6S+fFlGSrFLT5Vgt0XfcZKMaBNGE4 FpoHgW0sdRV6YIWE5novvQKTOn1rPSH2LyMNh2nRrSRaKp1vKNgtT/JljOQUvMglgrnLojhF4Cb faYFeX5zOH3wIXBj49GsALpT7SbIEIbh0y3cTOM17WkeOPgYN6xoBKeAuhu9knddMkY1CnK2puH aeUbk9e67LaocJEd180cE8p/EXja115rKO1T9aBcQc24tEuX6nDePDiDTHFOyx/ItOhUsKWXYAG JNymLiwvpBv9sJwuP+SAp8YdwB+LFlW1JDW/mPp1xvcvcjrRagRs48Nwqkt2Ep X-Received: by 2002:a17:90b:3a8e:b0:366:5283:cddf with SMTP id 98e67ed59e1d1-368f3d1b489mr2919831a91.9.1778678596480; Wed, 13 May 2026 06:23:16 -0700 (PDT) Received: from PXLDJ45XCM.bytedance.net ([61.213.176.10]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-368edf7cbc2sm3098406a91.14.2026.05.13.06.23.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 13 May 2026 06:23:16 -0700 (PDT) From: Muchun Song To: Andrew Morton , David Hildenbrand , Muchun Song , Oscar Salvador , Michael Ellerman , Madhavan Srinivasan Cc: Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Nicholas Piggin , Christophe Leroy , Ackerley Tng , Frank van der Linden , aneesh.kumar@linux.ibm.com, joao.m.martins@oracle.com, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v2 69/69] Documentation/mm: Rewrite vmemmap_dedup.rst for unified HVO Date: Wed, 13 May 2026 21:20:34 +0800 Message-ID: <20260513132044.41690-23-songmuchun@bytedance.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260513132044.41690-1-songmuchun@bytedance.com> References: <20260513130542.35604-1-songmuchun@bytedance.com> <20260513132044.41690-1-songmuchun@bytedance.com> X-Mailing-List: linuxppc-dev@lists.ozlabs.org List-Id: List-Help: List-Owner: List-Post: List-Archive: , List-Subscribe: , , List-Unsubscribe: Precedence: list MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=4.0.1 OzLabs 8 X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-25) on lists.ozlabs.org HVO is no longer specific to HugeTLB. The optimization has been generalized for other large compound-page users, including device DAX, but vmemmap_dedup.rst still describes the old split model. Rewrite the document around the shared HVO design and behavior, and drop the obsolete powerpc-specific document that only covered the old device DAX path. Signed-off-by: Muchun Song --- Documentation/arch/powerpc/index.rst | 1 - Documentation/arch/powerpc/vmemmap_dedup.rst | 101 --------- Documentation/mm/vmemmap_dedup.rst | 217 ++++--------------- 3 files changed, 42 insertions(+), 277 deletions(-) delete mode 100644 Documentation/arch/powerpc/vmemmap_dedup.rst diff --git a/Documentation/arch/powerpc/index.rst b/Documentation/arch/powerpc/index.rst index 40419bea8e10..4dcf6b0f218c 100644 --- a/Documentation/arch/powerpc/index.rst +++ b/Documentation/arch/powerpc/index.rst @@ -36,7 +36,6 @@ powerpc ultravisor vas-api vcpudispatch_stats - vmemmap_dedup vpa-dtl features diff --git a/Documentation/arch/powerpc/vmemmap_dedup.rst b/Documentation/arch/powerpc/vmemmap_dedup.rst deleted file mode 100644 index dc4db59fdf87..000000000000 --- a/Documentation/arch/powerpc/vmemmap_dedup.rst +++ /dev/null @@ -1,101 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -========== -Device DAX -========== - -The device-dax interface uses the tail deduplication technique explained in -Documentation/mm/vmemmap_dedup.rst - -On powerpc, vmemmap deduplication is only used with radix MMU translation. Also -with a 64K page size, only the devdax namespace with 1G alignment uses vmemmap -deduplication. - -With 2M PMD level mapping, we require 32 struct pages and a single 64K vmemmap -page can contain 1024 struct pages (64K/sizeof(struct page)). Hence there is no -vmemmap deduplication possible. - -With 1G PUD level mapping, we require 16384 struct pages and a single 64K -vmemmap page can contain 1024 struct pages (64K/sizeof(struct page)). Hence we -require 16 64K pages in vmemmap to map the struct page for 1G PUD level mapping. - -Here's how things look like on device-dax after the sections are populated:: - +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - | | | 0 | -------------> | 0 | - | | +-----------+ +-----------+ - | | | 1 | -------------> | 1 | - | | +-----------+ +-----------+ - | | | 2 | ----------------^ ^ ^ ^ ^ ^ - | | +-----------+ | | | | | - | | | 3 | ------------------+ | | | | - | | +-----------+ | | | | - | | | 4 | --------------------+ | | | - | PUD | +-----------+ | | | - | level | | . | ----------------------+ | | - | mapping | +-----------+ | | - | | | . | ------------------------+ | - | | +-----------+ | - | | | 15 | --------------------------+ - | | +-----------+ - | | - | | - | | - +-----------+ - - -With 4K page size, 2M PMD level mapping requires 512 struct pages and a single -4K vmemmap page contains 64 struct pages(4K/sizeof(struct page)). Hence we -require 8 4K pages in vmemmap to map the struct page for 2M pmd level mapping. - -Here's how things look like on device-dax after the sections are populated:: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - | | | 0 | -------------> | 0 | - | | +-----------+ +-----------+ - | | | 1 | -------------> | 1 | - | | +-----------+ +-----------+ - | | | 2 | ----------------^ ^ ^ ^ ^ ^ - | | +-----------+ | | | | | - | | | 3 | ------------------+ | | | | - | | +-----------+ | | | | - | | | 4 | --------------------+ | | | - | PMD | +-----------+ | | | - | level | | 5 | ----------------------+ | | - | mapping | +-----------+ | | - | | | 6 | ------------------------+ | - | | +-----------+ | - | | | 7 | --------------------------+ - | | +-----------+ - | | - | | - | | - +-----------+ - -With 1G PUD level mapping, we require 262144 struct pages and a single 4K -vmemmap page can contain 64 struct pages (4K/sizeof(struct page)). Hence we -require 4096 4K pages in vmemmap to map the struct pages for 1G PUD level -mapping. - -Here's how things look like on device-dax after the sections are populated:: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - | | | 0 | -------------> | 0 | - | | +-----------+ +-----------+ - | | | 1 | -------------> | 1 | - | | +-----------+ +-----------+ - | | | 2 | ----------------^ ^ ^ ^ ^ ^ - | | +-----------+ | | | | | - | | | 3 | ------------------+ | | | | - | | +-----------+ | | | | - | | | 4 | --------------------+ | | | - | PUD | +-----------+ | | | - | level | | . | ----------------------+ | | - | mapping | +-----------+ | | - | | | . | ------------------------+ | - | | +-----------+ | - | | | 4095 | --------------------------+ - | | +-----------+ - | | - | | - | | - +-----------+ diff --git a/Documentation/mm/vmemmap_dedup.rst b/Documentation/mm/vmemmap_dedup.rst index 44e80bd2e398..c3a68a923b0d 100644 --- a/Documentation/mm/vmemmap_dedup.rst +++ b/Documentation/mm/vmemmap_dedup.rst @@ -1,107 +1,34 @@ .. SPDX-License-Identifier: GPL-2.0 -========================================= -A vmemmap diet for HugeTLB and Device DAX -========================================= +=================================================== +Fundamentals of Hugepage Vmemmap Optimization (HVO) +=================================================== -HugeTLB -======= - -This section is to explain how Hugepage Vmemmap Optimization (HVO) for HugeTLB works. - -The ``struct page`` structures are used to describe a physical page frame. By -default, there is a one-to-one mapping from a page frame to its corresponding +The ``struct page`` structures are used to describe a physical base page frame. +By default, there is a one-to-one mapping from a page frame to its corresponding ``struct page``. -HugeTLB pages consist of multiple base page size pages and is supported by many -architectures. See Documentation/admin-guide/mm/hugetlbpage.rst for more -details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB are -currently supported. Since the base page size on x86 is 4KB, a 2MB HugeTLB page -consists of 512 base pages and a 1GB HugeTLB page consists of 262144 base pages. -For each base page, there is a corresponding ``struct page``. - -Within the HugeTLB subsystem, only the first 4 ``struct page`` are used to -contain unique information about a HugeTLB page. ``__NR_USED_SUBPAGE`` provides -this upper limit. The only 'useful' information in the remaining ``struct page`` -is the compound_info field, and this field is the same for all tail pages. - -By removing redundant ``struct page`` for HugeTLB pages, memory can be returned -to the buddy allocator for other uses. - -Different architectures support different HugeTLB pages. For example, the -following table is the HugeTLB page size supported by x86 and arm64 -architectures. Because arm64 supports 4k, 16k, and 64k base pages and -supports contiguous entries, so it supports many kinds of sizes of HugeTLB -page. - -+--------------+-----------+-----------------------------------------------+ -| Architecture | Page Size | HugeTLB Page Size | -+--------------+-----------+-----------+-----------+-----------+-----------+ -| x86-64 | 4KB | 2MB | 1GB | | | -+--------------+-----------+-----------+-----------+-----------+-----------+ -| | 4KB | 64KB | 2MB | 32MB | 1GB | -| +-----------+-----------+-----------+-----------+-----------+ -| arm64 | 16KB | 2MB | 32MB | 1GB | | -| +-----------+-----------+-----------+-----------+-----------+ -| | 64KB | 2MB | 512MB | 16GB | | -+--------------+-----------+-----------+-----------+-----------+-----------+ - -When the system boot up, every HugeTLB page has more than one ``struct page`` -structs which size is (unit: pages):: - - struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - -Where HugeTLB_Size is the size of the HugeTLB page. We know that the size -of the HugeTLB page is always n times PAGE_SIZE. So we can get the following -relationship:: - - HugeTLB_Size = n * PAGE_SIZE - -Then:: - - struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - = n * sizeof(struct page) / PAGE_SIZE +When huge pages (large compound page) are used, they consist of multiple base +page size pages. For each base page, there is a corresponding ``struct page``. +However, only a few ``struct page`` +structures are actually used to contain unique information about the huge page. +The only 'useful' information in the remaining tail ``struct page`` structures +is the ``->compound_info`` field to get the head page structure, and this field +is the same for all tail pages. -We can use huge mapping at the pud/pmd level for the HugeTLB page. +We can remove redundant ``struct page`` structures for huge pages to save memory. +This optimization is referred to as Hugepage Vmemmap Optimization (HVO). -For the HugeTLB page of the pmd level mapping, then:: +The optimization is only applied when the size of the ``struct page`` is a +power-of-2. In this case, all tail pages of the same order are identical. See +``compound_head()``. This allows us to remap the tail pages of the vmemmap to a +shared page. - struct_size = n * sizeof(struct page) / PAGE_SIZE - = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE - = sizeof(struct page) / sizeof(pte_t) - = 64 / 8 - = 8 (pages) +Let’s take a system with a 2 MB huge page and a base page size of 4 KB as an +example for illustration. Here is how things look before optimization:: -Where n is how many pte entries which one page can contains. So the value of -n is (PAGE_SIZE / sizeof(pte_t)). - -This optimization only supports 64-bit system, so the value of sizeof(pte_t) -is 8. And this optimization also applicable only when the size of ``struct page`` -is a power of two. In most cases, the size of ``struct page`` is 64 bytes (e.g. -x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the -size of ``struct page`` structs of it is 8 page frames which size depends on the -size of the base page. - -For the HugeTLB page of the pud level mapping, then:: - - struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) - = PAGE_SIZE / 8 * 8 (pages) - = PAGE_SIZE (pages) - -Where the struct_size(pmd) is the size of the ``struct page`` structs of a -HugeTLB page of the pmd level mapping. - -E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB -HugeTLB page consists in 4096. - -Next, we take the pmd level mapping of the HugeTLB page as an example to -show the internal implementation of this optimization. There are 8 pages -``struct page`` structs associated with a HugeTLB page which is pmd mapped. - -Here is how things look before optimization:: - - HugeTLB struct pages(8 pages) page frame(8 pages) + 2MB Hugepage struct pages (8 pages) page frame (8 pages) +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ | | | 0 | -------------> | 0 | | | +-----------+ +-----------+ @@ -112,9 +39,9 @@ Here is how things look before optimization:: | | | 3 | -------------> | 3 | | | +-----------+ +-----------+ | | | 4 | -------------> | 4 | - | PMD | +-----------+ +-----------+ - | level | | 5 | -------------> | 5 | - | mapping | +-----------+ +-----------+ + | | +-----------+ +-----------+ + | | | 5 | -------------> | 5 | + | | +-----------+ +-----------+ | | | 6 | -------------> | 6 | | | +-----------+ +-----------+ | | | 7 | -------------> | 7 | @@ -124,34 +51,27 @@ Here is how things look before optimization:: | | +-----------+ -The first page of ``struct page`` (page 0) associated with the HugeTLB page -contains the 4 ``struct page`` necessary to describe the HugeTLB. The remaining -pages of ``struct page`` (page 1 to page 7) are tail pages. - -The optimization is only applied when the size of the struct page is a power -of 2. In this case, all tail pages of the same order are identical. See -compound_head(). This allows us to remap the tail pages of the vmemmap to a -shared, read-only page. The head page is also remapped to a new page. This -allows the original vmemmap pages to be freed. +We remap the tail pages (page 1 to page 7) of the vmemmap to a shared, read-only +page (per-zone). Here is how things look after remapping:: - HugeTLB struct pages(8 pages) page frame (new) + 2MB Hugepage struct pages(8 pages) page frame (1 page) +-----------+ ---virt_to_page---> +-----------+ mapping to +----------------+ | | | 0 | -------------> | 0 | | | +-----------+ +----------------+ | | | 1 | ------┐ | | +-----------+ | - | | | 2 | ------┼ +----------------------------+ + | | | 2 | ------┼ + | | +-----------+ | + | | | 3 | ------┼ +----------------------------+ | | +-----------+ | | A single, per-zone page | - | | | 3 | ------┼------> | frame shared among all | + | | | 4 | ------┼------> | frame shared among all | | | +-----------+ | | hugepages of the same size | - | | | 4 | ------┼ +----------------------------+ + | | | 5 | ------┼ +----------------------------+ + | | +-----------+ | + | | | 6 | ------┼ | | +-----------+ | - | | | 5 | ------┼ - | PMD | +-----------+ | - | level | | 6 | ------┼ - | mapping | +-----------+ | | | | 7 | ------┘ | | +-----------+ | | @@ -159,65 +79,12 @@ Here is how things look after remapping:: | | +-----------+ -When a HugeTLB is freed to the buddy system, we should allocate 7 pages for -vmemmap pages and restore the previous mapping relationship. - -For the HugeTLB page of the pud level mapping. It is similar to the former. -We also can use this approach to free (PAGE_SIZE - 1) vmemmap pages. - -Apart from the HugeTLB page of the pmd/pud level mapping, some architectures -(e.g. aarch64) provides a contiguous bit in the translation table entries -that hints to the MMU to indicate that it is one of a contiguous set of -entries that can be cached in a single TLB entry. - -The contiguous bit is used to increase the mapping size at the pmd and pte -(last) level. So this type of HugeTLB page can be optimized only when its -size of the ``struct page`` structs is greater than **1** page. - -Device DAX -========== - -The device-dax interface uses the same tail deduplication technique explained -in the previous chapter, except when used with the vmemmap in -the device (altmap). - -The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), -PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). -For powerpc equivalent details see Documentation/arch/powerpc/vmemmap_dedup.rst - -The differences with HugeTLB are relatively minor. - -It only use 3 ``struct page`` for storing all information as opposed -to 4 on HugeTLB pages. - -There's no remapping of vmemmap given that device-dax memory is not part of -System RAM ranges initialized at boot. Thus the tail page deduplication -happens at a later stage when we populate the sections. HugeTLB reuses the -the head vmemmap page representing, whereas device-dax reuses the tail -vmemmap page. This results in only half of the savings compared to HugeTLB. - -Deduplicated tail pages are not mapped read-only. +Therefore, for any hugepage, if the total size of its corresponding ``struct pages`` +is greater than or equal to the size of two base pages, then HVO technology can +be applied to this hugepage to save memory. For example, in this case, the +smallest hugepage that can apply HVO is 512 KB (its order corresponds to +``OPTIMIZABLE_FOLIO_MIN_ORDER``). Therefore, any hugepage with an order greater +than or equal to ``OPTIMIZABLE_FOLIO_MIN_ORDER`` can apply HVO technology. -Here's how things look like on device-dax after the sections are populated:: - - +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - | | | 0 | -------------> | 0 | - | | +-----------+ +-----------+ - | | | 1 | -------------> | 1 | - | | +-----------+ +-----------+ - | | | 2 | ----------------^ ^ ^ ^ ^ ^ - | | +-----------+ | | | | | - | | | 3 | ------------------+ | | | | - | | +-----------+ | | | | - | | | 4 | --------------------+ | | | - | PMD | +-----------+ | | | - | level | | 5 | ----------------------+ | | - | mapping | +-----------+ | | - | | | 6 | ------------------------+ | - | | +-----------+ | - | | | 7 | --------------------------+ - | | +-----------+ - | | - | | - | | - +-----------+ +Meanwhile, each HVOed hugepage still has ``OPTIMIZED_FOLIO_VMEMMAP_NR_STRUCT_PAGES`` +available ``struct page`` structures.