{"id":2225610,"url":"http://patchwork.ozlabs.org/api/1.1/patches/2225610/?format=json","web_url":"http://patchwork.ozlabs.org/project/kvm-riscv/patch/20260421092457.37649-4-cuiyunhui@bytedance.com/","project":{"id":70,"url":"http://patchwork.ozlabs.org/api/1.1/projects/70/?format=json","name":"Linux KVM RISC-V","link_name":"kvm-riscv","list_id":"kvm-riscv.lists.infradead.org","list_email":"kvm-riscv@lists.infradead.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<20260421092457.37649-4-cuiyunhui@bytedance.com>","date":"2026-04-21T09:24:53","name":"[3/7] riscv: mm: add Svnapot-aware contiguous PTE wrappers","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"14f1b99a23f5fcd7ad87f88f50c771d57ad44627","submitter":{"id":88211,"url":"http://patchwork.ozlabs.org/api/1.1/people/88211/?format=json","name":"Yunhui Cui","email":"cuiyunhui@bytedance.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/kvm-riscv/patch/20260421092457.37649-4-cuiyunhui@bytedance.com/mbox/","series":[{"id":500766,"url":"http://patchwork.ozlabs.org/api/1.1/series/500766/?format=json","web_url":"http://patchwork.ozlabs.org/project/kvm-riscv/list/?series=500766","date":"2026-04-21T09:24:50","name":"riscv: add Svnapot-based contiguous PTE support","version":1,"mbox":"http://patchwork.ozlabs.org/series/500766/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/2225610/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/2225610/checks/","tags":{},"headers":{"Return-Path":"\n ","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n secure) header.d=lists.infradead.org header.i=@lists.infradead.org\n header.a=rsa-sha256 header.s=bombadil.20210309 header.b=YX6RhVau;\n\tdkim=fail reason=\"signature verification failed\" (2048-bit key;\n unprotected) header.d=bytedance.com header.i=@bytedance.com\n header.a=rsa-sha256 header.s=google header.b=cjTrd8Y3;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=none (no SPF record) smtp.mailfrom=lists.infradead.org\n (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org;\n envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from bombadil.infradead.org (bombadil.infradead.org\n [IPv6:2607:7c80:54:3::133])\n\t(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n\t key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384)\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4g0H5j4Cdxz1yGs\n\tfor ; Tue, 21 Apr 2026 19:25:45 +1000 (AEST)","from localhost ([::1] helo=bombadil.infradead.org)\n\tby bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux))\n\tid 1wF7MW-00000008KSA-058V;\n\tTue, 21 Apr 2026 09:25:44 +0000","from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a])\n\tby bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux))\n\tid 1wF7MR-00000008KPH-2kqz\n\tfor kvm-riscv@lists.infradead.org;\n\tTue, 21 Apr 2026 09:25:42 +0000","by mail-pl1-x62a.google.com with SMTP id\n d9443c01a7336-2b788a98557so3106965ad.2\n for ;\n Tue, 21 Apr 2026 02:25:39 -0700 (PDT)","from L6YN4KR4K9.bytedance.net ([61.213.176.6])\n by smtp.gmail.com with ESMTPSA id\n d9443c01a7336-2b5fa9ff39csm131965105ad.4.2026.04.21.02.25.29\n (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256);\n Tue, 21 Apr 2026 02:25:38 -0700 (PDT)"],"DKIM-Signature":["v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;\n\td=lists.infradead.org; s=bombadil.20210309; h=Sender:\n\tContent-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post:\n\tList-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:\n\tMessage-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description:\n\tResent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:\n\tList-Owner; bh=pFQS2oxEfqgV3EGOWNq3tuuRCxD1aPit4V8Zjd+3fqQ=; b=YX6RhVauj4IhJg\n\tAZHkgZeVPXzr3kJzatN68Xcbfb6enkd3qNy/cHVIKrCUkEy75KN6Q/EYTrvJvdowOBXU8dIqTxPlq\n\tirY10l68euzoPWgR34HeH4WdiAu4rwt7sIKoWgLt3wzeySQ21LZCrHMLiOOOR60NgHUJQOvH/H5hR\n\tHd92HkMSrgxixuR7loZXcd8iLEpPm6bW0I9wS7QOAmkLE4POhDYk9J5jMBk7MBK4Xxg56NbVF4XVE\n\tJ76DGIXE5ZTrZntMxUAUxB2XkekJN9bFU5zDiSIyFW1Hx8hLIgwV8TtdOtTqY3gdwDNV9kTkwJnZK\n\tr2sWOOfFcpKeQG+ij60w==;","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=bytedance.com; s=google; t=1776763539; x=1777368339;\n darn=lists.infradead.org;\n h=content-transfer-encoding:mime-version:references:in-reply-to\n :message-id:date:subject:to:from:from:to:cc:subject:date:message-id\n :reply-to;\n bh=e3Aiqpt/ykXwmcx+XDDP/rmLrEEbX7Jgl7wd7KeN5Vc=;\n b=cjTrd8Y3WmGSe4LU93+Ohzgrlcxh3tSw97LvyW//3mfCwFKnZ4nvaKwef/2DUlPmfu\n DqFwXZVqeyTaq8GR2oLkuYHd5BtXsX4xdhqFz4Fs7wwW57bNByarDJ/Lu6bZgkM3iI7x\n ZKBpc/rGZXnIBwNOB0nI3gYy1kRj3JZCsm2Pq+jzzl+UqJmrQq/gW2Ux0CMDOODr99Vj\n o0CPUNINWhQcfsc8PyYmxDJ4RIk9xTOREYOhhPS/N9tSxKfqCfrsrutfnM9zlyF4uo/o\n 0jAYm20fiza3JmN2AyGuPaM1CklFpQ/ZVFB59ccV35pIAW8QTBl1ZFeyqSc+TzwHPJ32\n kb4A=="],"X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20251104; t=1776763539; x=1777368339;\n h=content-transfer-encoding:mime-version:references:in-reply-to\n :message-id:date:subject:to:from:x-gm-gg:x-gm-message-state:from:to\n :cc:subject:date:message-id:reply-to;\n bh=e3Aiqpt/ykXwmcx+XDDP/rmLrEEbX7Jgl7wd7KeN5Vc=;\n b=Su+z0lPqDBRbIfWL1nclxkv6Nib/vcYHTpp1DobNFFvMrXapuaETkaaOCpDzDp7iVI\n z4I4nrowqmHh+1CtkrBeeaXLGxQ3MHoKcH16K+Aq8Gl9C1q+Kxmt+vbGPSmse21SxBZO\n 8ORBl4qeDeE1rPnloODGPWCgVF4at+nhfzOEfeRp5fKeJSS7YxbJQ6cZ7DOupEbRaKhZ\n PDde3yqQ66ammSqA9Ms7uZTWGYaWSKjTOWPNkGa1NO4BneL/6MxZxLMNQS+X3oaKIagd\n DpRnvqK2pNaQpPSb9cjFnSDEppe2B17rZ4YMgl6AYd7tLXpllmWNb5Q+GIS7ulEHAt3V\n 8fIg==","X-Forwarded-Encrypted":"i=1;\n AFNElJ+YDyrmGhs/ZzYX5LzEuhkj8SvZ+jIsG0G6dWGPh0OJygZFhdZN65MH1MP6j9Wppu42cJ24TiHw1bM=@lists.infradead.org","X-Gm-Message-State":"AOJu0YwoQZdoHhryXsSRA1XHxnQ82kImOhJFE93VP0LX5LnS9+/uFFn2\n\tLRiI6FMWPNnZXqrttUbyOprQfKlgkBpChu47IYUhJkDk3ckIfX0VL3dLZQYl/rYeJyg=","X-Gm-Gg":"AeBDieuqG3isRPIaCCv65Hh41TU7QU5Ai+s2Wdm7IfRn8/HqaOKJKB7kbza+173v7hg\n\trdls0ZQTFUZnMLRGgjOl7enl/5bOT+sYVcZ8h+xOyHTkOFy2/Vf4lOy7QOeZtN/wfrnTPl3TwcK\n\tvY+dEnqpXUkDz/UDt/NHuXQb2ZYDIXPn7stZtWfIiUzeeHB9s+JNdC86kYOuhSv25+EXrInCuSz\n\t/VTf7p7mWWnpQsgL6OTWO4bzM4BMlKTU4JSEysXHPXb838TcEw+QhaGOIo4rglNdEEHhzpdfyKO\n\tRX0GzMrrVa+U2aUYRwbdpm5wayo1kILv1XJbPwbkBrGN7Lz6hAD4n5RKVJ/L/UDlaS+TeqKeyP5\n\t9018i2nhxTgf6+4qEbGgeqdAeX4RCsfPdV5LXxSNiM45VI14tGeLv6XqB4qbrCnBvC4Sa+uULRT\n\tgyhopWJgOgvLbXzgyAPmgoMHQMHIvlVLFvL0MRLb9a8TDUFJgsils1sNWScH4fZjxNOjxJ","X-Received":"by 2002:a17:902:f54d:b0:2b4:5b1a:d09c with SMTP id\n d9443c01a7336-2b5f9edb4cbmr186430275ad.15.1776763538472;\n Tue, 21 Apr 2026 02:25:38 -0700 (PDT)","From":"Yunhui Cui ","To":"akpm@linux-foundation.org,\n\talex@ghiti.fr,\n\tandrew+kernel@donnellan.id.au,\n\tandreyknvl@gmail.com,\n\tanup@brainfault.org,\n\taou@eecs.berkeley.edu,\n\tapopple@nvidia.com,\n\tardb@kernel.org,\n\tatish.patra@linux.dev,\n\tbaolin.wang@linux.alibaba.com,\n\tcuiyunhui@bytedance.com,\n\tdavid@kernel.org,\n\tdebug@rivosinc.com,\n\tdjordje.todorovic@htecgroup.com,\n\tdvyukov@google.com,\n\telver@google.com,\n\tglider@google.com,\n\tilias.apalodimas@linaro.org,\n\tjunhui.liu@pigmoral.tech,\n\tkasan-dev@googlegroups.com,\n\tkees@kernel.org,\n\tkevin.brodsky@arm.com,\n\tkvm-riscv@lists.infradead.org,\n\tkvm@vger.kernel.org,\n\tlinux-efi@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org,\n\tlinux-riscv@lists.infradead.org,\n\tliu.xuemei1@zte.com.cn,\n\tljs@kernel.org,\n\tnamcao@linutronix.de,\n\tosalvador@suse.de,\n\tpalmer@dabbelt.com,\n\tpjw@kernel.org,\n\trmclure@linux.ibm.com,\n\trostedt@goodmis.org,\n\trppt@kernel.org,\n\tryabinin.a.a@gmail.com,\n\tsurenb@google.com,\n\tvincenzo.frascino@arm.com,\n\tvishal.moola@gmail.com,\n\twangruikang@iscas.ac.cn,\n\tzhangchunyan@iscas.ac.cn","Subject":"[PATCH 3/7] riscv: mm: add Svnapot-aware contiguous PTE wrappers","Date":"Tue, 21 Apr 2026 17:24:53 +0800","Message-Id":"<20260421092457.37649-4-cuiyunhui@bytedance.com>","X-Mailer":"git-send-email 2.39.2 (Apple Git-143)","In-Reply-To":"<20260421092457.37649-1-cuiyunhui@bytedance.com>","References":"<20260421092457.37649-1-cuiyunhui@bytedance.com>","MIME-Version":"1.0","X-CRM114-Version":"20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 ","X-CRM114-CacheID":"sfid-20260421_022539_731103_580B4B0C ","X-CRM114-Status":"GOOD ( 15.84 )","X-Spam-Score":"0.4 (/)","X-Spam-Report":"Spam detection software,\n running on the system \"bombadil.infradead.org\",\n has NOT identified this incoming email as spam. The original\n message has been attached to this so you can view it or label\n similar future email. If you have any questions, see\n the administrator of that system for details.\n Content preview: Add Svnapot-aware wrappers around the public PTE helpers so\n core MM callers can operate on contiguous mappings without learning the\n NAPOT\n encoding details. Introduce contpte.c to handle folding, unfol [...]\n Content analysis details: (0.4 points, 5.0 required)\n pts rule name description\n ---- ----------------------\n --------------------------------------------------\n -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no\n trust\n [2607:f8b0:4864:20:0:0:0:62a listed in]\n [list.dnswl.org]\n -0.0 SPF_PASS SPF: sender matches SPF record\n 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record\n -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from\n envelope-from domain\n -0.1 DKIM_VALID Message has at least one valid DKIM or DK\n signature\n -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from\n author's\n domain\n 0.1 DKIM_SIGNED Message has a DKIM or DK signature,\n not necessarily valid\n -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%\n [score: 0.0000]\n 2.5 SORTED_RECIPS Recipient list is sorted by address","X-BeenThere":"kvm-riscv@lists.infradead.org","X-Mailman-Version":"2.1.34","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Sender":"\"kvm-riscv\" ","Errors-To":"kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org"},"content":"Add Svnapot-aware wrappers around the public PTE helpers so core MM\ncallers can operate on contiguous mappings without learning the NAPOT\nencoding details. Introduce contpte.c to handle folding, unfolding and\naccessed/dirty state aggregation for contiguous PTE blocks.\n\nKeep the raw __* helpers unchanged so NAPOT-aware callers can continue\nto access the underlying PTE encoding directly, and centralize the\npublic Svnapot-aware wrappers under a single CONFIG_RISCV_ISA_SVNAPOT\nblock with simple alias fallbacks for the non-Svnapot case.\n\nSigned-off-by: Yunhui Cui \n---\n arch/riscv/include/asm/pgtable.h | 288 +++++++++++++++++--\n arch/riscv/mm/Makefile | 1 +\n arch/riscv/mm/contpte.c | 479 +++++++++++++++++++++++++++++++\n arch/riscv/mm/pgtable.c | 39 ++-\n 4 files changed, 769 insertions(+), 38 deletions(-)\n create mode 100644 arch/riscv/mm/contpte.c","diff":"diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h\nindex 4de1f40fa77ea..722483d4df37f 100644\n--- a/arch/riscv/include/asm/pgtable.h\n+++ b/arch/riscv/include/asm/pgtable.h\n@@ -11,6 +11,10 @@\n \n #include \n \n+#ifndef __ASSEMBLER__\n+#include \n+#endif\n+\n #ifndef CONFIG_MMU\n #ifdef CONFIG_RELOCATABLE\n #define KERNEL_LINK_ADDR\tUL(0)\n@@ -301,6 +305,12 @@ static inline unsigned long pte_napot(pte_t pte)\n \treturn 0;\n }\n \n+static inline pte_t pte_mknapot(pte_t pte, unsigned int order)\n+{\n+\t(void)order;\n+\treturn pte;\n+}\n+\n #endif /* CONFIG_RISCV_ISA_SVNAPOT */\n \n /* Yields the page frame number (PFN) of a page table entry */\n@@ -339,6 +349,11 @@ static inline int pte_present(pte_t pte)\n \treturn (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE));\n }\n \n+static inline bool pte_present_napot(pte_t pte)\n+{\n+\treturn pte_present(pte) && pte_napot(pte);\n+}\n+\n #define pte_accessible pte_accessible\n static inline unsigned long pte_accessible(struct mm_struct *mm, pte_t a)\n {\n@@ -392,6 +407,23 @@ static inline int pte_special(pte_t pte)\n \treturn pte_val(pte) & _PAGE_SPECIAL;\n }\n \n+static inline pte_t pte_mknonnapot(pte_t pte, unsigned long addr)\n+{\n+\tunsigned long pfn;\n+\tunsigned long offset;\n+\tpgprot_t prot;\n+\n+\tif (!pte_present_napot(pte))\n+\t\treturn pte;\n+\n+\toffset = (addr & (napot_cont_size(napot_cont_order(pte)) - 1)) >>\n+\t\t PAGE_SHIFT;\n+\tpfn = pte_pfn(pte) + offset;\n+\tprot = __pgprot((pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT);\n+\n+\treturn pfn_pte(pfn, prot);\n+}\n+\n /* static inline pte_t pte_rdprotect(pte_t pte) */\n \n static inline pte_t pte_wrprotect(pte_t pte)\n@@ -642,24 +674,12 @@ static inline void __set_ptes(struct mm_struct *mm, unsigned long addr,\n \n #define __set_ptes __set_ptes\n \n-static inline void set_ptes(struct mm_struct *mm, unsigned long addr,\n-\t\t\t pte_t *ptep, pte_t pteval, unsigned int nr)\n-{\n-\t__set_ptes(mm, addr, ptep, pteval, nr);\n-}\n-\n static inline void __pte_clear(struct mm_struct *mm,\n \t\t\t unsigned long addr, pte_t *ptep)\n {\n \t__set_pte_at(mm, ptep, __pte(0));\n }\n \n-static inline void pte_clear(struct mm_struct *mm,\n-\tunsigned long addr, pte_t *ptep)\n-{\n-\t__pte_clear(mm, addr, ptep);\n-}\n-\n #define __ptep_get __ptep_get\n static inline pte_t __ptep_get(pte_t *ptep)\n {\n@@ -672,6 +692,47 @@ static inline pte_t __ptep_get_lockless(pte_t *ptep)\n \treturn __ptep_get(ptep);\n }\n \n+static inline void __clear_young_dirty_pte(struct vm_area_struct *vma,\n+\t\t\t\t\t unsigned long addr, pte_t *ptep,\n+\t\t\t\t\t pte_t pte, cydp_t flags)\n+{\n+\tpte_t old_pte;\n+\n+\tdo {\n+\t\told_pte = pte;\n+\n+\t\tif (flags & CYDP_CLEAR_YOUNG)\n+\t\t\tpte = pte_mkold(pte);\n+\t\tif (flags & CYDP_CLEAR_DIRTY)\n+\t\t\tpte = pte_mkclean(pte);\n+\n+\t\tpte_val(pte) = cmpxchg_relaxed(&pte_val(*ptep),\n+\t\t\t\t\t pte_val(old_pte),\n+\t\t\t\t\t pte_val(pte));\n+\t} while (pte_val(pte) != pte_val(old_pte));\n+}\n+\n+static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma,\n+\t\t\t\t\t unsigned long addr, pte_t *ptep,\n+\t\t\t\t\t unsigned int nr, cydp_t flags)\n+{\n+\tpte_t pte;\n+\n+\tfor (;;) {\n+\t\tpte = __ptep_get(ptep);\n+\n+\t\tif (flags == (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY))\n+\t\t\t__set_pte(ptep, pte_mkclean(pte_mkold(pte)));\n+\t\telse\n+\t\t\t__clear_young_dirty_pte(vma, addr, ptep, pte, flags);\n+\n+\t\tif (--nr == 0)\n+\t\t\tbreak;\n+\t\tptep++;\n+\t\taddr += PAGE_SIZE;\n+\t}\n+}\n+\n #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS\t/* defined in mm/pgtable.c */\n extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address,\n \t\t\t\t pte_t *ptep, pte_t entry, int dirty);\n@@ -703,12 +764,6 @@ __ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep)\n \n #define __ptep_get_and_clear __ptep_get_and_clear\n \n-static inline pte_t ptep_get_and_clear(struct mm_struct *mm,\n-\t\t\t\t unsigned long address, pte_t *ptep)\n-{\n-\treturn __ptep_get_and_clear(mm, address, ptep);\n-}\n-\n static inline void\n __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep)\n {\n@@ -725,13 +780,6 @@ __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep)\n \n #define __ptep_set_wrprotect __ptep_set_wrprotect\n \n-#define __HAVE_ARCH_PTEP_SET_WRPROTECT\n-static inline void ptep_set_wrprotect(struct mm_struct *mm,\n-\t\t\t\t unsigned long address, pte_t *ptep)\n-{\n-\t__ptep_set_wrprotect(mm, address, ptep);\n-}\n-\n static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,\n \t\t\t\t unsigned long address,\n \t\t\t\t pte_t *ptep)\n@@ -744,9 +792,8 @@ static inline pte_t __ptep_clear_flush(struct vm_area_struct *vma,\n \treturn pte;\n }\n \n-#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH\n-static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,\n-\t\tunsigned long address, pte_t *ptep)\n+static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,\n+\t\t\t\t\t unsigned long address, pte_t *ptep)\n {\n \t/*\n \t * This comment is borrowed from x86, but applies equally to RISC-V:\n@@ -763,9 +810,192 @@ static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,\n \t * shouldn't really matter because there's no real memory\n \t * pressure for swapout to react to. ]\n \t */\n-\treturn ptep_test_and_clear_young(vma, address, ptep);\n+\treturn __ptep_test_and_clear_young(vma, address, ptep);\n+}\n+\n+#define __ptep_clear_flush_young __ptep_clear_flush_young\n+\n+#ifdef CONFIG_RISCV_ISA_SVNAPOT\n+\n+/*\n+ * The Svnapot helpers transparently manage napot-encoded PTEs for the public\n+ * core-MM-facing API below. The napot bit is a private implementation detail\n+ * of those public helpers. Callers that need direct access to the underlying\n+ * PTE encoding must use the low-level __* helpers instead.\n+ */\n+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t pte);\n+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t pte);\n+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte);\n+pte_t napotpte_ptep_get_lockless(pte_t *ptep);\n+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,\n+\t\t pte_t *ptep, pte_t pte, unsigned int nr);\n+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long addr, pte_t *ptep,\n+\t\t\t unsigned int nr, cydp_t flags);\n+bool napotpte_ptep_set_access_flags(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long address, pte_t *ptep,\n+\t\t\t pte_t entry, int dirty);\n+bool napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,\n+\t\t\t\t\tunsigned long address, pte_t *ptep);\n+bool napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long address, pte_t *ptep);\n+\n+static __always_inline bool riscv_pte_present_napot(pte_t pte)\n+{\n+\treturn riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT) &&\n+\t pte_present_napot(pte);\n+}\n+\n+static __always_inline void\n+napotpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep,\n+\t\t pte_t pte)\n+{\n+\tconst unsigned long contmask = napot_pte_num(NAPOT_CONT64KB_ORDER) - 1;\n+\tbool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask;\n+\n+\tif (unlikely(valign)) {\n+\t\tbool palign = (pte_pfn(pte) & contmask) == contmask;\n+\n+\t\tif (unlikely(palign && pte_present(pte) && !pte_napot(pte) &&\n+\t\t\t !pte_special(pte)))\n+\t\t\t__napotpte_try_fold(mm, addr, ptep, pte);\n+\t}\n+}\n+\n+static __always_inline void\n+napotpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep,\n+\t\t pte_t pte)\n+{\n+\tif (unlikely(pte_present_napot(pte)))\n+\t\t__napotpte_try_unfold(mm, addr, ptep, pte);\n+}\n+\n+#define set_ptes set_ptes\n+static inline void set_ptes(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t pteval, unsigned int nr)\n+{\n+\tpteval = pte_mknonnapot(pteval, addr);\n+\n+\tif (likely(nr == 1)) {\n+\t\tnapotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));\n+\t\t__set_ptes(mm, addr, ptep, pteval, 1);\n+\t\tnapotpte_try_fold(mm, addr, ptep, pteval);\n+\t\treturn;\n+\t}\n+\n+\tnapotpte_set_ptes(mm, addr, ptep, pteval, nr);\n+}\n+\n+static inline void pte_clear(struct mm_struct *mm,\n+\t\t\t unsigned long addr, pte_t *ptep)\n+{\n+\tnapotpte_try_unfold(mm, addr, ptep, __ptep_get(ptep));\n+\t__pte_clear(mm, addr, ptep);\n+}\n+\n+#define ptep_get ptep_get\n+static inline pte_t ptep_get(pte_t *ptep)\n+{\n+\tpte_t pte = __ptep_get(ptep);\n+\n+\tif (likely(!pte_present_napot(pte)))\n+\t\treturn pte;\n+\n+\treturn napotpte_ptep_get(ptep, pte);\n+}\n+\n+#define ptep_get_lockless ptep_get_lockless\n+static inline pte_t ptep_get_lockless(pte_t *ptep)\n+{\n+\tpte_t pte = __ptep_get_lockless(ptep);\n+\n+\tif (likely(!pte_present_napot(pte)))\n+\t\treturn pte;\n+\n+\treturn napotpte_ptep_get_lockless(ptep);\n+}\n+\n+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,\n+\t\t\t\t unsigned long address, pte_t *ptep)\n+{\n+\tnapotpte_try_unfold(mm, address, ptep, __ptep_get(ptep));\n+\n+\treturn __ptep_get_and_clear(mm, address, ptep);\n+}\n+\n+#define clear_young_dirty_ptes clear_young_dirty_ptes\n+static inline void clear_young_dirty_ptes(struct vm_area_struct *vma,\n+\t\t\t\t\t unsigned long addr, pte_t *ptep,\n+\t\t\t\t unsigned int nr, cydp_t flags)\n+{\n+\tnapotpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags);\n+}\n+\n+#define __HAVE_ARCH_PTEP_SET_WRPROTECT\n+static inline void ptep_set_wrprotect(struct mm_struct *mm,\n+\t\t\t\t unsigned long address, pte_t *ptep)\n+{\n+\t__ptep_set_wrprotect(mm, address, ptep);\n }\n \n+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH\n+static inline bool ptep_clear_flush_young(struct vm_area_struct *vma,\n+\t\t\t\t\t unsigned long address, pte_t *ptep)\n+{\n+\tpte_t orig_pte = __ptep_get(ptep);\n+\n+\tif (likely(!riscv_pte_present_napot(orig_pte)))\n+\t\treturn __ptep_clear_flush_young(vma, address, ptep);\n+\n+\treturn napotpte_ptep_clear_flush_young(vma, address, ptep);\n+}\n+\n+#else /* CONFIG_RISCV_ISA_SVNAPOT */\n+\n+static __always_inline bool riscv_pte_present_napot(pte_t pte)\n+{\n+\treturn false;\n+}\n+\n+static inline bool napotpte_ptep_set_access_flags(struct vm_area_struct *vma,\n+\t\t\t\t\t\t unsigned long address,\n+\t\t\t\t\t pte_t *ptep, pte_t entry,\n+\t\t\t\t\t int dirty)\n+{\n+\treturn false;\n+}\n+\n+static inline bool\n+napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long address,\n+\t\t\t\t\t pte_t *ptep)\n+{\n+\treturn false;\n+}\n+\n+static inline bool\n+napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,\n+\t\t\t\tunsigned long address,\n+\t\t\t\t\tpte_t *ptep)\n+{\n+\treturn false;\n+}\n+\n+#define set_ptes\t\t\t\t__set_ptes\n+#define pte_clear\t\t\t\t__pte_clear\n+#define ptep_get\t\t\t\t__ptep_get\n+#define ptep_get_lockless\t\t\t__ptep_get_lockless\n+#define ptep_get_and_clear\t\t\t__ptep_get_and_clear\n+#define clear_young_dirty_ptes\t\t\t__clear_young_dirty_ptes\n+#define __HAVE_ARCH_PTEP_SET_WRPROTECT\n+#define ptep_set_wrprotect\t\t\t__ptep_set_wrprotect\n+#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH\n+#define ptep_clear_flush_young\t\t\t__ptep_clear_flush_young\n+\n+#endif /* CONFIG_RISCV_ISA_SVNAPOT */\n+\n #define pgprot_nx pgprot_nx\n static inline pgprot_t pgprot_nx(pgprot_t _prot)\n {\ndiff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile\nindex b916a68d324ad..5855f923b83ec 100644\n--- a/arch/riscv/mm/Makefile\n+++ b/arch/riscv/mm/Makefile\n@@ -17,6 +17,7 @@ obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o pgtable.o tlbflush.o\n obj-y += cacheflush.o\n obj-y += context.o\n obj-y += pmem.o\n+obj-$(CONFIG_RISCV_ISA_SVNAPOT) += contpte.o\n \n obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o\n obj-$(CONFIG_PTDUMP) += ptdump.o\ndiff --git a/arch/riscv/mm/contpte.c b/arch/riscv/mm/contpte.c\nnew file mode 100644\nindex 0000000000000..f73af7d9b099a\n--- /dev/null\n+++ b/arch/riscv/mm/contpte.c\n@@ -0,0 +1,479 @@\n+// SPDX-License-Identifier: GPL-2.0-only\n+\n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+#include \n+\n+#include \n+\n+static inline bool napot_hw_supported(void)\n+{\n+\treturn riscv_has_extension_unlikely(RISCV_ISA_EXT_SVNAPOT);\n+}\n+\n+static inline bool mm_is_user(struct mm_struct *mm)\n+{\n+\tif (unlikely(mm_is_efi(mm)))\n+\t\treturn false;\n+\n+\treturn mm != &init_mm;\n+}\n+\n+static inline unsigned int napotpte_order(void)\n+{\n+\treturn NAPOT_CONT64KB_ORDER;\n+}\n+\n+static inline unsigned long napotpte_size(void)\n+{\n+\treturn napot_cont_size(napotpte_order());\n+}\n+\n+static inline unsigned int napotpte_pte_num(void)\n+{\n+\treturn napot_pte_num(napotpte_order());\n+}\n+\n+static inline unsigned long napotpte_mask(void)\n+{\n+\treturn napotpte_size() - 1;\n+}\n+\n+static inline unsigned long napot_align_addr(unsigned long addr)\n+{\n+\treturn ALIGN_DOWN(addr, napotpte_size());\n+}\n+\n+static inline pte_t *napot_align_ptep(pte_t *ptep)\n+{\n+\treturn PTR_ALIGN_DOWN(ptep, napotpte_pte_num() * sizeof(*ptep));\n+}\n+\n+static inline pte_t pte_mask_ad(pte_t pte)\n+{\n+\treturn pte_mkold(pte_mkclean(pte));\n+}\n+\n+static inline unsigned long pte_protval_no_pfn_no_napot(pte_t pte)\n+{\n+\treturn (pte_val(pte) & ~_PAGE_PFN_MASK) & ~_PAGE_NAPOT;\n+}\n+\n+static inline void napotpte_clear_young_dirty_pte(pte_t *ptep, cydp_t flags)\n+{\n+\tpte_t old_pte, new_pte;\n+\tunsigned long old_val, new_val;\n+\n+\tdo {\n+\t\told_pte = READ_ONCE(*ptep);\n+\t\tnew_pte = old_pte;\n+\t\tif (flags & CYDP_CLEAR_YOUNG)\n+\t\t\tnew_pte = pte_mkold(new_pte);\n+\t\tif (flags & CYDP_CLEAR_DIRTY)\n+\t\t\tnew_pte = pte_mkclean(new_pte);\n+\n+\t\told_val = pte_val(old_pte);\n+\t\tnew_val = pte_val(new_pte);\n+\t} while (cmpxchg_relaxed(&pte_val(*ptep), old_val, new_val) != old_val);\n+}\n+\n+static inline pte_t napotpte_subpte(pte_t *ptep, pte_t pte)\n+{\n+\tunsigned long pfn;\n+\tpgprot_t prot;\n+\n+\tif (!pte_present_napot(pte))\n+\t\treturn pte;\n+\n+\tpfn = pte_pfn(pte) + (ptep - napot_align_ptep(ptep));\n+\tprot = __pgprot(pte_protval_no_pfn_no_napot(pte));\n+\n+\treturn pfn_pte(pfn, prot);\n+}\n+\n+static inline pte_t\n+__napot_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)\n+{\n+\tpte_t pte;\n+\n+\tpte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0));\n+\tpage_table_check_pte_clear(mm, addr, pte);\n+\n+\treturn pte;\n+}\n+\n+static void napotpte_convert(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t target)\n+{\n+\tunsigned long start_addr, end;\n+\tpte_t *start_ptep;\n+\tpte_t ptent, pte;\n+\tunsigned int i, nr;\n+\n+\tstart_addr = napot_align_addr(addr);\n+\tstart_ptep = napot_align_ptep(ptep);\n+\tnr = napotpte_pte_num();\n+\tend = start_addr + napotpte_size();\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tptent = __napot_ptep_get_and_clear(mm, start_addr + i * PAGE_SIZE,\n+\t\t\t\t\t\t start_ptep + i);\n+\t\tif (pte_dirty(ptent))\n+\t\t\ttarget = pte_mkdirty(target);\n+\t\tif (pte_young(ptent))\n+\t\t\ttarget = pte_mkyoung(target);\n+\t}\n+\n+\tflush_tlb_mm_range(mm, start_addr, end, PAGE_SIZE);\n+\n+\tpage_table_check_ptes_set(mm, start_addr, start_ptep, target, nr);\n+\tif (pte_napot(target)) {\n+\t\tfor (i = 0; i < nr; i++)\n+\t\t\t__set_pte_at(mm, start_ptep + i, target);\n+\t\treturn;\n+\t}\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tpte = pfn_pte(pte_pfn(target) + i,\n+\t\t\t __pgprot(pte_protval_no_pfn_no_napot(target)));\n+\t\tif (pte_dirty(target))\n+\t\t\tpte = pte_mkdirty(pte);\n+\t\tif (pte_young(target))\n+\t\t\tpte = pte_mkyoung(pte);\n+\t\t__set_pte_at(mm, start_ptep + i, pte);\n+\t}\n+}\n+\n+static inline bool napotpte_is_consistent(pte_t pte, pte_t orig_pte)\n+{\n+\treturn pte_present_napot(pte) &&\n+\t pte_val(pte_mask_ad(pte)) == pte_val(pte_mask_ad(orig_pte));\n+}\n+\n+void __napotpte_try_fold(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t pte)\n+{\n+\tstruct page *page;\n+\tstruct folio *folio;\n+\tunsigned long folio_start, folio_end;\n+\tunsigned long cont_start, cont_end;\n+\tunsigned long pfn;\n+\tpgprot_t prot;\n+\tpte_t expected, cur;\n+\tpte_t *start;\n+\tunsigned int i, nr;\n+\n+\tif (!napot_hw_supported() || !mm_is_user(mm))\n+\t\treturn;\n+\n+\tif (!pte_present(pte) || pte_napot(pte) || pte_special(pte))\n+\t\treturn;\n+\n+\tpage = pte_page(pte);\n+\tfolio = page_folio(page);\n+\tfolio_start = addr - (page - &folio->page) * PAGE_SIZE;\n+\tfolio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE;\n+\tcont_start = napot_align_addr(addr);\n+\tcont_end = cont_start + napotpte_size();\n+\tif (folio_start > cont_start || folio_end < cont_end)\n+\t\treturn;\n+\n+\tnr = napotpte_pte_num();\n+\tstart = napot_align_ptep(ptep);\n+\n+\tpfn = ALIGN_DOWN(pte_pfn(pte), nr);\n+\tprot = pte_pgprot(pte_mask_ad(pte));\n+\texpected = pfn_pte(pfn, prot);\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tcur = READ_ONCE(start[i]);\n+\t\tif (pte_val(pte_mask_ad(cur)) != pte_val(expected))\n+\t\t\treturn;\n+\t\tpte_val(expected) += 1UL << _PAGE_PFN_SHIFT;\n+\t}\n+\n+\texpected = pte_mknapot(pfn_pte(pfn, prot), napotpte_order());\n+\tnapotpte_convert(mm, addr, ptep, expected);\n+}\n+EXPORT_SYMBOL(__napotpte_try_fold);\n+\n+void __napotpte_try_unfold(struct mm_struct *mm, unsigned long addr,\n+\t\t\t pte_t *ptep, pte_t pte)\n+{\n+\tpte_t target;\n+\tpgprot_t prot;\n+\n+\tif (!napot_hw_supported() || !mm_is_user(mm))\n+\t\treturn;\n+\n+\tprot = __pgprot(pte_protval_no_pfn_no_napot(pte));\n+\ttarget = pfn_pte(pte_pfn(pte), prot);\n+\n+\tnapotpte_convert(mm, addr, ptep, target);\n+}\n+EXPORT_SYMBOL(__napotpte_try_unfold);\n+\n+pte_t napotpte_ptep_get(pte_t *ptep, pte_t orig_pte)\n+{\n+\tpte_t pte, cur;\n+\tpte_t *start;\n+\tunsigned int i, nr;\n+\n+\tif (!napot_hw_supported() || !pte_present_napot(orig_pte))\n+\t\treturn orig_pte;\n+\n+\tpte = orig_pte;\n+\tstart = napot_align_ptep(ptep);\n+\tnr = napotpte_pte_num();\n+\n+\tfor (i = 0; i < nr; i++) {\n+\t\tcur = READ_ONCE(start[i]);\n+\t\tif (!napotpte_is_consistent(cur, orig_pte))\n+\t\t\treturn napotpte_subpte(ptep, orig_pte);\n+\t\tif (pte_dirty(cur))\n+\t\t\tpte = pte_mkdirty(pte);\n+\t\tif (pte_young(cur))\n+\t\t\tpte = pte_mkyoung(pte);\n+\t}\n+\n+\treturn napotpte_subpte(ptep, pte);\n+}\n+EXPORT_SYMBOL(napotpte_ptep_get);\n+\n+pte_t napotpte_ptep_get_lockless(pte_t *orig_ptep)\n+{\n+\tpte_t orig_pte, pte;\n+\tpte_t *ptep;\n+\tunsigned int i, nr;\n+\n+\tif (!napot_hw_supported())\n+\t\treturn READ_ONCE(*orig_ptep);\n+\n+\tnr = napotpte_pte_num();\n+\n+retry:\n+\torig_pte = READ_ONCE(*orig_ptep);\n+\tif (!pte_present_napot(orig_pte))\n+\t\treturn orig_pte;\n+\n+\tptep = napot_align_ptep(orig_ptep);\n+\n+\tfor (i = 0; i < nr; i++, ptep++) {\n+\t\tpte = READ_ONCE(*ptep);\n+\n+\t\tif (!napotpte_is_consistent(pte, orig_pte))\n+\t\t\tgoto retry;\n+\n+\t\tif (pte_dirty(pte)) {\n+\t\t\torig_pte = pte_mkdirty(orig_pte);\n+\t\t\tfor (; i < nr; i++, ptep++) {\n+\t\t\t\tpte = READ_ONCE(*ptep);\n+\n+\t\t\t\tif (!napotpte_is_consistent(pte, orig_pte))\n+\t\t\t\t\tgoto retry;\n+\n+\t\t\t\tif (pte_young(pte)) {\n+\t\t\t\t\torig_pte = pte_mkyoung(orig_pte);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (pte_young(pte)) {\n+\t\t\torig_pte = pte_mkyoung(orig_pte);\n+\t\t\ti++;\n+\t\t\tptep++;\n+\t\t\tfor (; i < nr; i++, ptep++) {\n+\t\t\t\tpte = READ_ONCE(*ptep);\n+\n+\t\t\t\tif (!napotpte_is_consistent(pte, orig_pte))\n+\t\t\t\t\tgoto retry;\n+\n+\t\t\t\tif (pte_dirty(pte)) {\n+\t\t\t\t\torig_pte = pte_mkdirty(orig_pte);\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\treturn napotpte_subpte(orig_ptep, orig_pte);\n+}\n+EXPORT_SYMBOL(napotpte_ptep_get_lockless);\n+\n+void napotpte_set_ptes(struct mm_struct *mm, unsigned long addr,\n+\t\t pte_t *ptep, pte_t pte, unsigned int nr)\n+{\n+\tunsigned long next, end;\n+\tunsigned long pfn, size, boundary;\n+\tpgprot_t prot;\n+\tunsigned int chunk, i;\n+\tpte_t cur;\n+\n+\tif (!napot_hw_supported() || !mm_is_user(mm)) {\n+\t\t__set_ptes(mm, addr, ptep, pte, nr);\n+\t\treturn;\n+\t}\n+\n+\tsize = napotpte_size();\n+\tend = addr + ((unsigned long)nr << PAGE_SHIFT);\n+\tpfn = pte_pfn(pte);\n+\tprot = __pgprot(pte_protval_no_pfn_no_napot(pte));\n+\n+\tdo {\n+\t\tboundary = (addr + size) & ~napotpte_mask();\n+\t\tnext = (boundary - 1 < end - 1) ? boundary : end;\n+\t\tchunk = (next - addr) >> PAGE_SHIFT;\n+\n+\t\tcur = pfn_pte(pfn, prot);\n+\t\tif (((addr | next | (pfn << PAGE_SHIFT)) & napotpte_mask()) == 0) {\n+\t\t\tcur = pte_mknapot(cur, napotpte_order());\n+\t\t\tpage_table_check_ptes_set(mm, addr, ptep, cur, chunk);\n+\t\t\tfor (i = 0; i < chunk; i++)\n+\t\t\t\t__set_pte_at(mm, ptep + i, cur);\n+\t\t} else {\n+\t\t\t__set_ptes(mm, addr, ptep, cur, chunk);\n+\t\t}\n+\n+\t\taddr = next;\n+\t\tptep += chunk;\n+\t\tpfn += chunk;\n+\t} while (addr != end);\n+}\n+EXPORT_SYMBOL(napotpte_set_ptes);\n+\n+void napotpte_clear_young_dirty_ptes(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long addr, pte_t *ptep,\n+\t\t\t\t unsigned int nr, cydp_t flags)\n+{\n+\tstruct mm_struct *mm;\n+\tunsigned long start, end;\n+\tunsigned int total;\n+\n+\tmm = vma->vm_mm;\n+\tif (!napot_hw_supported() || !mm_is_user(mm)) {\n+\t\tfor (;;) {\n+\t\t\tif (flags == CYDP_CLEAR_YOUNG)\n+\t\t\t\t__ptep_test_and_clear_young(vma, addr, ptep);\n+\t\t\telse\n+\t\t\t\tnapotpte_clear_young_dirty_pte(ptep, flags);\n+\t\t\tif (--nr == 0)\n+\t\t\t\tbreak;\n+\t\t\tptep++;\n+\t\t\taddr += PAGE_SIZE;\n+\t\t}\n+\t\treturn;\n+\t}\n+\n+\tstart = addr;\n+\tend = start + nr * PAGE_SIZE;\n+\n+\tif (pte_present_napot(READ_ONCE(*(ptep + nr - 1))))\n+\t\tend = ALIGN(end, napotpte_size());\n+\n+\tif (pte_present_napot(READ_ONCE(*ptep))) {\n+\t\tstart = napot_align_addr(start);\n+\t\tptep = napot_align_ptep(ptep);\n+\t}\n+\n+\ttotal = (end - start) >> PAGE_SHIFT;\n+\tfor (; total; total--, ptep++, start += PAGE_SIZE)\n+\t\tnapotpte_clear_young_dirty_pte(ptep, flags);\n+}\n+EXPORT_SYMBOL(napotpte_clear_young_dirty_ptes);\n+\n+bool napotpte_ptep_set_access_flags(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long address, pte_t *ptep,\n+\t\t\t\t pte_t entry, int dirty)\n+{\n+\tpte_t orig_pte, raw_pte, napot_pte;\n+\tpte_t *start;\n+\tpgprot_t prot;\n+\tunsigned long start_addr;\n+\tunsigned int i, nr;\n+\tbool changed;\n+\n+\traw_pte = READ_ONCE(*ptep);\n+\tif (!napot_hw_supported() || !pte_present_napot(raw_pte))\n+\t\treturn false;\n+\n+\torig_pte = ptep_get(ptep);\n+\tif (pte_val(orig_pte) == pte_val(entry))\n+\t\treturn false;\n+\n+\tif (pte_write(orig_pte) != pte_write(entry)) {\n+\t\t__napotpte_try_unfold(vma->vm_mm, address, ptep, raw_pte);\n+\t\tentry = pte_mknonnapot(entry, address);\n+\n+\t\treturn ptep_set_access_flags(vma, address, ptep, entry, dirty);\n+\t}\n+\n+\tprot = pte_pgprot(entry);\n+\tnapot_pte = pfn_pte(pte_pfn(raw_pte), prot);\n+\tnapot_pte = pte_mknapot(napot_pte, napotpte_order());\n+\n+\tstart = napot_align_ptep(ptep);\n+\tstart_addr = napot_align_addr(address);\n+\tnr = napotpte_pte_num();\n+\tchanged = false;\n+\n+\tpage_table_check_ptes_set(vma->vm_mm, start_addr, start, napot_pte, nr);\n+\tfor (i = 0; i < nr; i++) {\n+\t\tif (!pte_same(READ_ONCE(start[i]), napot_pte)) {\n+\t\t\t__set_pte_at(vma->vm_mm, start + i, napot_pte);\n+\t\t\tchanged = true;\n+\t\t}\n+\t}\n+\n+\tif (changed)\n+\t\tflush_tlb_range(vma, start_addr, start_addr + napotpte_size());\n+\n+\treturn changed;\n+}\n+EXPORT_SYMBOL(napotpte_ptep_set_access_flags);\n+\n+bool napotpte_ptep_test_and_clear_young(struct vm_area_struct *vma,\n+\t\t\t\t\tunsigned long address, pte_t *ptep)\n+{\n+\tpte_t *start;\n+\tunsigned int i, nr;\n+\tbool young;\n+\n+\tif (!napot_hw_supported() || !pte_present_napot(READ_ONCE(*ptep)))\n+\t\treturn false;\n+\n+\tstart = napot_align_ptep(ptep);\n+\tnr = napotpte_pte_num();\n+\tyoung = false;\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tyoung |= test_and_clear_bit(_PAGE_ACCESSED_OFFSET,\n+\t\t\t\t\t &pte_val(start[i]));\n+\n+\treturn young;\n+}\n+EXPORT_SYMBOL(napotpte_ptep_test_and_clear_young);\n+\n+bool napotpte_ptep_clear_flush_young(struct vm_area_struct *vma,\n+\t\t\t\t unsigned long address, pte_t *ptep)\n+{\n+\tunsigned long start_addr;\n+\tbool young;\n+\n+\tyoung = napotpte_ptep_test_and_clear_young(vma, address, ptep);\n+\tif (!young)\n+\t\treturn false;\n+\n+\tstart_addr = napot_align_addr(address);\n+\tflush_tlb_range(vma, start_addr, start_addr + napotpte_size());\n+\n+\treturn true;\n+}\n+EXPORT_SYMBOL(napotpte_ptep_clear_flush_young);\ndiff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c\nindex 9131a78fe15c4..85ff49286f91c 100644\n--- a/arch/riscv/mm/pgtable.c\n+++ b/arch/riscv/mm/pgtable.c\n@@ -9,6 +9,14 @@ int ptep_set_access_flags(struct vm_area_struct *vma,\n \t\t\t unsigned long address, pte_t *ptep,\n \t\t\t pte_t entry, int dirty)\n {\n+\tpte_t raw_pte;\n+\n+\tentry = pte_mknonnapot(entry, address);\n+\traw_pte = READ_ONCE(*ptep);\n+\tif (riscv_pte_present_napot(raw_pte))\n+\t\treturn napotpte_ptep_set_access_flags(vma, address, ptep, entry,\n+\t\t\t\t\t dirty);\n+\n \treturn __ptep_set_access_flags(vma, address, ptep, entry, dirty);\n }\n \n@@ -16,19 +24,26 @@ int __ptep_set_access_flags(struct vm_area_struct *vma,\n \t\t\t unsigned long address, pte_t *ptep,\n \t\t\t pte_t entry, int dirty)\n {\n-\tif (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {\n-\t\tif (!pte_same(ptep_get(ptep), entry)) {\n-\t\t\t__set_pte_at(vma->vm_mm, ptep, entry);\n-\t\t\t/* Here only not svadu is impacted */\n-\t\t\tflush_tlb_page(vma, address);\n-\t\t\treturn true;\n-\t\t}\n+\tpte_t raw_pte;\n+\tbool changed;\n+\n+\tentry = pte_mknonnapot(entry, address);\n+\traw_pte = READ_ONCE(*ptep);\n+\tif (riscv_pte_present_napot(raw_pte))\n+\t\treturn false;\n \n+\tchanged = !pte_same(raw_pte, entry);\n+\tif (!changed)\n \t\treturn false;\n+\n+\t__set_pte_at(vma->vm_mm, ptep, entry);\n+\n+\tif (riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)) {\n+\t\t/* Here only not svadu is impacted */\n+\t\tflush_tlb_page(vma, address);\n+\t\treturn true;\n \t}\n \n-\tif (!pte_same(ptep_get(ptep), entry))\n-\t\t__set_pte_at(vma->vm_mm, ptep, entry);\n \t/*\n \t * update_mmu_cache will unconditionally execute, handling both\n \t * the case that the PTE changed and the spurious fault case.\n@@ -39,6 +54,12 @@ int __ptep_set_access_flags(struct vm_area_struct *vma,\n bool ptep_test_and_clear_young(struct vm_area_struct *vma,\n \t\tunsigned long address, pte_t *ptep)\n {\n+\tpte_t raw_pte;\n+\n+\traw_pte = READ_ONCE(*ptep);\n+\tif (riscv_pte_present_napot(raw_pte))\n+\t\treturn napotpte_ptep_test_and_clear_young(vma, address, ptep);\n+\n \treturn __ptep_test_and_clear_young(vma, address, ptep);\n }\n EXPORT_SYMBOL_GPL(ptep_test_and_clear_young);\n","prefixes":["3/7"]}