From patchwork Wed May 8 19:19:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933166 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=sDjliRaB; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=idYCjN/R; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ3H49gwz20fc for ; Thu, 9 May 2024 05:20:43 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xk4G4F36OZ0fSlftKFOuK1FYIfLfOfagOaHesCvMtlM=; b=sDjliRaB+sgGSu wHpWQv4TFHDpNG5ptJfLTtGDmA3HeypiTdq869XYF5Z/wwuyY4+zE2txGs+NFExOb8vyK2GBIe2Ip zf+oW18RSxlYGRp6hFSI+igMuII7Q/L4miEqgp53xoRJEW9rFBqeQ2f+GUSaxyNbaNBX7fjdoecXr /whSnRGS92n85qLWgNeqPJMqc8HSp3CUKGcny9eXq19Yft0AxOQ3gbHu8LBXUmAWQvQgktf6nyhFv R1Kjk7jk++PgeWvhpfvywc+POfe6BznunVJ4mYJ0LfTXTX4Bs2FDLUvO86RCscXVL5VQP1MyLVN57 EGj/xZc2XzsUovQ90ELA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mqI-0000000GhDT-1IfJ; Wed, 08 May 2024 19:20:42 +0000 Received: from mail-ej1-x62a.google.com ([2a00:1450:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mqD-0000000GhAb-2ses for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:20:39 +0000 Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-a599eedc8eeso24302866b.1 for ; Wed, 08 May 2024 12:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196035; x=1715800835; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=datyv3e5LLeWOM64wvZ1mfI1WId3P2rVazcirFn2Oxk=; b=idYCjN/RXOGcRI3Et4CstnRmWY7BqUzuDTvPi8RJcXMTWwqEUFPJaUipvibNYTiA/P Qh69keCy6p2qQDsv1uLiFxheMS8qU0xAeBvroXXllP92NNjuRXZs0xPWpu2nqG2NBw5g 771x6wU+aux4NjJK09Ga82hBuxJ6mWwdpmPS51sJFuZRmbzE2yluttQvN3G8hedVf2DE vjeubBEXR6JFKiGV5TbxJ8+zLLk+x6cr/HyClGFJvD6sv8YjqicU85XGTnEuvocFaPKl GUMpx6YSz3q9STVAt+M5GwIqSH8Q1R5V7otXcGUJIJMsIrMOGsxvQ8C7GHnVCnKGpmdx XQjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196035; x=1715800835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=datyv3e5LLeWOM64wvZ1mfI1WId3P2rVazcirFn2Oxk=; b=CUNKtmM5OeWVDb52EjRaLV+uS0uxSpbmllLCc5/+D1at4wkhx6fsFQkKBC5Kvz+ZAg Wifet0qZ3EQ5UMnFMHl0WAf99USKYv7tSV0h4ISNFTs59S/iE6LVaPESvyAQDw+4DHeQ N0o43eXmRqse6FulTk4P3OBvwI2z6nRPt2VAbNNiHNHblWBpL3g7mFh3/9VcxBeJ2kmM /X8MI61n2sjn+HbjX1ltYZaxgN/ExSEE0dMhpzFkOdYrZtU2La7zBRl3RiB3RDFAOLra uGeiCVjJDmUP66NCvTYpeILHM1ZP3t+c4oyKn88Jp14rMNNu6eHdh6wjG41Wt5kF3Ppk S/wA== X-Forwarded-Encrypted: i=1; AJvYcCUYyZ/LYSlGyZOA3UiHHrsiCgVzmMszSQ038RRFXaem9KIMoNzLwSjqK4SMvNaZQgAcNk0xLzJlD2mPL8QWk2GiZh/H2C7AY79041/fAw== X-Gm-Message-State: AOJu0Yz+Gx7OzkmZ58gPdiedNS1hHkhhCP9ZS0GHwZAPFzTpHQnNwRSl C3y5sE5RxapTiBlV0Vzq+NdOpKjy6/NEd/IbbBPyPYVfhNwEqlxQpejtpUiUOZE= X-Google-Smtp-Source: AGHT+IFPtUz+3X9HHx3rOZWYiSnLgxFfxDjotG1sGB7J9SeveUOh2JXAABXt+ni1vfq6RI0lDXdHfA== X-Received: by 2002:a50:ab59:0:b0:570:3b8:a990 with SMTP id 4fb4d7f45d1cf-5731da6977emr2508293a12.39.1715196034646; Wed, 08 May 2024 12:20:34 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id g17-20020a056402091100b00571bbaa1c45sm7881992edz.1.2024.05.08.12.20.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:20:34 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 01/12] mm, arm64: Rename ARM64_CONTPTE to THP_CONTPTE Date: Wed, 8 May 2024 21:19:20 +0200 Message-Id: <20240508191931.46060-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122037_773670_E4B04D10 X-CRM114-Status: GOOD ( 12.66 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: The ARM64_CONTPTE config represents the capability to transparently use contpte mappings for THP userspace mappings, which will be implemented in the next commits for riscv, so make this config more g [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:62a listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The ARM64_CONTPTE config represents the capability to transparently use contpte mappings for THP userspace mappings, which will be implemented in the next commits for riscv, so make this config more generic and move it to mm. Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 9 --------- arch/arm64/include/asm/pgtable.h | 6 +++--- arch/arm64/mm/Makefile | 2 +- mm/Kconfig | 9 +++++++++ 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index ac2f6d906cc3..9d823015b4e5 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2227,15 +2227,6 @@ config UNWIND_PATCH_PAC_INTO_SCS select UNWIND_TABLES select DYNAMIC_SCS -config ARM64_CONTPTE - bool "Contiguous PTE mappings for user memory" if EXPERT - depends on TRANSPARENT_HUGEPAGE - default y - help - When enabled, user mappings are configured using the PTE contiguous - bit, for any mappings that meet the size and alignment requirements. - This reduces TLB pressure and improves performance. - endmenu # "Kernel Features" menu "Boot options" diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7c2938cb70b9..1758ce71fae9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1369,7 +1369,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); -#ifdef CONFIG_ARM64_CONTPTE +#ifdef CONFIG_THP_CONTPTE /* * The contpte APIs are used to transparently manage the contiguous bit in ptes @@ -1622,7 +1622,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } -#else /* CONFIG_ARM64_CONTPTE */ +#else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_pte __set_pte @@ -1642,7 +1642,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags -#endif /* CONFIG_ARM64_CONTPTE */ +#endif /* CONFIG_THP_CONTPTE */ int find_num_contig(struct mm_struct *mm, unsigned long addr, pte_t *ptep, size_t *pgsize); diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 60454256945b..52a1b2082627 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,7 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o -obj-$(CONFIG_ARM64_CONTPTE) += contpte.o +obj-$(CONFIG_THP_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/mm/Kconfig b/mm/Kconfig index c325003d6552..fd4de221a1c6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -984,6 +984,15 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_HAS_CONTPTE bool +config THP_CONTPTE + bool "Contiguous PTE mappings for user memory" if EXPERT + depends on ARCH_HAS_CONTPTE && TRANSPARENT_HUGEPAGE + default y + help + When enabled, user mappings are configured using the PTE contiguous + bit, for any mappings that meet the size and alignment requirements. + This reduces TLB pressure and improves performance. + config ARCH_HAS_CURRENT_STACK_POINTER bool help From patchwork Wed May 8 19:19:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933167 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=HmMwDJe7; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=Z8Ocxw2e; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ4W6Rspz20fc for ; Thu, 9 May 2024 05:21:47 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oDDBLH2igu9GEKoLfTFdx8gu5TU31p1RBr5CZbu68RQ=; b=HmMwDJe7SEU7xS IJKcLXxkVmtfQXW4LEHooI2Xe03ZKCnyhiWPY1JtvJ3uXHyRQ1cOG2Kg73HFq4krp6UyePeQHyA+4 Xqd2wT2wMmKW3/89HAHPQ02QF0HF8xuxy/nh/QyHiCnwJmLzYftN8pMYwrujuEmJzDU5oTSXvCFba 1L+OTZAD62h2vYlU4HptWlFlbuEabe7ndq2unhkLP987cWYYaZGrO/BHlG326XuNN3ornFCiO1rJ3 hE1M+wqGDyUduCfS5VFlS+YvF0Iez0U3VCurlj0aQc+c0y4q6RfrMg1q2ueAKsvNL+3ip2CrNIKqv cT7cSIWRTZkkF/dlH1Sw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mrI-0000000Ghab-3Wg1; Wed, 08 May 2024 19:21:46 +0000 Received: from mail-lf1-x12f.google.com ([2a00:1450:4864:20::12f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mrE-0000000GhY0-31Jc for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:21:43 +0000 Received: by mail-lf1-x12f.google.com with SMTP id 2adb3069b0e04-51f12ccff5eso60083e87.1 for ; Wed, 08 May 2024 12:21:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196097; x=1715800897; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ex1T8OVqbNmmxq9WLkEl/7It9+lEf4Ajscw14zap9bQ=; b=Z8Ocxw2eUqk7GEdWA0yWv99Y63siH9Qdqj732HGtAFD/DAw/xXLH/cDwgB4tDy+yU6 BWV4b3rM21pKx1rZx5GQWf3AECtinoPXKHvx9Do5uRdJHDEAG3VKpzDCOKeXRKqSCpze Gv82ddEK2SV91XYxIFutkbv0ugUMNsm8/VP/0SgveYXqprwQdte765N/4E8WMKQgcU10 gCH352fy4qxmaJ+Ehs6+BTcIgdknCax/yq6/SwcOnQXSH6kY5VdovIw4RYBeNHuMvMGy /+G+GTVIbHmDFsuij2Rkh6vubiioVQvEhQyYHALXRBNqKrAmOAm2NL13vcmwA8YeEzGV FaGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196097; x=1715800897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ex1T8OVqbNmmxq9WLkEl/7It9+lEf4Ajscw14zap9bQ=; b=FMGgNYDBRivnWRF+XkZ/uqvj65B3C5XQ38ZJuA1brIhgtCX/13BxfzuOzUinKIdfOs H4AlksiG/4E77zq1CnzBUaea7PRgg6y5i5hZF1ubdYDY+ez01L5SfRE87KthGQHZsFpd +SsFlza1+zg4d5IT+flffrgDLG7t+QI9vW9DShZKxDQiSGrMquXNxKWM/qxL6eS8/WX1 28jvkeqZUlMbE3P8D7PlfPbXxmOFVQA8IR+L2nBVpJkMHrLunZhThN8E/izEFBBWyw0/ fK6E7EXAi6LFhAi9ZIN2XaRyvUIdzC35Ai8UkcJ/6ImAYER19v+nLnYOb+hM8eWNJpk+ SHNA== X-Forwarded-Encrypted: i=1; AJvYcCVClSflMbfEZfqheUR7Z92oeWWNUYiBqcQGDHv/du3cXL9+CjQNQXmpz05mamTrIPrryx/KVR/mTbWKAzYTuZPL7IHs5Kf40/09yUo8RQ== X-Gm-Message-State: AOJu0Yw9Cc2kpEuaVOdHQR1U3hUhVQMHfT0cneibU48pIXeVdDAFsK+w eV74VD2O2RkOX3K6ExEj504jLSrvfBHW3Dfv3E7qZhVtiPIUeqe8Mmq2QeYFvwk= X-Google-Smtp-Source: AGHT+IFjgJRFJyj/slQR8FaD0MqHewWXYnSXcVKmWwkcfJ76gy4SFUv7AbGxIQbLfo1flUN0YcCWiw== X-Received: by 2002:ac2:454b:0:b0:51c:8b45:c9fb with SMTP id 2adb3069b0e04-5217cd4b3e1mr2066413e87.69.1715196097125; Wed, 08 May 2024 12:21:37 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t18-20020a195f12000000b0051f95499c00sm2324036lfb.103.2024.05.08.12.21.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:21:36 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 02/12] mm, riscv, arm64: Use common ptep_get() function Date: Wed, 8 May 2024 21:19:21 +0200 Message-Id: <20240508191931.46060-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122140_820074_97669F72 X-CRM114-Status: GOOD ( 22.50 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_get() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 30 ++++++++++ arch/arm64/mm/contpte.c | 47 +++++ arch/arm64/mm/huget [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:12f listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_get() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 30 ++++++++++---------- arch/arm64/mm/contpte.c | 47 +++++--------------------------- arch/arm64/mm/hugetlbpage.c | 6 ++-- arch/riscv/include/asm/kfence.h | 4 +-- arch/riscv/include/asm/pgtable.h | 22 +++++++++++++++ arch/riscv/kernel/efi.c | 2 +- arch/riscv/kvm/mmu.c | 16 +++++------ arch/riscv/mm/fault.c | 2 +- arch/riscv/mm/kasan_init.c | 2 +- arch/riscv/mm/pageattr.c | 4 +-- arch/riscv/mm/pgtable.c | 4 +-- include/linux/contpte.h | 12 ++++++++ mm/contpte.c | 45 ++++++++++++++++++++++++++++++ 13 files changed, 122 insertions(+), 74 deletions(-) create mode 100644 include/linux/contpte.h diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1758ce71fae9..a878735deb9f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -38,6 +38,7 @@ #include #include #include +#include #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE @@ -1379,8 +1380,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); -extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); + pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); @@ -1456,16 +1456,8 @@ static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) * setting it in the pgtable. */ +extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get -static inline pte_t ptep_get(pte_t *ptep) -{ - pte_t pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(pte))) - return pte; - - return contpte_ptep_get(ptep, pte); -} #define ptep_get_lockless ptep_get_lockless static inline pte_t ptep_get_lockless(pte_t *ptep) @@ -1659,9 +1651,10 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, * find out the number of contiguous ptes. */ if (size == 0) - return find_num_contig(mm, addr, ptep, pgsize); + return mm ? find_num_contig(mm, addr, ptep, pgsize) : CONT_PTES; - *pgsize = size; + if (pgsize) + *pgsize = size; switch (size) { #ifndef __PAGETABLE_PMD_FOLDED @@ -1674,11 +1667,13 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, contig_ptes = 1; break; case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; + if (pgsize) + *pgsize = PMD_SIZE; contig_ptes = CONT_PMDS; break; case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; + if (pgsize) + *pgsize = PAGE_SIZE; contig_ptes = CONT_PTES; break; } @@ -1686,6 +1681,11 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, return contig_ptes; } +static inline pte_t *arch_contpte_align_down(pte_t *ptep) +{ + return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 1b64b4c3f8bf..d5512ebb26e9 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -21,11 +21,6 @@ static inline bool mm_is_user(struct mm_struct *mm) return mm != &init_mm; } -static inline pte_t *contpte_align_down(pte_t *ptep) -{ - return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); -} - static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { @@ -34,10 +29,10 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, * of the range. */ - if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + if (ptep != arch_contpte_align_down(ptep) || nr < CONT_PTES) contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - if (ptep + nr != contpte_align_down(ptep + nr)) { + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); pte_t *last_ptep = ptep + nr - 1; @@ -54,7 +49,7 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *start_ptep; int i; - start_ptep = ptep = contpte_align_down(ptep); + start_ptep = ptep = arch_contpte_align_down(ptep); start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); @@ -122,7 +117,7 @@ void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); expected_pte = pfn_pte(pfn, prot); orig_ptep = ptep; - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); for (i = 0; i < CONT_PTES; i++) { subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); @@ -152,34 +147,6 @@ void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(__contpte_try_unfold); -pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) -{ - /* - * Gather access/dirty bits, which may be populated in any of the ptes - * of the contig range. We are guaranteed to be holding the PTL, so any - * contiguous range cannot be unfolded or otherwise modified under our - * feet. - */ - - pte_t pte; - int i; - - ptep = contpte_align_down(ptep); - - for (i = 0; i < CONT_PTES; i++, ptep++) { - pte = __ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} -EXPORT_SYMBOL_GPL(contpte_ptep_get); - pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) { /* @@ -214,7 +181,7 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) return orig_pte; orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); - ptep = contpte_align_down(orig_ptep); + ptep = arch_contpte_align_down(orig_ptep); pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { @@ -312,7 +279,7 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, int young = 0; int i; - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) @@ -389,7 +356,7 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, * faults. Avoid per-page tlb flush in __ptep_set_access_flags() * and instead flush the whole range at the end. */ - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 5869f20ca28e..083e80ac5790 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -101,12 +101,14 @@ int find_num_contig(struct mm_struct *mm, unsigned long addr, pud_t *pudp; pmd_t *pmdp; - *pgsize = PAGE_SIZE; + if (pgsize) + *pgsize = PAGE_SIZE; p4dp = p4d_offset(pgdp, addr); pudp = pud_offset(p4dp, addr); pmdp = pmd_offset(pudp, addr); if ((pte_t *)pmdp == ptep) { - *pgsize = PMD_SIZE; + if (pgsize) + *pgsize = PMD_SIZE; return CONT_PMDS; } return CONT_PTES; diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index 7388edd88986..f303fef8591c 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -18,9 +18,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) pte_t *pte = virt_to_kpte(addr); if (protect) - set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT)); + set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); else - set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT)); + set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9e397935536e..8d05179f6bbe 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -299,6 +299,7 @@ static inline unsigned long pte_napot(pte_t pte) #define pte_cont pte_napot #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) +#define pte_valid_cont pte_valid_napot static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { @@ -571,6 +572,17 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, return size >> hugepage_shift; } + +static inline pte_t *arch_contpte_align_down(pte_t *ptep) +{ + pte_t __pte = READ_ONCE(*ptep); + int ncontig; + + ncontig = napot_pte_num(napot_cont_order(__pte)); + + return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); +} + #endif static inline pte_t __ptep_get(pte_t *ptep) @@ -696,8 +708,18 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#ifdef CONFIG_THP_CONTPTE + +extern pte_t ptep_get(pte_t *ptep); +#define ptep_get ptep_get + +#else /* CONFIG_THP_CONTPTE */ + #define ptep_get __ptep_get #define set_ptes __set_ptes + +#endif /* CONFIG_THP_CONTPTE */ + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define pte_clear __pte_clear diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index b64bf1624a05..3d2a635c69ac 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md) static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { efi_memory_desc_t *md = data; - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); unsigned long val; if (md->attribute & EFI_MEMORY_RO) { diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index a9e2fd7245e1..70c6cb3864d6 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -103,7 +103,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, *ptep_level = current_level; ptep = (pte_t *)kvm->arch.pgd; ptep = &ptep[gstage_pte_index(addr, current_level)]; - while (ptep && pte_val(ptep_get(ptep))) { + while (ptep && pte_val(__ptep_get(ptep))) { if (gstage_pte_leaf(ptep)) { *ptep_level = current_level; *ptepp = ptep; @@ -113,7 +113,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, if (current_level) { current_level--; *ptep_level = current_level; - ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); ptep = &ptep[gstage_pte_index(addr, current_level)]; } else { ptep = NULL; @@ -149,7 +149,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, if (gstage_pte_leaf(ptep)) return -EEXIST; - if (!pte_val(ptep_get(ptep))) { + if (!pte_val(__ptep_get(ptep))) { if (!pcache) return -ENOMEM; next_ptep = kvm_mmu_memory_cache_alloc(pcache); @@ -160,7 +160,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, } else { if (gstage_pte_leaf(ptep)) return -EEXIST; - next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); } current_level--; @@ -239,11 +239,11 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, BUG_ON(addr & (page_size - 1)); - if (!pte_val(ptep_get(ptep))) + if (!pte_val(__ptep_get(ptep))) return; if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(next_ptep_level, &next_page_size); @@ -261,7 +261,7 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); + set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); gstage_remote_tlb_flush(kvm, ptep_level, addr); } } @@ -603,7 +603,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return pte_young(ptep_get(ptep)); + return pte_young(__ptep_get(ptep)); } int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 3ba1d4dde5dd..0e08afc1fc6a 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -175,7 +175,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * silently loop forever. */ pte_k = pte_offset_kernel(pmd_k, addr); - if (!pte_present(ptep_get(pte_k))) { + if (!pte_present(__ptep_get(pte_k))) { no_context(regs, addr); return; } diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..381d61f42ab8 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -39,7 +39,7 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned ptep = pte_offset_kernel(pmd, vaddr); do { - if (pte_none(ptep_get(ptep))) { + if (pte_none(__ptep_get(ptep))) { phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 410056a50aa9..98c9dc4b983c 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -68,7 +68,7 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr, static int pageattr_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val = ptep_get(pte); + pte_t val = __ptep_get(pte); val = __pte(set_pageattr_masks(pte_val(val), walk)); set_pte(pte, val); @@ -435,5 +435,5 @@ bool kernel_page_present(struct page *page) return true; pte = pte_offset_kernel(pmd, addr); - return pte_present(ptep_get(pte)); + return pte_present(__ptep_get(pte)); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index e86df7ef193c..5756bde9eb42 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -9,7 +9,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { - if (!pte_same(ptep_get(ptep), entry)) + if (!pte_same(__ptep_get(ptep), entry)) __set_pte_at(vma->vm_mm, ptep, entry); /* * update_mmu_cache will unconditionally execute, handling both @@ -22,7 +22,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) + if (!pte_young(__ptep_get(ptep))) return 0; return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); } diff --git a/include/linux/contpte.h b/include/linux/contpte.h new file mode 100644 index 000000000000..46acac7222ca --- /dev/null +++ b/include/linux/contpte.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTPTE_H +#define _LINUX_CONTPTE_H + +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); + +#endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 15791f6d9c41..d365356bbf92 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -6,6 +6,7 @@ #include #include #include +#include /* * Any arch that wants to use that needs to define: @@ -17,6 +18,8 @@ * - __ptep_set_wrprotect() * - pte_cont() * - arch_contpte_get_num_contig() + * - pte_valid_cont() + * - arch_contpte_align_down() */ /* @@ -28,6 +31,7 @@ * - huge_ptep_set_access_flags() * - huge_ptep_set_wrprotect() * - huge_ptep_clear_flush() + * - ptep_get() */ pte_t huge_ptep_get(pte_t *ptep) @@ -270,3 +274,44 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); } + +#ifdef CONFIG_THP_CONTPTE +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We are guaranteed to be holding the PTL, so any + * contiguous range cannot be unfolded or otherwise modified under our + * feet. + */ + + pte_t pte; + int i, ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, 0, NULL); + + for (i = 0; i < ncontig; i++, ptep++) { + pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL_GPL(contpte_ptep_get); + +__always_inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get(ptep, pte); +} +#endif /* CONTPTE_THP_CONTPTE */ From patchwork Wed May 8 19:19:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933168 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=AUvcBtLA; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=riTdFF1I; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ5j53jRz20fc for ; Thu, 9 May 2024 05:22:49 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YBn+xL0wr7bhbiDso4mofd0ZZ5hSaiJ5zorhUWtjJho=; b=AUvcBtLAZAHjSo iGURcJzgzzhj5IH/Cm5fvVVsci6vHkYRsbvaLkCsCzxFflciV09Ty72Ecf34aybYJO9eJOfUmRvoB tg2gXXSTselIYam8NU9JGhKADTLyx1lF0r5GdxZNRVlSwXh0eKUgHXVLxxv/fjTViKHb0px1hSpgq Tv/809XVA1UkpCTmx+H9ZcLbpl1XNUbrqYjkqEa4YQKYVWgKpVNPr+OJ4AKY6Jjd8motWepxvzxUp iC/Ud/GMjsDeL4CEbDJpADp5ZYp2YD+sh06cLvtX4Ofwi275JSSWCth/frTWXmDgp4vkX032Z3rE7 DNcDE/kaGlBAYvuX0q0w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4msK-0000000Ghzy-1ZpH; Wed, 08 May 2024 19:22:48 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4msE-0000000Ghwc-3BGk for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:22:46 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-41ba1ba55ffso746595e9.1 for ; Wed, 08 May 2024 12:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196159; x=1715800959; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/S8/CVFlZq+KrirwWJV8BSI4dNCVaFyTIK5dU7sIVgI=; b=riTdFF1IsChQ81Ml1LRUFuIPJtHmYGvzM8T0COHRZCSrnW5QXVeM7BoJmvdDTxAawc HgitARQg66mP4p1extUpCFuerbHlH77C5okh8H4xhLVYbMQXHa0VoGrCAB6WhOkad/q6 kSRzeunh/t4M9wUvyP1EDCY2smCPIAjmdGHe++dEfUdhQoGylFm7VSN8ZBypUfG31iNN 9yRRHWq6ov2mfFgTn5UXJZPMiYFp1UJyVAsgUXD0nKzw2Uzp5gIC6yoSLyBdtYsQM/zk lOk9fSo3xBfpyoNFqVpp5RBNO1QTfvAp6M53pETEByXXDLinPnyY8wfnS4sJBAOSrWv8 NJkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196159; x=1715800959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/S8/CVFlZq+KrirwWJV8BSI4dNCVaFyTIK5dU7sIVgI=; b=E1yCcYcsKqjMHIXg8UmOebBVSScAuejtVcSuCjOyGxwiJrYO/TvgVK0ZFuli+tL7Cc ewzzVmw2C1gsErpiBhjNbUvuIorDD96zmWmERAxtm0WNUzktM5vJrbaAvB1OhQRYy36d 5OMUCQk4mZmwlhfnolBvf/fNzmK+9DZ4aIvVw0gOP3fEVXDf1Nf9INsCfSehOo4B0xYQ Hmga5V6FrKmm/YEiGNw+cs+JOJImabKOJfxl/nndf4tVE/1573UnGphuOfe1zfV83ZHB eJ94hSltRTc5daHSUjf9KnqPhTJvB2yzb4V9t1i6IhXIRM79TSVmZhkK/aq0hWGfR9Wa 2JHg== X-Forwarded-Encrypted: i=1; AJvYcCXa/oOq33HTVkrcPdKI+tdekY6XQX4D4ZhEF4k+YGt7LICDg1+VvAENoEsGikfx0WoM51nsM6WRil7bviyzHSXDGTRbghRR1XmqukPgIA== X-Gm-Message-State: AOJu0YwPGhRPLgNmJ08aQ4vDKLpoBs0uhqekALxCic6SmSkADdPP7dQz zHypo1btkpHfJvoDLy7wMWD5Rnq8J/s0pUuxAHH0T5Pda/gz+LlU/QGMX2scnto= X-Google-Smtp-Source: AGHT+IGv6186BLCsPjXiNHxvqIGGbP+8Nt6Ojq3omkEdJDigbcQHRKHJSnceRKhu854h+/CEdBxYBA== X-Received: by 2002:a05:600c:3b83:b0:41b:4506:9fd with SMTP id 5b1f17b1804b1-41fbcb42dd6mr4932615e9.6.1715196158601; Wed, 08 May 2024 12:22:38 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id f20-20020a05600c155400b0041bf7da4200sm3310028wmg.33.2024.05.08.12.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:22:38 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 03/12] mm, riscv, arm64: Use common set_ptes() function Date: Wed, 8 May 2024 21:19:22 +0200 Message-Id: <20240508191931.46060-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122242_994385_15BCE517 X-CRM114-Status: GOOD ( 29.21 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware set_ptes() function from arm64. Note that riscv can support multiple contpte sizes so the arm64 code was modified to take into account this possibility. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 88 ++++------ arch/arm64/mm/contpte.c | 162 arch/arm64/mm/mmu.c | 2 +- arch/riscv/incl [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:32f listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware set_ptes() function from arm64. Note that riscv can support multiple contpte sizes so the arm64 code was modified to take into account this possibility. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 88 ++++------ arch/arm64/mm/contpte.c | 162 ------------------ arch/arm64/mm/mmu.c | 2 +- arch/riscv/include/asm/pgtable.h | 76 +++++++++ include/linux/contpte.h | 10 ++ mm/contpte.c | 277 ++++++++++++++++++++++++++++++- 6 files changed, 398 insertions(+), 217 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index a878735deb9f..e85b3a052a02 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -118,10 +118,17 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) #define pte_tagged(pte) ((pte_val(pte) & PTE_ATTRINDX_MASK) == \ PTE_ATTRINDX(MT_NORMAL_TAGGED)) -#define pte_cont_addr_end(addr, end) \ -({ unsigned long __boundary = ((addr) + CONT_PTE_SIZE) & CONT_PTE_MASK; \ - (__boundary - 1 < (end) - 1) ? __boundary : (end); \ -}) +static inline unsigned long arch_contpte_addr_end(unsigned long addr, + unsigned long end, + int *ncontig) +{ + unsigned long __boundary = (addr + CONT_PTE_SIZE) & CONT_PTE_MASK; + + if (ncontig) + *ncontig = CONT_PTES; + + return (__boundary - 1 < end - 1) ? __boundary : end; +} #define pmd_cont_addr_end(addr, end) \ ({ unsigned long __boundary = ((addr) + CONT_PMD_SIZE) & CONT_PMD_MASK; \ @@ -1377,13 +1384,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ -extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); -extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); -extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr); extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, @@ -1399,36 +1400,6 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); -static __always_inline void contpte_try_fold(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte) -{ - /* - * Only bother trying if both the virtual and physical addresses are - * aligned and correspond to the last entry in a contig range. The core - * code mostly modifies ranges from low to high, so this is the likely - * the last modification in the contig range, so a good time to fold. - * We can't fold special mappings, because there is no associated folio. - */ - - const unsigned long contmask = CONT_PTES - 1; - bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; - - if (unlikely(valign)) { - bool palign = (pte_pfn(pte) & contmask) == contmask; - - if (unlikely(palign && - pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) - __contpte_try_fold(mm, addr, ptep, pte); - } -} - -static __always_inline void contpte_try_unfold(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte) -{ - if (unlikely(pte_valid_cont(pte))) - __contpte_try_unfold(mm, addr, ptep, pte); -} - #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) { @@ -1485,20 +1456,9 @@ static inline void set_pte(pte_t *ptep, pte_t pte) __set_pte(ptep, pte_mknoncont(pte)); } +extern void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); #define set_ptes set_ptes -static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr) -{ - pte = pte_mknoncont(pte); - - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __set_ptes(mm, addr, ptep, pte, 1); - contpte_try_fold(mm, addr, ptep, pte); - } else { - contpte_set_ptes(mm, addr, ptep, pte, nr); - } -} static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1686,6 +1646,28 @@ static inline pte_t *arch_contpte_align_down(pte_t *ptep) return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); } +static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range(vma, start, end, stride, true, 3); +} + +static inline int arch_contpte_get_first_ncontig(size_t *pgsize) +{ + if (pgsize) + *pgsize = PAGE_SIZE; + + return CONT_PTES; +} + +/* Must return 0 when ncontig does not have any next. */ +static inline int arch_contpte_get_next_ncontig(int ncontig, size_t *pgsize) +{ + return 0; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index d5512ebb26e9..e225e458856e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -8,19 +8,6 @@ #include #include -static inline bool mm_is_user(struct mm_struct *mm) -{ - /* - * Don't attempt to apply the contig bit to kernel mappings, because - * dynamically adding/removing the contig bit can cause page faults. - * These racing faults are ok for user space, since they get serialized - * on the PTL. But kernel mappings can't tolerate faults. - */ - if (unlikely(mm_is_efi(mm))) - return false; - return mm != &init_mm; -} - static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { @@ -41,112 +28,6 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, } } -static void contpte_convert(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long start_addr; - pte_t *start_ptep; - int i; - - start_ptep = ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { - pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); - - if (pte_dirty(ptent)) - pte = pte_mkdirty(pte); - - if (pte_young(ptent)) - pte = pte_mkyoung(pte); - } - - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); - - __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); -} - -void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - /* - * We have already checked that the virtual and pysical addresses are - * correctly aligned for a contpte mapping in contpte_try_fold() so the - * remaining checks are to ensure that the contpte range is fully - * covered by a single folio, and ensure that all the ptes are valid - * with contiguous PFNs and matching prots. We ignore the state of the - * access and dirty bits for the purpose of deciding if its a contiguous - * range; the folding process will generate a single contpte entry which - * has a single access and dirty bit. Those 2 bits are the logical OR of - * their respective bits in the constituent pte entries. In order to - * ensure the contpte range is covered by a single folio, we must - * recover the folio from the pfn, but special mappings don't have a - * folio backing them. Fortunately contpte_try_fold() already checked - * that the pte is not special - we never try to fold special mappings. - * Note we can't use vm_normal_page() for this since we don't have the - * vma. - */ - - unsigned long folio_start, folio_end; - unsigned long cont_start, cont_end; - pte_t expected_pte, subpte; - struct folio *folio; - struct page *page; - unsigned long pfn; - pte_t *orig_ptep; - pgprot_t prot; - - int i; - - if (!mm_is_user(mm)) - return; - - page = pte_page(pte); - folio = page_folio(page); - folio_start = addr - (page - &folio->page) * PAGE_SIZE; - folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; - cont_start = ALIGN_DOWN(addr, CONT_PTE_SIZE); - cont_end = cont_start + CONT_PTE_SIZE; - - if (folio_start > cont_start || folio_end < cont_end) - return; - - pfn = ALIGN_DOWN(pte_pfn(pte), CONT_PTES); - prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); - expected_pte = pfn_pte(pfn, prot); - orig_ptep = ptep; - ptep = arch_contpte_align_down(ptep); - - for (i = 0; i < CONT_PTES; i++) { - subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); - if (!pte_same(subpte, expected_pte)) - return; - expected_pte = pte_advance_pfn(expected_pte, 1); - ptep++; - } - - pte = pte_mkcont(pte); - contpte_convert(mm, addr, orig_ptep, pte); -} -EXPORT_SYMBOL_GPL(__contpte_try_fold); - -void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - /* - * We have already checked that the ptes are contiguous in - * contpte_try_unfold(), so just check that the mm is user space. - */ - if (!mm_is_user(mm)) - return; - - pte = pte_mknoncont(pte); - contpte_convert(mm, addr, ptep, pte); -} -EXPORT_SYMBOL_GPL(__contpte_try_unfold); - pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) { /* @@ -204,49 +85,6 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) } EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); -void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr) -{ - unsigned long next; - unsigned long end; - unsigned long pfn; - pgprot_t prot; - - /* - * The set_ptes() spec guarantees that when nr > 1, the initial state of - * all ptes is not-present. Therefore we never need to unfold or - * otherwise invalidate a range before we set the new ptes. - * contpte_set_ptes() should never be called for nr < 2. - */ - VM_WARN_ON(nr == 1); - - if (!mm_is_user(mm)) - return __set_ptes(mm, addr, ptep, pte, nr); - - end = addr + (nr << PAGE_SHIFT); - pfn = pte_pfn(pte); - prot = pte_pgprot(pte); - - do { - next = pte_cont_addr_end(addr, end); - nr = (next - addr) >> PAGE_SHIFT; - pte = pfn_pte(pfn, prot); - - if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) - pte = pte_mkcont(pte); - else - pte = pte_mknoncont(pte); - - __set_ptes(mm, addr, ptep, pte, nr); - - addr = next; - ptep += nr; - pfn += nr; - - } while (addr != end); -} -EXPORT_SYMBOL_GPL(contpte_set_ptes); - void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 495b732d5af3..b7ad732660aa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -222,7 +222,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, do { pgprot_t __prot = prot; - next = pte_cont_addr_end(addr, end); + next = arch_contpte_addr_end(addr, end, NULL); /* use a contiguous mapping if the range is suitably aligned */ if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 8d05179f6bbe..ebfe6b16529e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -301,6 +301,20 @@ static inline unsigned long pte_napot(pte_t pte) #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) #define pte_valid_cont pte_valid_napot +/* + * contpte is what we expose to the core mm code, this is not exactly a napot + * mapping since the size is not encoded in the pfn yet. + */ +static inline pte_t pte_mknoncont(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + +static inline pte_t pte_mkcont(pte_t pte) +{ + return __pte(pte_val(pte) | _PAGE_NAPOT); +} + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -329,6 +343,11 @@ static inline unsigned long pte_napot(pte_t pte) #endif /* CONFIG_RISCV_ISA_SVNAPOT */ +static inline pgprot_t pte_pgprot(pte_t pte) +{ + return __pgprot(pte_val(pte) & ~_PAGE_PFN_MASK); +} + /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { @@ -354,6 +373,11 @@ static inline int pte_present(pte_t pte) return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); } +static inline int pte_valid(pte_t pte) +{ + return (pte_val(pte) & _PAGE_PRESENT); +} + static inline int pte_none(pte_t pte) { return (pte_val(pte) == 0); @@ -583,6 +607,55 @@ static inline pte_t *arch_contpte_align_down(pte_t *ptep) return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); } +static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + flush_tlb_mm_range(vma->vm_mm, start, end, stride); +} + +static inline int arch_contpte_get_first_ncontig(size_t *pgsize) +{ + if (pgsize) + *pgsize = PAGE_SIZE; + + return 1 << NAPOT_CONT64KB_ORDER; +} + +/* Must return 0 when ncontig does not have any next. */ +static inline int arch_contpte_get_next_ncontig(int ncontig, size_t *pgsize) +{ + return 0; +} + +#define for_each_contpte_order_rev(ncontig, order, pgsize) \ + for (pgsize = PAGE_SIZE, order = NAPOT_ORDER_MAX - 1, ncontig = BIT(order); \ + ncontig >= BIT(NAPOT_CONT_ORDER_BASE); \ + order--, ncontig = BIT(order)) + +static inline unsigned long arch_contpte_addr_end(unsigned long addr, + unsigned long end, + int *ncontig) +{ + unsigned long contpte_saddr, contpte_eaddr, contpte_size; + size_t pgsize; + int contig, order; + + for_each_contpte_order_rev(contig, order, pgsize) { + contpte_size = contig * pgsize; + contpte_saddr = ALIGN_DOWN(addr, contpte_size); + contpte_eaddr = contpte_saddr + contpte_size; + + if (contpte_saddr >= addr && contpte_eaddr <= end) { + *ncontig = contig; + return contpte_eaddr; + } + } + + *ncontig = 0; + return end; +} #endif static inline pte_t __ptep_get(pte_t *ptep) @@ -712,6 +785,9 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr); +#define set_ptes set_ptes #else /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 46acac7222ca..54d10204e9af 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -8,5 +8,15 @@ * a private implementation detail of the public ptep API (see below). */ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index d365356bbf92..566745d7842f 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -7,6 +7,7 @@ #include #include #include +#include /* * Any arch that wants to use that needs to define: @@ -20,6 +21,14 @@ * - arch_contpte_get_num_contig() * - pte_valid_cont() * - arch_contpte_align_down() + * - arch_contpte_flush_tlb_range() + * - arch_contpte_get_first_ncontig() + * - arch_contpte_get_next_ncontig() + * - arch_contpte_addr_end() + * - pte_mkcont() + * - pte_mknoncont() + * - pte_valid() + * - pte_pgprot() */ /* @@ -32,6 +41,7 @@ * - huge_ptep_set_wrprotect() * - huge_ptep_clear_flush() * - ptep_get() + * - set_ptes() */ pte_t huge_ptep_get(pte_t *ptep) @@ -314,4 +324,269 @@ __always_inline pte_t ptep_get(pte_t *ptep) return contpte_ptep_get(ptep, pte); } -#endif /* CONTPTE_THP_CONTPTE */ +EXPORT_SYMBOL_GPL(ptep_get); + +static inline bool mm_is_user(struct mm_struct *mm) +{ + /* + * Don't attempt to apply the contig bit to kernel mappings, because + * dynamically adding/removing the contig bit can cause page faults. + * These racing faults are ok for user space, since they get serialized + * on the PTL. But kernel mappings can't tolerate faults. + */ + if (unlikely(mm_is_efi(mm))) + return false; + return mm != &init_mm; +} + +static void contpte_convert(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, + int ncontig, size_t pgsize) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr; + pte_t *start_ptep; + int i; + + start_addr = addr; + start_ptep = ptep; + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) { + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(ptent)) + pte = pte_mkdirty(pte); + + if (pte_young(ptent)) + pte = pte_mkyoung(pte); + } + + arch_contpte_flush_tlb_range(&vma, start_addr, addr, pgsize); + + __set_ptes(mm, start_addr, start_ptep, pte, ncontig); +} + +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + unsigned long start_addr; + pte_t *start_ptep; + size_t pgsize; + int ncontig; + + /* + * We have already checked that the ptes are contiguous in + * contpte_try_unfold(), so just check that the mm is user space. + */ + if (!mm_is_user(mm)) + return; + + pte = pte_mknoncont(pte); + start_ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(mm, addr, start_ptep, 0, &pgsize); + start_addr = ALIGN_DOWN(addr, ncontig * pgsize); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), ncontig), pte_pgprot(pte)); + + contpte_convert(mm, start_addr, start_ptep, pte, ncontig, pgsize); +} + +__always_inline void contpte_try_unfold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ + if (unlikely(pte_valid_cont(pte))) + __contpte_try_unfold(mm, addr, ptep, pte); +} +EXPORT_SYMBOL_GPL(contpte_try_unfold); + +static bool contpte_is_last_pte(unsigned long addr, pte_t pte, int ncontig) +{ + const unsigned long contmask = ncontig - 1; + bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + + if (unlikely(valign)) { + bool palign = (pte_pfn(pte) & contmask) == contmask; + + if (unlikely(palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) + return true; + } + + return false; +} + +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + unsigned long folio_start, folio_end; + unsigned long cont_start, cont_end; + pte_t expected_pte, subpte; + struct folio *folio; + struct page *page; + unsigned long pfn; + pte_t *cur_ptep; + pgprot_t prot; + size_t pgsize; + int i, ncontig, ncontig_prev; + + if (!mm_is_user(mm)) + return; + + page = pte_page(pte); + folio = page_folio(page); + folio_start = addr - (page - &folio->page) * PAGE_SIZE; + folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; + + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + ncontig_prev = 0; + ncontig = arch_contpte_get_first_ncontig(&pgsize); + + do { + /* Make sure we still belong to the same folio */ + cont_start = ALIGN_DOWN(addr, ncontig * pgsize); + cont_end = cont_start + ncontig * pgsize; + + if (folio_start > cont_start || folio_end < cont_end) + break; + + pfn = ALIGN_DOWN(pte_pfn(pte), ncontig); + expected_pte = pfn_pte(pfn, prot); + cur_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); + + /* + * Either the region is already a contpte mapping or make sure + * the ptes belong to a contpte region. + */ + if (!pte_valid_cont(__ptep_get(cur_ptep))) { + for (i = 0; i < ncontig - ncontig_prev; i++) { + subpte = pte_mkold(pte_mkclean(__ptep_get(cur_ptep))); + if (!pte_same(subpte, expected_pte)) + goto out; + expected_pte = pte_advance_pfn(expected_pte, 1); + cur_ptep++; + } + } + + /* + * Compute the next contpte region to check: avoid checking + * the same region again by only checking the "second-half" + * (the "region buddy") of the current region, which keeps the + * whole thing in O(n). + */ + ncontig_prev = ncontig; + ncontig = arch_contpte_get_next_ncontig(ncontig, &pgsize); + /* set_ptes spec states that "The PTEs are all in the same PMD" */ + if (!ncontig || pgsize > PAGE_SIZE) + break; + } while (contpte_is_last_pte(addr, pte, ncontig)); + +out: + if (!ncontig_prev) + return; + + ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig_prev); + cont_start = ALIGN_DOWN(addr, ncontig_prev * PAGE_SIZE); + pte = pte_mkcont(pte); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), ncontig_prev), pte_pgprot(pte)); + + contpte_convert(mm, cont_start, ptep, pte, ncontig_prev, PAGE_SIZE); +} +EXPORT_SYMBOL_GPL(__contpte_try_fold); + +__always_inline void contpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + * + * Only test if the first ncontig is aligned, because it can't be the + * last pte of a size larger than the first ncontig and not the last + * of the first ncontig. + */ + int ncontig = arch_contpte_get_first_ncontig(NULL); + + if (contpte_is_last_pte(addr, pte, ncontig)) + __contpte_try_fold(mm, addr, ptep, pte); +} + +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long contmask; + unsigned long next; + unsigned long end; + unsigned long pfn; + pgprot_t prot; + int ncontig; + + /* + * The set_ptes() spec guarantees that when nr > 1, the initial state of + * all ptes is not-present. Therefore we never need to unfold or + * otherwise invalidate a range before we set the new ptes. + * contpte_set_ptes() should never be called for nr < 2. + */ + VM_WARN_ON(nr == 1); + + if (!mm_is_user(mm)) + return __set_ptes(mm, addr, ptep, pte, nr); + + end = addr + (nr << PAGE_SHIFT); + pfn = pte_pfn(pte); + prot = pte_pgprot(pte); + + do { + next = arch_contpte_addr_end(addr, end, &ncontig); + nr = (next - addr) >> PAGE_SHIFT; + pte = pfn_pte(pfn, prot); + contmask = (ncontig << PAGE_SHIFT) - 1; + + if (ncontig && ((addr | next | (pfn << PAGE_SHIFT)) & contmask) == 0) + pte = pte_mkcont(pte); + else + pte = pte_mknoncont(pte); + + __set_ptes(mm, addr, ptep, pte, nr); + + addr = next; + ptep += nr; + pfn += nr; + } while (addr != end); +} +EXPORT_SYMBOL_GPL(contpte_set_ptes); + +__always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + pte = pte_mknoncont(pte); + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pte, 1); + contpte_try_fold(mm, addr, ptep, pte); + } else { + contpte_set_ptes(mm, addr, ptep, pte, nr); + } +} +#endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933169 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=IBqn1x4e; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=ITdXruC5; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ6q4KzHz20fc for ; Thu, 9 May 2024 05:23:47 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q6qcGyucrrhLCAZYh73B20Ar789dILrfYvwQVFsS2Hc=; b=IBqn1x4endzDxF XToZubcTlza8xbnacKRBTiPfs5afhySSubulbB5ZxR+cRDTepSfeFO0Z1Kpf42kDXnG8n9SKabB9M xtq0CPmXgB9/ngzfTTLNz+IkRYDtut4fSTBiqeBS+xoPEtE3FY8jmTlJ6r4GUpYUqhauMXBm9fGde 8GMC3JujIjdPGPwDWIEZIFsPnEXf4GAXYbcndbA5CB24QOxeQ6GmoBaZhhz1JS3STITkkW2XlBDEm eefc7/z/d4RWONo+azH2KwTRjXImaAq7AM61srA0f5cTRNvUWygD55g4inx8+vrHG6JGoHq0HnYjn 1AcRRk54pDCHmtjzTdAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mtG-0000000GiMg-0lE9; Wed, 08 May 2024 19:23:46 +0000 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mtC-0000000GiK3-3acE for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:23:44 +0000 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-41ebcf01013so855135e9.0 for ; Wed, 08 May 2024 12:23:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196220; x=1715801020; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99tlcIPBJBGOnw31Oe30kKMBjt8TwvO6BCavKhZeDaw=; b=ITdXruC5GguL6WoGtm/qK61hLx6XeqZXMfbp0mUEp6cxCSRTGWlJWcD5/EuWVhNNGc uhWECY47I/rK27q/Hb28q1u1rxJnYartBFTaJ2xgDf7dPT/6Q8aS7zTenkkZ1/QvhwDI +8rvR2O4mm5zVK0slgqHXrCQfWqDGWPRbhbvvtRnIsgNnuY0artGAFnEUI3e54eosuaf ubbcZDP179cjJPz/VG5BTib/TC0nL8MxR0lNXbMJUtRb9ZknxUvgQHhIbLeVg+KMpk4v /cf5h39f64vOnWnRRl9bRwWVwAoCz3AALyN5uk30AxlDwVfgPQdzBkC7UcUPrB+4dguy d1Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196220; x=1715801020; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99tlcIPBJBGOnw31Oe30kKMBjt8TwvO6BCavKhZeDaw=; b=aXk3UXNONpl2xp/iJDXl72bfUnLgxmKT3JnS9VtgDTl+dfcbLF3fO1dyp0mo2H+3Mr nxSzWYxVe5cgUa92FPXSjwc/Z1zHaM0moRJk4SOyicY738Zd3O8hVbYtc8pYNrLRRB/k 2TXaZOYoVc2eGQNIYoVxmmDtDZJ2REjI08omljq8SE3ajSnA1MN/PqHHaOy69IeMX/j5 auq1j/5ZUC1KrDSwwPVinos+eBaOh9Gqd0/T6f9rXMZXZOsbg/TH8SW6fQaojspzIR9I P/0ENF4SqM7tsCgBxt/TFKH3o9OvQSBeUxnvgUYJ1OM1ALKxtgeggxH7hJrarDsjltds EjMw== X-Forwarded-Encrypted: i=1; AJvYcCUDRdcnU+0FF8SiorQ3awU9tU6+/qXHDfssekMv/2JiNryB77+eYNVzFGUdB3SXbPCUuBLlWaqwQ2EH2kvF+h0zqBfZrj71rfjejCpmiQ== X-Gm-Message-State: AOJu0Yx8KX4se/K0R7FrEUFBARkf+mL1tAzTvizceXY/8Xwl0MO5bNBE Fq8jGMd4BwPxQmLqNRXwtdrTPERQsOrZxpSeEBR99n0W1TIjSSXqUyeuNEX04EI= X-Google-Smtp-Source: AGHT+IFuqFkdx1GrRMX9VnSrFsQl595h33hB5u187da/mHSb1Bien42UkrHvFd9ITnGVH1r4SnsHfw== X-Received: by 2002:a05:600c:3ca9:b0:41b:ed36:e055 with SMTP id 5b1f17b1804b1-41fbcb4b4fbmr4761445e9.7.1715196220131; Wed, 08 May 2024 12:23:40 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f42e74625sm47255965e9.0.2024.05.08.12.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:23:39 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 04/12] mm, riscv, arm64: Use common ptep_get_lockless() function Date: Wed, 8 May 2024 21:19:23 +0200 Message-Id: <20240508191931.46060-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122342_973816_8BC98311 X-CRM114-Status: GOOD ( 20.64 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_get_lockless() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 11 +---- arch/arm64/mm/contpte.c | 57 arch/riscv/include/asm/pgtable.h | 2 + i [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:32a listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_get_lockless() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 11 +---- arch/arm64/mm/contpte.c | 57 -------------------------- arch/riscv/include/asm/pgtable.h | 2 + include/linux/contpte.h | 1 + mm/contpte.c | 69 ++++++++++++++++++++++++++++++++ 5 files changed, 73 insertions(+), 67 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e85b3a052a02..8a0603257436 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1384,7 +1384,6 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ -extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, @@ -1430,16 +1429,8 @@ static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern pte_t ptep_get_lockless(pte_t *ptep); #define ptep_get_lockless ptep_get_lockless -static inline pte_t ptep_get_lockless(pte_t *ptep) -{ - pte_t pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(pte))) - return pte; - - return contpte_ptep_get_lockless(ptep); -} static inline void set_pte(pte_t *ptep, pte_t pte) { diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index e225e458856e..5e9e40145085 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -28,63 +28,6 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, } } -pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) -{ - /* - * The ptep_get_lockless() API requires us to read and return *orig_ptep - * so that it is self-consistent, without the PTL held, so we may be - * racing with other threads modifying the pte. Usually a READ_ONCE() - * would suffice, but for the contpte case, we also need to gather the - * access and dirty bits from across all ptes in the contiguous block, - * and we can't read all of those neighbouring ptes atomically, so any - * contiguous range may be unfolded/modified/refolded under our feet. - * Therefore we ensure we read a _consistent_ contpte range by checking - * that all ptes in the range are valid and have CONT_PTE set, that all - * pfns are contiguous and that all pgprots are the same (ignoring - * access/dirty). If we find a pte that is not consistent, then we must - * be racing with an update so start again. If the target pte does not - * have CONT_PTE set then that is considered consistent on its own - * because it is not part of a contpte range. - */ - - pgprot_t orig_prot; - unsigned long pfn; - pte_t orig_pte; - pgprot_t prot; - pte_t *ptep; - pte_t pte; - int i; - -retry: - orig_pte = __ptep_get(orig_ptep); - - if (!pte_valid_cont(orig_pte)) - return orig_pte; - - orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); - ptep = arch_contpte_align_down(orig_ptep); - pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); - - for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { - pte = __ptep_get(ptep); - prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); - - if (!pte_valid_cont(pte) || - pte_pfn(pte) != pfn || - pgprot_val(prot) != pgprot_val(orig_prot)) - goto retry; - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} -EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); - void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ebfe6b16529e..62cad1b974f1 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -785,6 +785,8 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern pte_t ptep_get_lockless(pte_t *ptep); +#define ptep_get_lockless ptep_get_lockless extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptes diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 54d10204e9af..01da4bfc3af6 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -8,6 +8,7 @@ * a private implementation detail of the public ptep API (see below). */ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void contpte_try_fold(struct mm_struct *mm, unsigned long addr, diff --git a/mm/contpte.c b/mm/contpte.c index 566745d7842f..060e0bc1a2a3 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -42,6 +42,7 @@ * - huge_ptep_clear_flush() * - ptep_get() * - set_ptes() + * - ptep_get_lockless() */ pte_t huge_ptep_get(pte_t *ptep) @@ -589,4 +590,72 @@ __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, contpte_set_ptes(mm, addr, ptep, pte, nr); } } + +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) +{ + /* + * The ptep_get_lockless() API requires us to read and return *orig_ptep + * so that it is self-consistent, without the PTL held, so we may be + * racing with other threads modifying the pte. Usually a READ_ONCE() + * would suffice, but for the contpte case, we also need to gather the + * access and dirty bits from across all ptes in the contiguous block, + * and we can't read all of those neighbouring ptes atomically, so any + * contiguous range may be unfolded/modified/refolded under our feet. + * Therefore we ensure we read a _consistent_ contpte range by checking + * that all ptes in the range are valid and have CONT_PTE set, that all + * pfns are contiguous and that all pgprots are the same (ignoring + * access/dirty). If we find a pte that is not consistent, then we must + * be racing with an update so start again. If the target pte does not + * have CONT_PTE set then that is considered consistent on its own + * because it is not part of a contpte range. + */ + + pgprot_t orig_prot; + unsigned long pfn; + pte_t orig_pte; + pgprot_t prot; + pte_t *ptep; + pte_t pte; + int i, ncontig; + +retry: + orig_pte = __ptep_get(orig_ptep); + + if (!pte_valid_cont(orig_pte)) + return orig_pte; + + orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + ptep = arch_contpte_align_down(orig_ptep); + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, 0, NULL); + pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + + for (i = 0; i < ncontig; i++, ptep++, pfn++) { + pte = __ptep_get(ptep); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + + if (!pte_valid_cont(pte) || + pte_pfn(pte) != pfn || + pgprot_val(prot) != pgprot_val(orig_prot)) + goto retry; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); + +__always_inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get_lockless(ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933170 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=WhUkQwUm; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=Zgy4TDmC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ824HrYz20fc for ; Thu, 9 May 2024 05:24:50 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xQtmeL2+IjmRxT2qLDwN/ag5uA+whDGf1h7GFNvdoVw=; b=WhUkQwUm6liZ6u 7/w+90TjD8mdeGvmz+FxdW+CK4Od9sp2hLsD93SZ/RN9evR/H2eS6al5mTH6ZWrV27wpYF94fr5WH f805tb4jtkXT+IFpBalaHQ0ly7mmLp3068O8vG71cl6Demd+OCB43QroClR8QzF2Uf6xqem8deJvc E1GI8GslTVFXO6iiHD3KSer3W64odprmfU+fXNRv31FZjCysBpboF+8tGqiuGscHXE4XVE9YqdllV RdKMXFFvvAQ409cIXb3WWFFJuyqOiOM0n8cQ6TpkdaEdY8aFZxYyjK9J4oL8WcILlpQHmkNcCqIhc CvyDtWOg80LHJvgodSyw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4muH-0000000Gim3-18tZ; Wed, 08 May 2024 19:24:49 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4muA-0000000Giiz-2iwC for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:24:46 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-41b782405d5so799765e9.2 for ; Wed, 08 May 2024 12:24:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196281; x=1715801081; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bID6WfA+atBgi3UdotMTiBHvNsJiSSNtZP3FlzKWl8Q=; b=Zgy4TDmCtxj+2Cutmx4wp9hxmDU/ADugJTkusYzZ065bk3LJq8QyqU013AaMFZQ/jF i5DW7/IKqK7eiwfmux+0ec9OJyDkPr30oLaBcPGd9tsHuzne5+okNE+nYpAqgNszCT4P YyJVAuF7LSQ2yVyjSyxt/0/iAfAVbYXevHojKs0tzsGblRPCHSxFHF44pvXWEwFa4Fw7 h8xDqFHua5rcGQYg4Nt1pD4YDMMipRf4Es1wdrnfmwcKV1S5879dSyngiIypTdW8dsWR iLq7+okbQCAew9qDLanL1hZd16hZeVlN4BifPDK1TTcZ1YpR66v2swMNPd6glyUDT66F WEvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196281; x=1715801081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bID6WfA+atBgi3UdotMTiBHvNsJiSSNtZP3FlzKWl8Q=; b=emlIQ2fmwaYC32xjsNlU9bYDfw1Q+Y/e8gTaYpf3CxWgmlF9EZEMPXwdalJKvpQia2 1Rf3CM2JnMmD4e1gbG4/NrMM01wkRkcDctnowjCxrUIz1GOkt2SkTA6SwMbiVkiX8IBn evmbg2DpIhsJNe5zRdTGw+RT2T49f6KRW2mE5ljbZ3afRl65UBGv3mnqrDwi7LtELTK6 bzLTFlxkvrifyoCqlvYGPfoEiVjadZETOCWpJ3jNwL0VqrPw6iJUhOE/Ia8gWfHlOsdv +7+TBKf2nAEIYttWtTByErHtsYIDIpcHfQzgL+b56vBUeUpeCZN87opPa/cbxH72/gIp vYeQ== X-Forwarded-Encrypted: i=1; AJvYcCV88NCOhKZHhW8R8r+a7Z0IlBuyTPQV/GJ5kYGNTU9k4lOPQ6LS15wfA4MIH8PS6uELLvFty8HM4EXeouozmLzpm5++A1PRvWU7VAzpng== X-Gm-Message-State: AOJu0YwmIK4/XrPZYkQz1a8YCfS6O4ER+IjAsxZpdC7hIjKEUZ3Hj+oT 3pSAugGo1xiCfKHJ+YNd9+vuHjjg8seXB6dX47Ym23qS0liRuhUENmZMUrtZc9k= X-Google-Smtp-Source: AGHT+IFH7sC2GowLam3Jd+gaWz8gHH4sQsrwabW1J5xMuNqZ6dTlKzWFO1Hh7H4Mfqebgr3mIcqVow== X-Received: by 2002:a05:600c:1991:b0:41d:803c:b945 with SMTP id 5b1f17b1804b1-41f71309fafmr39869075e9.10.1715196281225; Wed, 08 May 2024 12:24:41 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f882089cbsm32567815e9.48.2024.05.08.12.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:24:40 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 05/12] mm, riscv, arm64: Use common set_pte() function Date: Wed, 8 May 2024 21:19:24 +0200 Message-Id: <20240508191931.46060-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122445_392556_C7A714C4 X-CRM114-Status: GOOD ( 20.82 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware set_pte() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 16 ++ arch/riscv/include/asm/kfence.h | 4 ++-- arch/riscv/include/asm/pgtable.h | 7 +++++-- [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:334 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware set_pte() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 16 ++-------------- arch/riscv/include/asm/kfence.h | 4 ++-- arch/riscv/include/asm/pgtable.h | 7 +++++-- arch/riscv/kernel/efi.c | 2 +- arch/riscv/kernel/hibernate.c | 2 +- arch/riscv/kvm/mmu.c | 10 +++++----- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/kasan_init.c | 14 +++++++------- arch/riscv/mm/pageattr.c | 4 ++-- mm/contpte.c | 18 ++++++++++++++++++ 10 files changed, 44 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8a0603257436..bb6210fb72c8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1432,20 +1432,8 @@ extern pte_t ptep_get(pte_t *ptep); extern pte_t ptep_get_lockless(pte_t *ptep); #define ptep_get_lockless ptep_get_lockless -static inline void set_pte(pte_t *ptep, pte_t pte) -{ - /* - * We don't have the mm or vaddr so cannot unfold contig entries (since - * it requires tlb maintenance). set_pte() is not used in core code, so - * this should never even be called. Regardless do our best to service - * any call and emit a warning if there is any attempt to set a pte on - * top of an existing contig range. - */ - pte_t orig_pte = __ptep_get(ptep); - - WARN_ON_ONCE(pte_valid_cont(orig_pte)); - __set_pte(ptep, pte_mknoncont(pte)); -} +extern void set_pte(pte_t *ptep, pte_t pte); +#define set_pte set_pte extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index f303fef8591c..36e9f638abf6 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -18,9 +18,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) pte_t *pte = virt_to_kpte(addr); if (protect) - set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); else - set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 62cad1b974f1..4f8f673787e7 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -539,7 +539,7 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) * a page table are directly modified. Thus, the following hook is * made available. */ -static inline void set_pte(pte_t *ptep, pte_t pteval) +static inline void __set_pte(pte_t *ptep, pte_t pteval) { WRITE_ONCE(*ptep, pteval); } @@ -551,7 +551,7 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(mm, pteval); - set_pte(ptep, pteval); + __set_pte(ptep, pteval); } #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT @@ -790,11 +790,14 @@ extern pte_t ptep_get_lockless(pte_t *ptep); extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptes +extern void set_pte(pte_t *ptep, pte_t pte); +#define set_pte set_pte #else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_ptes __set_ptes +#define set_pte __set_pte #endif /* CONFIG_THP_CONTPTE */ diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index 3d2a635c69ac..673eca7705ba 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) val = pte_val(pte) & ~_PAGE_EXEC; pte = __pte(val); } - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..97ed3df7a308 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -186,7 +186,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long pte_t pte = READ_ONCE(*src_ptep); if (pte_present(pte)) - set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + __set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); return 0; diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 70c6cb3864d6..1ee6139d495f 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -155,7 +155,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, next_ptep = kvm_mmu_memory_cache_alloc(pcache); if (!next_ptep) return -ENOMEM; - set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), + __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), __pgprot(_PAGE_TABLE))); } else { if (gstage_pte_leaf(ptep)) @@ -167,7 +167,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, ptep = &next_ptep[gstage_pte_index(addr, current_level)]; } - set_pte(ptep, *new_pte); + __set_pte(ptep, *new_pte); if (gstage_pte_leaf(ptep)) gstage_remote_tlb_flush(kvm, current_level, addr); @@ -251,7 +251,7 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, return; if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); for (i = 0; i < PTRS_PER_PTE; i++) gstage_op_pte(kvm, addr + i * next_page_size, &next_ptep[i], next_ptep_level, op); @@ -259,9 +259,9 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, put_page(virt_to_page(next_ptep)); } else { if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); + __set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); gstage_remote_tlb_flush(kvm, ptep_level, addr); } } diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fe8e159394d8..bb5c6578204c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -325,7 +325,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) ptep = &fixmap_pte[pte_index(addr)]; if (pgprot_val(prot)) - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else pte_clear(&init_mm, addr, ptep); local_flush_tlb_page(addr); diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 381d61f42ab8..b5061cb3ce4d 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -41,7 +41,7 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned do { if (pte_none(__ptep_get(ptep))) { phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); - set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); } } while (ptep++, vaddr += PAGE_SIZE, vaddr != end); @@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); for (i = 0; i < PTRS_PER_PTE; ++i) - set_pte(kasan_early_shadow_pte + i, - pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); + __set_pte(kasan_early_shadow_pte + i, + pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); for (i = 0; i < PTRS_PER_PMD; ++i) set_pmd(kasan_early_shadow_pmd + i, @@ -523,10 +523,10 @@ void __init kasan_init(void) kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G)); for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], - mk_pte(virt_to_page(kasan_early_shadow_page), - __pgprot(_PAGE_PRESENT | _PAGE_READ | - _PAGE_ACCESSED))); + __set_pte(&kasan_early_shadow_pte[i], + mk_pte(virt_to_page(kasan_early_shadow_page), + __pgprot(_PAGE_PRESENT | _PAGE_READ | + _PAGE_ACCESSED))); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth = 0; diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 98c9dc4b983c..d623e4fc11fc 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -71,7 +71,7 @@ static int pageattr_pte_entry(pte_t *pte, unsigned long addr, pte_t val = __ptep_get(pte); val = __pte(set_pageattr_masks(pte_val(val), walk)); - set_pte(pte, val); + __set_pte(pte, val); return 0; } @@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp, ptep_new = (pte_t *)page_address(pte_page); for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new) - set_pte(ptep_new, pfn_pte(pfn + i, prot)); + __set_pte(ptep_new, pfn_pte(pfn + i, prot)); smp_wmb(); diff --git a/mm/contpte.c b/mm/contpte.c index 060e0bc1a2a3..543ae5b5a863 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -17,6 +17,7 @@ * - __pte_clear() * - __ptep_set_access_flags() * - __ptep_set_wrprotect() + * - __set_pte() * - pte_cont() * - arch_contpte_get_num_contig() * - pte_valid_cont() @@ -43,6 +44,7 @@ * - ptep_get() * - set_ptes() * - ptep_get_lockless() + * - set_pte() */ pte_t huge_ptep_get(pte_t *ptep) @@ -658,4 +660,20 @@ __always_inline pte_t ptep_get_lockless(pte_t *ptep) return contpte_ptep_get_lockless(ptep); } + +void set_pte(pte_t *ptep, pte_t pte) +{ + /* + * We don't have the mm or vaddr so cannot unfold contig entries (since + * it requires tlb maintenance). set_pte() is not used in core code, so + * this should never even be called. Regardless do our best to service + * any call and emit a warning if there is any attempt to set a pte on + * top of an existing contig range. + */ + pte_t orig_pte = __ptep_get(ptep); + + WARN_ON_ONCE(pte_valid_cont(orig_pte)); + __set_pte(ptep, pte_mknoncont(pte)); +} + #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933171 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=C+YiBiEs; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=YrcPVxU+; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQ9D38Rtz1xnT for ; Thu, 9 May 2024 05:25:52 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Rt397mob6CeFhjLWbQllw9Af2XdtnZTQyAGqd1/VW5E=; b=C+YiBiEsh6ivDF 5+oJxAEeX3jE8na1rTKfxbB4Me/nB3V/W4i1FuVQ2RjNse50liINkYj2N6U7vyoEgy/G7HrOTv/em G3T6AlHQ0hsw6QeW98ginXk6B15LgdZgv3DSOaNMnAVIagYIiYFkCdyaeG7CVr0cvgDSL3lBxDDW4 aESWT1Sv+WOHM3tYwTaVlaRNriAlJyqWUHzPnDAEgM8h3tL7FKdTwlG3vlRwL9pNQAM5M1V18utO8 iB9NDlGtO2b1yWSGUV1e2j7nsR1aI8rQol+MAbf4ZbKrwPL1qWYhSw7Is2qwjNplNGScmHQDS1D5d oUW2RSNal1+NbkozhzFg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mvG-0000000Gj6y-3crL; Wed, 08 May 2024 19:25:51 +0000 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mvD-0000000Gj4S-0rty for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:25:48 +0000 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-34c8592b8dbso32485f8f.3 for ; Wed, 08 May 2024 12:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196343; x=1715801143; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nb289X82Pwy2psbtYSkuOYlUdcLMn+kmFJiTkQU5Z+c=; b=YrcPVxU+LPzZiUkNNM6fu0m6FTOQSJwDMXWX4JJ6UykoVzy+Zc/Flra/QiMcSJ5uTJ YprDNtkOrJNabA4OPYHK3OaMWqXNlvjKaSw4K1pw0AtmUBOLhVy2WZvMRsQaFkXUA45J juEEq9iG7Oi3HZE6/VcNc7uAOCAr6i3JPafyI/xBVcQCqFrrnWxl6Bb5GNLc8cvB0/Mc rA61xTFTtNAOBnvItXZrLg6kcpjS4NpBvy1H8hHHk/fbP4O0ljKsvRTm+FJU8Rlr4jDY QNOIvRkHF7vXSHqjEyDmrRdWtfOD+/m2qTs3BqFpxmEmyTm63QXhxCP1A0tYqxY8qGuB WnOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196343; x=1715801143; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nb289X82Pwy2psbtYSkuOYlUdcLMn+kmFJiTkQU5Z+c=; b=w0AzBvui3m7PeDkekzy2J6sf+fIsN+cGhfsu6JAA8OsZKrVzRgEmWGvabIic/MPk6O fAzSaze2v7Ghu0ofoeikyjPY4F/f/Cwo0QgoUtjgyN5fjckXxGsqbJiAaX687831nMM0 xQV/pfrOCgGL1HnKu1G8DwQflA7GxCQjR4cO4FoCSolIes2xZy1+yXWdu//raNdVr0gj RQNmT4ikvvX3zHV+W49HMeWOjnD7/bK1g4vhKlKS9OMDK6FiWZTQ5wj45oT2ZWB/0jXb toWhJq1mcbFxeSko800jpQeIH5UrUWnCht/bWAJc+VCTgwPPj0XiVvFui4+S8arvg0dT v7Ug== X-Forwarded-Encrypted: i=1; AJvYcCV+BuoAuiJxECxCrm5nmFDmlvdKEpJ8vHo5ysUkw0Uyis7QTTK+tuwKPxoCZmspGCEGElV1w8spm0GFSUShWXyQa/DJTDwjPfcefZH9cA== X-Gm-Message-State: AOJu0Yzpb7gfAajHpVMMmBHL/ziPLgTtglc4hnXKrge3yvC7IGdiN4+j s0f1eNYizDLunWTeoua2tuOZqV8PYldePUH8OrU8kBcXbszNA7pDCOKjY05upj0= X-Google-Smtp-Source: AGHT+IFUGL9dDH2gwgI9tPWACeaOFPW71lxlPF9m81I7+sLroC1dBTrL+wbzR1KIUMB3+0Tz7vFgZA== X-Received: by 2002:adf:a492:0:b0:343:a368:f792 with SMTP id ffacd0b85a97d-34fca621315mr2849241f8f.52.1715196342717; Wed, 08 May 2024 12:25:42 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b12-20020a5d4d8c000000b0034e65b8b43fsm14038517wru.8.2024.05.08.12.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:25:42 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 06/12] mm, riscv, arm64: Use common pte_clear() function Date: Wed, 8 May 2024 21:19:25 +0200 Message-Id: <20240508191931.46060-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122547_266436_A0BC1319 X-CRM114-Status: GOOD ( 11.36 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware pte_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 9 +++------ arch/riscv/include/asm/pgtable.h | 4 +++- arch/riscv/mm/init.c | 2 +- mm/contpte.c | 6 ++++++ [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:42e listed in] [list.dnswl.org] X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware pte_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 9 +++------ arch/riscv/include/asm/pgtable.h | 4 +++- arch/riscv/mm/init.c | 2 +- mm/contpte.c | 6 ++++++ 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index bb6210fb72c8..74e582f2884f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1439,12 +1439,9 @@ extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); #define set_ptes set_ptes -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __pte_clear(mm, addr, ptep); -} +extern void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +#define pte_clear pte_clear #define clear_full_ptes clear_full_ptes static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 4f8f673787e7..41534f4b8a6d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -792,18 +792,20 @@ extern void set_ptes(struct mm_struct *mm, unsigned long addr, #define set_ptes set_ptes extern void set_pte(pte_t *ptep, pte_t pte); #define set_pte set_pte +extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define pte_clear pte_clear #else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_ptes __set_ptes #define set_pte __set_pte +#define pte_clear __pte_clear #endif /* CONFIG_THP_CONTPTE */ #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear -#define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index bb5c6578204c..c82f17b3060b 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -327,7 +327,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) if (pgprot_val(prot)) __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); local_flush_tlb_page(addr); } diff --git a/mm/contpte.c b/mm/contpte.c index 543ae5b5a863..c9eff6426ca0 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -45,6 +45,7 @@ * - set_ptes() * - ptep_get_lockless() * - set_pte() + * - pte_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -676,4 +677,9 @@ void set_pte(pte_t *ptep, pte_t pte) __set_pte(ptep, pte_mknoncont(pte)); } +void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __pte_clear(mm, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933172 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=sR54JYYo; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=vjk3wQw6; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQBT3zXwz1xnT for ; Thu, 9 May 2024 05:26:57 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=r8gJJwbrDmgCgo80cIEuarUnUnFNQun2ft6XUiJ5tHU=; b=sR54JYYo11wruQ g4Hp6+cqRNxitxnHdX3XD3uhVcpZAsOmbG/x4j5YheUT3OvBxBgn3VKuQjEmYFhFtIpBLlWERU3Yr h7AkNxcDcHZd2ZdW3QV/poDgOLYncJgDbczz0pXCeuyXTDt99kIGBsOkyIbOimJ4zRIYwzPX/RrAq xuBjHjYJk9KpeP2mMbzGrUMkzjKM+IHve8kMJsLB3bFg86hil9SFY2Qx1LQVCIUKTGY7CY4laYWzQ dOtRws618r9zOrgpwhO5LOSoXH87IO3ie2lFSQk8OLU+o1n6ds8Z5kSA0NVddPBcsKG+3hMj0L7Mb cM+0r6n/tUuftmDTCi5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mwK-0000000GjX3-1NfI; Wed, 08 May 2024 19:26:56 +0000 Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mwH-0000000GjRN-0Ejf for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:26:54 +0000 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-34db6a29a1eso44978f8f.1 for ; Wed, 08 May 2024 12:26:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196404; x=1715801204; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7NXxlYw1RM9+WdFNtDaqsh0kbg9AU8PQDaN/vzezo7A=; b=vjk3wQw6TBu7alB+EL4idt6qj88jT9SM/dj/g4vEueM7S5l5lAH/eAK+EvYnhzpVGD n268WIDOSCx7nwsmmv/aQe7auaDfoyBrDOwjbbebycdZcg5/m71YqxozH+HzNtGoupxK cmgvtTu1EnJllVmmoRCwnqIaMkqZB3I9qleB8mxI0ctSWcEZvU1OdjUCNjCTHJHgyfMS ysa2nVResSkMLjX9cNwnEERebQxpKmrf4CYbW+ZyGkCAl64iQhbLx9h/EqjBRFTunU4I STOg+ZkoVPr+Asngi+OK3uLKNvc8MYp1cajzoZ5QAMV9/93eQEPBEWhAIu/TaZWssts1 izww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196404; x=1715801204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7NXxlYw1RM9+WdFNtDaqsh0kbg9AU8PQDaN/vzezo7A=; b=ILL6UT63Z+owsWiw40MKKGO3xwadQsfpxYRwDFmdCo70kKDZcB74E7hFiXInC9G5PY Bm9V/jXfPZ5KyDg6Ex+W1qPHKsfc4d8aKyJdGyOYNJmkYE0+TjPbRq/hXURVlgvUPhc/ FrTGCAQqFddC7CIpcUSA5KRcISFu47psSSZWp2Wd7yqT/rrw6T+CYHnlo5S/irSdkLQt JDTDVcxc9uA/JPBDZjp5Gcg7CFI8zZnssRiiXS+txn/v8+zeYjuzYrQ8NZPUcsQ4JwLp saJ83mgPmvlSbtVD5AHsNEwue341EVY3kt3SUe9Ir/cSESrqBZWa1OHm4JGnwRgudV04 +51w== X-Forwarded-Encrypted: i=1; AJvYcCUJpXSfxtOdM3S4awdQZldR5fEqRqTerFRZK3hrS6559eBVD1DughW+HE1IGHOyeFbToZntKljf4gJ1JHhOwij2NAvzWdgDYtCEfxyOhA== X-Gm-Message-State: AOJu0YyMe1jixYO/8v0usYV3W99hID63mse8aeO0Vn6uUK9PH6cHYYt0 OhQBF1T2WTNdIjGs2rAS6a74Q37PSB0++Cu15cc70gPYApKzyq8YHRh7Zt8w7yk= X-Google-Smtp-Source: AGHT+IGACzzezThLxVMxwcYVQ6pqU/BhrbSjplAZdXD5UPpMsvfoJ1LwcNLet4jFhPFrbMRRJqyyWw== X-Received: by 2002:a5d:4522:0:b0:34c:bb79:452b with SMTP id ffacd0b85a97d-34fca62159dmr2733749f8f.52.1715196403826; Wed, 08 May 2024 12:26:43 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id o16-20020adfcf10000000b0034b1bd76d30sm15921429wrj.28.2024.05.08.12.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:26:43 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 07/12] mm, riscv, arm64: Use common ptep_get_and_clear() function Date: Wed, 8 May 2024 21:19:26 +0200 Message-Id: <20240508191931.46060-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122653_128717_344DB615 X-CRM114-Status: GOOD ( 10.47 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_get_and_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 8 ++------ arch/riscv/include/asm/pgtable.h | 7 +++++-- mm/contpte.c | 8 ++++++++ 3 files changed, 15 ins [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:430 listed in] [list.dnswl.org] X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_get_and_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 8 ++------ arch/riscv/include/asm/pgtable.h | 7 +++++-- mm/contpte.c | 8 ++++++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 74e582f2884f..ff7fe1d9cabe 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1473,12 +1473,8 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - return __ptep_get_and_clear(mm, addr, ptep); -} +extern pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 41534f4b8a6d..03cd640137ed 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -794,6 +794,9 @@ extern void set_pte(pte_t *ptep, pte_t pte); #define set_pte set_pte extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define pte_clear pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +extern pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -801,11 +804,11 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define set_ptes __set_ptes #define set_pte __set_pte #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -#define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT diff --git a/mm/contpte.c b/mm/contpte.c index c9eff6426ca0..5bf939639233 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -46,6 +46,7 @@ * - ptep_get_lockless() * - set_pte() * - pte_clear() + * - ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -682,4 +683,11 @@ void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __pte_clear(mm, addr, ptep); } + +pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933173 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=ZTxecL6L; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=biaup3lX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQCZ07p6z1xnT for ; Thu, 9 May 2024 05:27:54 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JWswr4N8Mmbyh77fm+zo5Regb7Hk7Ka/vvNyfnV1CJo=; b=ZTxecL6LvYasn9 SwV0MszEZsG/aecBuESqOAV8wQ/jxqzYoUM7fg0yX08lvGAZZb89wUEuta6fMjFx3b1V0H4yL//F6 IR0/n+gowS81o76eRfG89yMvNomcuHIlo4qIhBc6cAebWKmUgMDTKyBlMgdpSFcgBcSVOdh3O9LuK d7xefV5NrUpQHs/JyYT5O0ghUWqC+L/X1kE7XOXSMui0dTG6686JyS+i6TMJhYOKywxDkmZdEZAR+ Wr8U/xBRrLsYsIIr8MY7DDu5RYlEstw0d9zKuyIurh45ZnyupIYt29dBsHzbtZN4529HrJJNNxsTM E7PfFC8h/Xf6znlCePLw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mxD-0000000Gjt4-1NuS; Wed, 08 May 2024 19:27:52 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mx9-0000000Gjqd-4BdF for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:27:49 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-41ba1ba55ffso795195e9.1 for ; Wed, 08 May 2024 12:27:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196465; x=1715801265; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jcfQWqIQyCa6XQTZ9KciN3KqArzYv/m686IwSCSxNEA=; b=biaup3lXOls5HVRQjMhn8UsqPRxZm2xe/UYlhZLeNfWVvuiYOWOoB/pUDEhtF+EngV WquRkLfgQrZRLoOX/8adAApDztI5D+imQjA3cFpDNdqJ+PI3hjRHdry55ePRyAowuCwA 30IkXgxROlafJIjx8mn6ST1Oqo7KqJ6q94Yg34GQcdp+rfEfnESf5rC4xetHH2LEJ3xk yxZZF5rE9tlsRQmBkN9cSX6np0SZQIn4kmeOmDUzz6gA3UKD2ITGmHjZz1Cz9AcLVHWh frxxfEJek40/mX3X9tZ0Eh7sZwRRwcDvmdYddfZRIxLQHNtuFDaU5t/AndOMml88jWnm MRCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196465; x=1715801265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jcfQWqIQyCa6XQTZ9KciN3KqArzYv/m686IwSCSxNEA=; b=IoQkyo898qG/QtqmJ46WjlX1FKX0/PpfCxiUfTUYXyNhdtxYP8St/3ExVfTl3yulIR be1GjhhgVXXhAKxxAX8HTpr6JvrMeRA2kzsQYsBgYATUFVFMCa067u8ysYT4p7jg25HS QxG5WxS4cMhQQ8qsJmqFsQB/iANC6Ij6PeDBhHvT06dmaMpjUuZY99lst1KkT7Ka0T2T JAuB6ErE92BZ03IoMhO2vu+tLEDt8UHP4qb3TnZO3WTtLtRHwzFRTEy7LBURLoeyRygn zOrbnekGzeh/A9Ohteh8zMcisZ9fsyAcJMJWk6xjyVTy06kZiQH+P2Zx19moE1WGzV/2 O7Ew== X-Forwarded-Encrypted: i=1; AJvYcCV0urnPHPAYU7Wm0PSkusc8IvoM8N9kdb0DenE7EndeFilc+i2RI8Rph8lxuZWt7IljCOt3+XywdDrbgEHz1sWXwOub+RixHjPIqrfvRg== X-Gm-Message-State: AOJu0YyMlZMUzGxcX7nXLD47FVGKZKh7314+MzbLc9zxnH5DIBCEduxF zucqeQL7IWSWgVx9XkWQLxENMU/kN58v0d5Ppdr74dkzQBAYYvnF9Ch36mH7Ml0= X-Google-Smtp-Source: AGHT+IG713uLBQWbwM0MRsX1LOPSAYD3hwSeYc9+OS0/mfL1VayM0FBzmqDZCPlWciEj7G1Kaomx7g== X-Received: by 2002:a05:600c:3103:b0:41b:f43b:e263 with SMTP id 5b1f17b1804b1-41fbc12bdcbmr5274575e9.0.1715196465070; Wed, 08 May 2024 12:27:45 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f87c24f8fsm33175985e9.15.2024.05.08.12.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:27:44 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 08/12] mm, riscv, arm64: Use common ptep_test_and_clear_young() function Date: Wed, 8 May 2024 21:19:27 +0200 Message-Id: <20240508191931.46060-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122748_073274_A798BA18 X-CRM114-Status: GOOD ( 16.49 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_test_and_clear_young() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 ++ arch/arm64/mm/contpte.c | 25 arch/riscv/include/asm/pgtable.h | 12 + [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:336 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_test_and_clear_young() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 ++---------- arch/arm64/mm/contpte.c | 25 -------------------- arch/riscv/include/asm/pgtable.h | 12 ++++++---- arch/riscv/kvm/mmu.c | 2 +- arch/riscv/mm/pgtable.c | 2 +- include/linux/contpte.h | 2 ++ mm/contpte.c | 39 ++++++++++++++++++++++++++++++++ 7 files changed, 53 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ff7fe1d9cabe..9a8702d1ad00 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, @@ -1477,16 +1475,8 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_test_and_clear_young(vma, addr, ptep); - - return contpte_ptep_test_and_clear_young(vma, addr, ptep); -} +extern int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5e9e40145085..9bf471633ca4 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,31 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - /* - * ptep_clear_flush_young() technically requires us to clear the access - * flag for a _single_ pte. However, the core-mm code actually tracks - * access/dirty per folio, not per page. And since we only create a - * contig range when the range is covered by a single folio, we can get - * away with clearing young for the whole contig range here, so we avoid - * having to unfold. - */ - - int young = 0; - int i; - - ptep = arch_contpte_align_down(ptep); - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - young |= __ptep_test_and_clear_young(vma, addr, ptep); - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_test_and_clear_young); - int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 03cd640137ed..d39cb24c6c4a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -739,8 +739,7 @@ static inline void __pte_clear(struct mm_struct *mm, extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ -extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, +extern int __ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, @@ -778,7 +777,7 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, * shouldn't really matter because there's no real memory * pressure for swapout to react to. ] */ - return ptep_test_and_clear_young(vma, address, ptep); + return __ptep_test_and_clear_young(vma, address, ptep); } #ifdef CONFIG_THP_CONTPTE @@ -797,6 +796,9 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_GET_AND_CLEAR extern pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +extern int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -806,6 +808,8 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +#define ptep_test_and_clear_young __ptep_test_and_clear_young #endif /* CONFIG_THP_CONTPTE */ @@ -987,7 +991,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 1ee6139d495f..554926e33760 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -585,7 +585,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return ptep_test_and_clear_young(NULL, 0, ptep); + return __ptep_test_and_clear_young(NULL, 0, ptep); } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 5756bde9eb42..5f31d0594109 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -18,7 +18,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, return true; } -int ptep_test_and_clear_young(struct vm_area_struct *vma, +int __ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 01da4bfc3af6..38092adbe0d4 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -19,5 +19,7 @@ void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 5bf939639233..220e9d81f401 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -47,6 +47,7 @@ * - set_pte() * - pte_clear() * - ptep_get_and_clear() + * - ptep_test_and_clear_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -690,4 +691,42 @@ pte_t ptep_get_and_clear(struct mm_struct *mm, contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); return __ptep_get_and_clear(mm, addr, ptep); } + +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + /* + * ptep_clear_flush_young() technically requires us to clear the access + * flag for a _single_ pte. However, the core-mm code actually tracks + * access/dirty per folio, not per page. And since we only create a + * contig range when the range is covered by a single folio, we can get + * away with clearing young for the whole contig range here, so we avoid + * having to unfold. + */ + + size_t pgsize; + int young = 0; + int i, ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + young |= __ptep_test_and_clear_young(vma, addr, ptep); + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_test_and_clear_young); + +__always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_ptep_test_and_clear_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933174 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=GyqFZeJT; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=SIfVnHkC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQDg6jWZz1xnT for ; Thu, 9 May 2024 05:28:51 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6O587SGlsYbHpYBL7jL2MtJgyFRrQIaqIeTyMVgZDaY=; b=GyqFZeJTLarFm6 lemw3K28QdQpuZpNmX/S/YYrcEtMgwerqFjdwV7A4qCPBi6bsPwHjR3fNl3DHYE9YgHXRSN8Xqy7i PLlzesstmxPMHGmeA0qGptSLj8Zc7YzKMLUH2zJFyPB3Ai+HrGpljdMVbKeKQRZ4CAW2l0AHFOLRP JrMDkYuZp+JYTy6msxOzQfQYjKEXSLiaBqt+y6rXDP079LeuMriew5mRuEgWbDgBjLhcjq13AHmXR HalppTebsvAYgD08de84PDPSFq/J4I26J7M1KNOhdSSGkXi++tnenS0Yk+0gMnEVgjkWINM5w0FSa auFBKWkppc9OiT8SPcMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4myA-0000000GkCt-2bZa; Wed, 08 May 2024 19:28:50 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4my7-0000000GkBM-2ohB for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:28:49 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-41b794510cdso601655e9.2 for ; Wed, 08 May 2024 12:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196526; x=1715801326; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=SIfVnHkCie+hKEQP1F11dwW6kROYkdvWaUxuxpBoppwOY/guxPVLWdVta2q9ujrRM9 ZJKoH1kPSMkqErmAv4gefMavdTGZDksQfDKr5/CBfuAb6tmkz9Sc2BvDAz4fl6QMVSFP 2v4lhwYRo90MZIKQ97SToHSnOhE2kg4fYyyXNUHj8EFAylmH37WjzL2m1AE7urJ6IgwX mWv2+qdvpXIlZpGEcIYUXqlkbxzsFG3j5WPGLTIKAYzRTekRaGZNhpWerBILSJ8K0PBA 1/lBsEPoTzXQMzjf0wpDIRZmhB8c6wiL6s5nABSgECoJhoAqfnEHkDT1+OmSq8isBeUc mFlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196526; x=1715801326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=Cz/YIobYMnEtfLxFn4lz7pdHiXKHwi0sfgJxfkeFh2pUli1VFxAU0oQgms9WzxC3Ig Tg+LZ1SbezSA6PUDa4qYDvjumrWhGJMaRfy+aJeOyehE0RIh4rk7htNLbDNWtIJogggA EsxCgSwCoZLUoU2VrpNSWQl+UI0Mka6KQ6t7aGRsF1oF9MwYudavvfLARtkzGU3h7pAD GCxXXWMUtkRAs9qP+zCKCFWVCOrqSy3X9VjmMyaf1C7AV41102sRWOqInjh4AiYiG/GA AbDgfOenUf2eMtzWl4ryt/EQWnZJi38Fp5IKMsLrbFwRWR/k+LMBhGTZK8CDKKnKd2dL QTPw== X-Forwarded-Encrypted: i=1; AJvYcCVkakYCP9xVhcgBxr/R9Y/xxn0qG6ae5ImMUHuZ2e1fMi77AiqUNRF6Ikrcja8JUd4ZlxnjE/HfgBGTUdJXeUZERLcrtE7b9zru+WxgiQ== X-Gm-Message-State: AOJu0YxmRetBwrRnS9UaDDLGqr3LIc5pd4NZAyodWz3swJwcSyfhQooF LCJwqsHxugj59hxNsUtopmw5Z00naNVjbW9qjf0I044nKnOjl39cz7KhafpV41s= X-Google-Smtp-Source: AGHT+IEyIjL1IsEMqXGLkJGiy2tkR0X5tVHC2TKSU1+3Y1r/Xzm4QA5qwQ487USNifryKvOLnO9vbQ== X-Received: by 2002:a5d:6350:0:b0:34c:65ba:5d43 with SMTP id ffacd0b85a97d-34fca621699mr2523239f8f.46.1715196526270; Wed, 08 May 2024 12:28:46 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id cx8-20020a056000092800b0034e01a80176sm16002694wrb.114.2024.05.08.12.28.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:28:45 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 09/12] mm, riscv, arm64: Use common ptep_clear_flush_young() function Date: Wed, 8 May 2024 21:19:28 +0200 Message-Id: <20240508191931.46060-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122847_742194_3A21F19B X-CRM114-Status: GOOD ( 16.74 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:336 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 22 ++++++++---------- arch/arm64/mm/contpte.c | 21 ----------------- arch/riscv/include/asm/pgtable.h | 12 +++++++--- include/linux/contpte.h | 2 ++ mm/contpte.c | 40 ++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 36 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9a8702d1ad00..92c12fb85cb4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, @@ -1479,16 +1477,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_clear_flush_young(vma, addr, ptep); - - return contpte_ptep_clear_flush_young(vma, addr, ptep); -} +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes static __always_inline void wrprotect_ptes(struct mm_struct *mm, @@ -1616,6 +1606,14 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range(vma, start, end, stride, true, 3); } +static inline void arch_contpte_flush_tlb_range_nosync(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range_nosync(vma, start, end, stride, true, 3); +} + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 9bf471633ca4..16940511943c 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,27 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - int young; - - young = contpte_ptep_test_and_clear_young(vma, addr, ptep); - - if (young) { - /* - * See comment in __ptep_clear_flush_young(); same rationale for - * eliding the trailing DSB applies here. - */ - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, - PAGE_SIZE, true, 3); - } - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); - void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d39cb24c6c4a..42c7884b8d2e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -615,6 +615,8 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, flush_tlb_mm_range(vma->vm_mm, start, end, stride); } +#define arch_contpte_flush_tlb_range_nosync arch_contpte_flush_tlb_range + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) @@ -758,9 +760,8 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) { /* * This comment is borrowed from x86, but applies equally to RISC-V: @@ -799,6 +800,9 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -810,6 +814,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 38092adbe0d4..76a49ac8b6f5 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -21,5 +21,7 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 220e9d81f401..600277b1196c 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -48,6 +48,7 @@ * - pte_clear() * - ptep_get_and_clear() * - ptep_test_and_clear_young() + * - ptep_clear_flush_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -729,4 +730,43 @@ __always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, return contpte_ptep_test_and_clear_young(vma, addr, ptep); } + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, + 0, &pgsize); + + addr = ALIGN_DOWN(addr, ncontig * pgsize); + arch_contpte_flush_tlb_range_nosync(vma, addr, + addr + ncontig * pgsize, + pgsize); + } + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); + +__always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933175 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=mgDexMWY; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=iLUER6Ym; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQFx4ZcWz1xnT for ; Thu, 9 May 2024 05:29:57 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=geRtwi7fb+LweqCYyqEu5uJoNdmcqOzan07Ov8TxwjM=; b=mgDexMWYfz9Pci VwrVOUnUpiARCKRd/3MGF5H+6pxTIIlAYlT/hGQiMsadMMYOTw4vxQR3KEAnejzhhy/3usEp2Vkb4 ZPzQ5TRPXI6XPgMOrDOEN+GhH/6+8LqjtFjoFo4lbxMO4r2nas+Fw+3fwkXHFNoXkPmoPOslrlM9I rXim9SgiYnE5RpDJdqIFTjat68mpuHAIR6aqTjGrkg259aIVdp6k/98TdrrHfABZtrgSrrO+3YzR7 DvidDqBomNYbcmilmZzzAyux4zsN6td+1dmPmUrzKXXiJAY6EkcYWQheDd0CzFGuRCw2KOFDR+NQ2 192jO3dh8wBCGG2dm4PQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mzE-0000000GkaY-1XrB; Wed, 08 May 2024 19:29:56 +0000 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4mz7-0000000GkX5-0nnp for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:29:53 +0000 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-41b5dd5af48so738595e9.0 for ; Wed, 08 May 2024 12:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196587; x=1715801387; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=iLUER6YmnJUGR0/ODaBehjGEsKRbIQR+z994kMSjTjLmm+/iy7FxYeK8uLRSqjBPQp 4T0G9TC9elLg+cquOgU6SGhB3jhYEmdY4VHh6HD7paIffIzabpmLJP7n1tjNMs1/PBL4 gH+/OTjWdlFdoP/NnsFTR1pC6HrueOegYDyd4ZUHRhE3HA8i1m+kfoQA9CXxNm8H/gAw /Md/QVxg9mBioGt7oi2ZYbB17la9INZlTJEILACebxbIoxR6mFWh3V8MYNqZdTf0Y6/W trs3QtTsEArLUtgyMhwNn84hdxgFFj1LV1iRKU0f7G84wyX2SESeytbX3yuuReub8zW4 wx2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196587; x=1715801387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=NnaizTEMcpzlIxVSbRqAfzZe1vfASjZrpCd5ohGMWmE9B/yGQivkO9TouEPiScCZ3z lP8rH7bMwphUuWkpbTpRDoYmRCgeRs7dzgrri8hijoO+gAHBcjFd8hqstmbYwNciZK0j 1GklLa7Sqb4K6715Xj7/jlsR21tnfq4iOIjV3MSR2U3JO1wwe1Z8BZdbiPtjThXotz1n bVNo/OFlzY8bUqdbuywG9nfLURCnqHowR7KwuSQwGDpTCjkRwYbqRZk0iuuZWluLPKOK 8Ek3iXhqATuBYrzF1y7nNNzKd8WWhGoODoJi+itBOqF/Ki/LQywrwdDizpjMtQYaH59V PmwQ== X-Forwarded-Encrypted: i=1; AJvYcCXu9Q1eRYP1ZzoBfa35eiAynpVFgw9TQrbWj4NaZOdG2b8uNcL581iJtZ/E+bonFqZo8tGmNBc0MB13EqRiqDOXA6xqOnTxZa32Re687Q== X-Gm-Message-State: AOJu0YyVX+fFMbQVBrz9illuRU24Ji1bP8nmQxQ3Nwx3v8iz95j/l306 HeGZsPBuKulWtvj4z5acSfQ0TEPST9qGlsK6zewDPoFlS9gwxR8OHan3jU+RIr8= X-Google-Smtp-Source: AGHT+IGQ61Vb4NuwkawA/0/DWqDCfOJMo7+D5f/tM30tLGXtIG6xkHE311QLb6FnYOpUeATe3M4jpQ== X-Received: by 2002:a05:600c:4e93:b0:41b:fc3a:f1ef with SMTP id 5b1f17b1804b1-41f71acca18mr25217385e9.33.1715196587426; Wed, 08 May 2024 12:29:47 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id d16-20020a05600c34d000b00419f572671dsm3314921wmq.20.2024.05.08.12.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:29:47 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 10/12] mm, riscv, arm64: Use common ptep_set_access_flags() function Date: Wed, 8 May 2024 21:19:29 +0200 Message-Id: <20240508191931.46060-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_122949_685954_58A896EE X-CRM114-Status: GOOD ( 20.26 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_set_access_flags() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 19 ++-------- arch/arm64/mm/contpte.c | 46 arch/riscv/include/asm/pgtable.h | 10 [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:32c listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_set_access_flags() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 19 ++-------- arch/arm64/mm/contpte.c | 46 ----------------------- arch/riscv/include/asm/pgtable.h | 10 +++-- include/linux/contpte.h | 3 ++ mm/contpte.c | 63 ++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 65 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 92c12fb85cb4..6591aab11c67 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1391,9 +1391,6 @@ extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned int nr, int full); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); -extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1512,19 +1509,9 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -static inline int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - pte_t orig_pte = __ptep_get(ptep); - - entry = pte_mknoncont(entry); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - - return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); -} +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 16940511943c..5675a61452ac 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -62,49 +62,3 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, __wrprotect_ptes(mm, addr, ptep, nr); } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); - -int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - unsigned long start_addr; - pte_t orig_pte; - int i; - - /* - * Gather the access/dirty bits for the contiguous range. If nothing has - * changed, its a noop. - */ - orig_pte = pte_mknoncont(ptep_get(ptep)); - if (pte_val(orig_pte) == pte_val(entry)) - return 0; - - /* - * We can fix up access/dirty bits without having to unfold the contig - * range. But if the write bit is changing, we must unfold. - */ - if (pte_write(orig_pte) == pte_write(entry)) { - /* - * For HW access management, we technically only need to update - * the flag on a single pte in the range. But for SW access - * management, we need to update all the ptes to prevent extra - * faults. Avoid per-page tlb flush in __ptep_set_access_flags() - * and instead flush the whole range at the end. - */ - ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - __ptep_set_access_flags(vma, addr, ptep, entry, 0); - - if (dirty) - __flush_tlb_range(vma, start_addr, addr, - PAGE_SIZE, true, 3); - } else { - __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); - __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - } - - return 1; -} -EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 42c7884b8d2e..b151a5aa4de8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -803,6 +803,10 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ @@ -816,11 +820,11 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -#define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect @@ -990,7 +994,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); } #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76a49ac8b6f5..76244b0c678a 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -23,5 +23,8 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 600277b1196c..9cbbff1f67ad 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -769,4 +769,67 @@ __always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + unsigned long start_addr; + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = pte_mknoncont(ptep_get(ptep)); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold the contig + * range. But if the write bit is changing, we must unfold. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + size_t pgsize; + int ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + start_addr = addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + arch_contpte_flush_tlb_range(vma, start_addr, addr, pgsize); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + } + + return 1; +} +EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); + +__always_inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933176 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=WhhUjBLw; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=QcUpKHm2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQH648twz1ymg for ; Thu, 9 May 2024 05:30:58 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5d7vvLf5WtZie6Rj43PHvtjASbGwWif+DsEaBlywI+g=; b=WhhUjBLwQfEXb8 jmD5JdZVF0Eh0t1C7CJfUjQRi2gPYVbKrI3p25wjEK0kFSpAGORL7ztLc4Vfd6BNcsHOLFR7r/agK PNJdP6NoZN4cisJB1/DOQTLnxW3uIhuj7BxEiJQTxqtDgWGzJHV9lgShi2c7But3kqK5ssgVqHe1U DoOpYrMJEuT4p67A4dV9/MrxDFYt1Yv0MBhXFmENYRfcdmwi2cCt3KqsXC49pPduUbmuZeAjbevIm EvUGJO2MLIh2gI+orWq2BGEKr32XXZEaYU/T1bYtBTS8/WPJtxvCiS3X1F9psO4tSkVQsMo+vKE91 ljuEhTOl2ZhipLfVhN8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4n09-0000000GkvO-1F84; Wed, 08 May 2024 19:30:57 +0000 Received: from mail-lj1-x22d.google.com ([2a00:1450:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4n06-0000000Gkta-1Nu4 for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:30:51 +0000 Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2e4939c5323so1628671fa.2 for ; Wed, 08 May 2024 12:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196649; x=1715801449; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=QcUpKHm2C5Z2YEYsURvZ2S9P8DWZ+oF7mecbKm4winzL5yxvqDa4SaaWII8IfClolv DtQ6b2vJFSZq8f2v88dyfIu9QBmsV0Y3l5McxObwuT1cU3qsTi2dowzTx+pWtGqhm9C5 vLl7u3010bmgwLsz3oAifR3MGj8pl70JHS9d8596khXbKrMRgqd8ZLnIWo91aXLJdz8g L2OU+wI8mHgXA7yPO23dsOHInJJfoqgcuyh8Rm5ZxiYew/vHnKAr9P+ekJpb80ayjTU6 unBnzUVrK9gND4+NTBXC0D0xlmr/AHhpDfKFt1WIbXjlkspXmr+HX1dMAoLhfi4t1Ya4 bgpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196649; x=1715801449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=w48SCO3PZQJzr1jRSZ0ri65LB5OJqOzl57MnqzJ+eqrU7MQMx/wSp608YpKaFvFtaB JuUuJWZTxl5/2bI4ZS6NrvSg6OwD74Muhs8t2C4cqWxgsN3Ex2X/xGwC/t57mopY0Qst 8FMDnCTTTykTz4x+HiMDBRvSFdh/IoIfMWFwb3HiCKKNhB82HeSGKAT5HpMVBu1MCsNC 6O/hXHInXFZyv2eYk+2ymT9HvQnQGwYpDppEqyWWC31DZ82msXftrB9zGLFXPCSphXYz b0/UIuL9tqvupw5SAQ107VYXqSpGFv++rYaDC+n178ToVSalHCZ5JlQW7U/8sPNlyrJh 62DA== X-Forwarded-Encrypted: i=1; AJvYcCUEmxTb2NJ3SUrZ9qulgglDdCKYp23CxtVbv9PpgTMzVtH0mdgr7bxKGEC34jfmbww44lTchonb1485ilhxnedvCk+oV00feCb51rn8Gw== X-Gm-Message-State: AOJu0Yz3r9uRqjIBQ5U8xSO+EbEwj5eVE4pP90+QjA3h+yHwVEtcOTSS Jl8td6CcuoE+cLuDZJxIbn64KAofIj6S5qD8sRERU53Rv1YXAN04rnvhg5ETI1s= X-Google-Smtp-Source: AGHT+IGXrMpV3e8NWTuY3a54mvQHDm5XG0kBaIk0g0HXiaoXOXDbgZBz/5WZyQXp8WNGVe1a2PspPg== X-Received: by 2002:a05:651c:1541:b0:2df:e192:47ec with SMTP id 38308e7fff4ca-2e447081ef2mr37890971fa.29.1715196648606; Wed, 08 May 2024 12:30:48 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b15-20020a05600c4e0f00b0041aa79f27a0sm3273819wmq.38.2024.05.08.12.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:30:48 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 11/12] mm, riscv, arm64: Use common ptep_set_wrprotect()/wrprotect_ptes() functions Date: Wed, 8 May 2024 21:19:30 +0200 Message-Id: <20240508191931.46060-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_123050_438559_1BC80188 X-CRM114-Status: GOOD ( 21.68 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware ptep_set_wrprotect()/wrprotect_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 56 ++++++ arch/arm64/mm/contpte.c | 18 -------- arch/riscv/include/asm/pgtable.h | 25 + [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:22d listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware ptep_set_wrprotect()/wrprotect_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 56 ++++++------------------ arch/arm64/mm/contpte.c | 18 -------- arch/riscv/include/asm/pgtable.h | 25 +++++++++-- include/linux/contpte.h | 2 + mm/contpte.c | 75 +++++++++++++++++++++++++++++++- 5 files changed, 110 insertions(+), 66 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6591aab11c67..162efd9647dd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1208,7 +1208,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline void ___ptep_set_wrprotect(struct mm_struct *mm, +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) { @@ -1222,23 +1226,13 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) -{ - ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); -} - static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, pte_t *ptep, unsigned int nr) { unsigned int i; for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) - __ptep_set_wrprotect(mm, address, ptep); + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1246,7 +1240,7 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp, __ptep_get((pte_t *)pmdp)); } #define pmdp_establish pmdp_establish @@ -1389,8 +1383,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1478,35 +1470,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes -static __always_inline void wrprotect_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - if (likely(nr == 1)) { - /* - * Optimization: wrprotect_ptes() can only be called for present - * ptes so we only need to check contig bit as condition for - * unfold, and we can remove the contig bit from the pte we read - * to avoid re-reading. This speeds up fork() which is sensitive - * for order-0 folios. Equivalent to contpte_try_unfold(). - */ - pte_t orig_pte = __ptep_get(ptep); - - if (unlikely(pte_cont(orig_pte))) { - __contpte_try_unfold(mm, addr, ptep, orig_pte); - orig_pte = pte_mknoncont(orig_pte); - } - ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); - } else { - contpte_wrprotect_ptes(mm, addr, ptep, nr); - } -} +extern void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr); #define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - wrprotect_ptes(mm, addr, ptep, 1); -} +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma, @@ -1528,7 +1497,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5675a61452ac..1cef93b15d6e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -44,21 +44,3 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); - -void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * If wrprotecting an entire contig range, we can avoid unfolding. Just - * set wrprotect and wait for the later mmu_gather flush to invalidate - * the tlb. Until the flush, the page may or may not be wrprotected. - * After the flush, it is guaranteed wrprotected. If it's a partial - * range though, we must unfold, because we can't have a case where - * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this - * would cause it to continue to be unpredictable after the flush. - */ - - contpte_try_unfold_partial(mm, addr, ptep, nr); - __wrprotect_ptes(mm, addr, ptep, nr); -} -EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b151a5aa4de8..728f31da5e6a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -755,11 +755,21 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, } static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) + unsigned long address, pte_t *ptep, + pte_t pte) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { @@ -807,6 +817,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); +#define wrprotect_ptes wrprotect_ptes #else /* CONFIG_THP_CONTPTE */ @@ -822,12 +838,13 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) +#define wrprotect_ptes __wrprotect_ptes #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect - #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76244b0c678a..d1439db1706c 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -26,5 +26,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 9cbbff1f67ad..fe36b6b1d20a 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -49,6 +49,8 @@ * - ptep_get_and_clear() * - ptep_test_and_clear_young() * - ptep_clear_flush_young() + * - wrprotect_ptes() + * - ptep_set_wrprotect() */ pte_t huge_ptep_get(pte_t *ptep) @@ -266,7 +268,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)); return; } @@ -832,4 +834,75 @@ __always_inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } + +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + if (ptep != arch_contpte_align_down(ptep) || nr < ncontig) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { + unsigned long last_addr = addr + pgsize * (nr - 1); + pte_t *last_ptep = ptep + nr - 1; + + contpte_try_unfold(mm, last_addr, last_ptep, + __ptep_get(last_ptep)); + } +} + +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guaranteed wrprotected. If it's a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); + +__always_inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + __ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + +__always_inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + wrprotect_ptes(mm, addr, ptep, 1); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 1933177 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=OMs03d/l; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=s0ocXD7G; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VZQJG3QtRz1ymg for ; Thu, 9 May 2024 05:31:58 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YQG6mwDiq3VWu4fzcptPjzMOrrLRnK3FYLv6a1bn3c8=; b=OMs03d/l3i9vXX +H3Mqrg6ZWiu1tr5cP5IbNwDZhqjwQSWhjYuRVpSnl0Lho0OcRcd4UKMNFYv4vdi+4DilqyNPBpUK KqV9POVKTlOkrl2U5XRzFJPXnQTAyBrku9sXyD22kkqFFHmvq0Qg7NBpK4+HAT91+My+l4zjc34Lg aYKOzaDXoWwK2mllJTHbv2qwW3YH2NY0/CWhy0UcsykShgpuJDUxxrE/mut6LVB7HQjlftyn3oTH4 tGgBbf+EscJ83rCpgPjYLwi/poIdBLxoCaPl5YNrelqgXSkH4WhYANObEgJJopOQDCq6OEOWezQRL pbALuzrbVYvxpiQoOiiQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4n1B-0000000GlI8-0JB3; Wed, 08 May 2024 19:31:57 +0000 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s4n17-0000000GlFM-0HE2 for kvm-riscv@lists.infradead.org; Wed, 08 May 2024 19:31:54 +0000 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-34e0d47bd98so692614f8f.0 for ; Wed, 08 May 2024 12:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196710; x=1715801510; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Mj4lZQ0sNzizek27x00kn2/PQ3ffbB6QFBBUZYNyzk=; b=s0ocXD7GCjzWOGRkHWVK0cYoCtGsslAovNUb31+56j3pRTyxD1esrjxlQ2+cxbTH7b i9Dh0nqXShbRArkq/5YuDAhT7ijBek/1HOM8VBRX8ZKYjwZE292Pk/xY0uEYTs5JBppK fDkbZqqMSn6tj+VFT+6g7qazMdLbJ+kU3BFEMJedPQuQZkpBzEcf2nkhIXIQnkRS7INB e+D2ZdCBkp8JyWBycUEzX5NMoPNc7RDTe7DF57nMW/BVsZF+ccw8rcZMvl5Hk3Ch7Z06 vRTkdUsM54tae4k6n1ofFLNCOQiGALKWWnMWD9EjV7vESofSsQq+RfnAO98wugVExXdz VYYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196710; x=1715801510; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Mj4lZQ0sNzizek27x00kn2/PQ3ffbB6QFBBUZYNyzk=; b=HGMxyK38AHQFQDwFZuKwEDm1QjgG33RJBt5mEaiJnvE6cPczZb/Gk5+gfhox3Csh5h /Eg2xCi4MT08atT6C4SO5biuSuQ2wBcz8/vi3+7Du+YrteMszgiVOA3pZ//ZETqORwD2 Qb9aTGiQsQGcLXLybeqCXQK4EvNVJlS0E4bHO/rpImp0TL9HPFNAkP99TE/axl8ub5iQ 8HrhgfHlNBZGOPfY1d1Ji0g3ZPTof1HXf5Eazt8+ExOGou9KNupjr1Blc4eOzaxKqzT5 nUE+LUoOLy439bd3AXkelE3Lgzq0HLYrR9lPNYWelRGZeV/iMChbAsLnzKIiBoNQ6BSh lh6Q== X-Forwarded-Encrypted: i=1; AJvYcCVUEqbwOPm9+njKjrJhjY+3nsg3jwaD5+533v0dd4fQ1Hm6utDx3WxqFmd+n8GtIspQyLO2I1amJ0643yndMgD8/+UlHuhLiJmyAggqwQ== X-Gm-Message-State: AOJu0YyVOEkPbZBuvGJtfOPHKKehXwtUVOq9K4uAN8837ArI3Xuyf/Rb WZ1zEmHrdZq+0KpJHDSD8I7Zyc5ULS1n446ZxvA8JNj0NxVK+P8FzK9/HFrSdDI= X-Google-Smtp-Source: AGHT+IGynnks5bQyXkefndTUc8PNt1+uZJK8w5+sVciCnroxfstzlHH5YDMjCv7Za3kPs8Z2dwdXzA== X-Received: by 2002:adf:fe8a:0:b0:34c:fd92:3359 with SMTP id ffacd0b85a97d-350185d57e7mr489056f8f.21.1715196709811; Wed, 08 May 2024 12:31:49 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id s2-20020a5d4242000000b003472489d26fsm15924162wrr.19.2024.05.08.12.31.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:31:49 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 12/12] mm, riscv, arm64: Use common get_and_clear_full_ptes()/clear_full_ptes() functions Date: Wed, 8 May 2024 21:19:31 +0200 Message-Id: <20240508191931.46060-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240508_123153_158376_AA2BE9F2 X-CRM114-Status: GOOD ( 17.75 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Make riscv use the contpte aware get_and_clear_full_ptes()/clear_full_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 41 ++++ arch/arm64/mm/Makefile | 1 - arch/arm64/mm/contpte.c | 46 [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:42c listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Make riscv use the contpte aware get_and_clear_full_ptes()/clear_full_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 41 ++++------------------------ arch/arm64/mm/Makefile | 1 - arch/arm64/mm/contpte.c | 46 ------------------------------- arch/riscv/include/asm/pgtable.h | 39 ++++++++++++++++++++++++++ include/linux/contpte.h | 5 ++++ mm/contpte.c | 47 ++++++++++++++++++++++++++++++++ 6 files changed, 96 insertions(+), 83 deletions(-) delete mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 162efd9647dd..f8a3159f9df0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1373,17 +1373,6 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #ifdef CONFIG_THP_CONTPTE -/* - * The contpte APIs are used to transparently manage the contiguous bit in ptes - * where it is possible and makes sense to do so. The PTE_CONT bit is considered - * a private implementation detail of the public ptep API (see below). - */ -extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full); -extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full); - #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) { @@ -1428,34 +1417,14 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define pte_clear pte_clear +extern void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); #define clear_full_ptes clear_full_ptes -static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full) -{ - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __clear_full_ptes(mm, addr, ptep, nr, full); - } else { - contpte_clear_full_ptes(mm, addr, ptep, nr, full); - } -} +extern pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); #define get_and_clear_full_ptes get_and_clear_full_ptes -static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full) -{ - pte_t pte; - - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); - } else { - pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); - } - - return pte; -} #define __HAVE_ARCH_PTEP_GET_AND_CLEAR extern pte_t ptep_get_and_clear(struct mm_struct *mm, diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 52a1b2082627..dbd1bc95967d 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,7 +3,6 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o -obj-$(CONFIG_THP_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c deleted file mode 100644 index 1cef93b15d6e..000000000000 --- a/arch/arm64/mm/contpte.c +++ /dev/null @@ -1,46 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Copyright (C) 2023 ARM Ltd. - */ - -#include -#include -#include -#include - -static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * Unfold any partially covered contpte block at the beginning and end - * of the range. - */ - - if (ptep != arch_contpte_align_down(ptep) || nr < CONT_PTES) - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - - if (ptep + nr != arch_contpte_align_down(ptep + nr)) { - unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); - pte_t *last_ptep = ptep + nr - 1; - - contpte_try_unfold(mm, last_addr, last_ptep, - __ptep_get(last_ptep)); - } -} - -void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full) -{ - contpte_try_unfold_partial(mm, addr, ptep, nr); - __clear_full_ptes(mm, addr, ptep, nr, full); -} -EXPORT_SYMBOL_GPL(contpte_clear_full_ptes); - -pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full) -{ - contpte_try_unfold_partial(mm, addr, ptep, nr); - return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); -} -EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 728f31da5e6a..a4843bdfdb37 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -754,6 +754,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) @@ -823,6 +854,13 @@ extern void ptep_set_wrprotect(struct mm_struct *mm, extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); #define wrprotect_ptes wrprotect_ptes +extern void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +#define clear_full_ptes clear_full_ptes +extern pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); +#define get_and_clear_full_ptes get_and_clear_full_ptes #else /* CONFIG_THP_CONTPTE */ @@ -842,6 +880,7 @@ extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, #define ptep_set_wrprotect(mm, addr, ptep) \ __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes +#define clear_full_ptes __clear_full_ptes #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index d1439db1706c..b24554ebca41 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -28,5 +28,10 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty); void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index fe36b6b1d20a..677344e0e3c3 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -51,6 +51,8 @@ * - ptep_clear_flush_young() * - wrprotect_ptes() * - ptep_set_wrprotect() + * - clear_full_ptes() + * - get_and_clear_full_ptes() */ pte_t huge_ptep_get(pte_t *ptep) @@ -905,4 +907,49 @@ __always_inline void ptep_set_wrprotect(struct mm_struct *mm, { wrprotect_ptes(mm, addr, ptep, 1); } + +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL_GPL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); + +__always_inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +__always_inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #endif /* CONFIG_THP_CONTPTE */