From patchwork Sat Mar 7 13:04:52 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinyu Tang X-Patchwork-Id: 2206648 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=mShto7+9; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=163.com header.i=@163.com header.a=rsa-sha256 header.s=s110527 header.b=Pe4LzOid; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fSk7Z276nz1y04 for ; Sun, 08 Mar 2026 00:06:44 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=09ZIGEVKDBXoc7WOvUoyT0EITJ7qBzlGcyu7OQiTy/Q=; b=mShto7+9vOv4Kl HyNJi3QNRhdzPLOc0apgKzaoMWYaGM5DJHzIBp9X4nfE3YtqHCWDff94EHOo8DsiMM6GnMyjiHF5F MJcAp61IbfHqAfdUsdr2wAJpCcvB17vKUu3Pj/pjMsyytcytebzysxYJ6Z6/rSPyh6TiMl4EStlFV Ub5uuhnZIu+raszvOt1/ZxsmZOYZE8LH9xH9cwlcfPiQsdT3JaSrSZn11tNDTNmFysCWuG6JwG35E o3ZiAD1jARszuMd4z62mFvQwLL1i9Frz9PJ82bkUTFpxnGv1u9Kao0/a3nd+ZAShSQIMxDI3O+Jsx uDNOo2d8FP45B5RUlv/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1vyrM4-00000005D6J-0PL3; Sat, 07 Mar 2026 13:06:04 +0000 Received: from m16.mail.163.com ([117.135.210.2]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1vyrM1-00000005D5v-20ua; Sat, 07 Mar 2026 13:06:03 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=2N nYf0TylnfesYJfPKbFSikhT8HSeI1rOTmDrhxI7/g=; b=Pe4LzOidB+y8SxKGp5 NhkRI+7MjIz+nY6TDHnE64AePj1JGKqg8Yvfv2X/5VZ6swujgnDftKVKJos/yc6H +fmpHb7/B7y/p8M/nMP7SfjtNGlLIhw3Nu0igxU4AIh38JfQpM+0paKYP2GhhCzX eynHVXc29X4JEEyloJuGsFmnE= Received: from localhost.localdomain (unknown []) by gzga-smtp-mtada-g1-3 (Coremail) with SMTP id _____wCHbWl3IqxpV5Y+Pw--.38359S2; Sat, 07 Mar 2026 21:04:56 +0800 (CST) From: Jinyu Tang To: Anup Patel , Atish Patra , Paul Walmsley , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexandre Ghiti , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , Andrew Jones , Conor Dooley , Yong-Xuan Wang , Nutty Liu Cc: kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Jinyu Tang Subject: [PATCH v1] RISC-V: KVM: Batch stage-2 remote TLB flushes Date: Sat, 7 Mar 2026 21:04:52 +0800 Message-ID: <20260307130452.721375-1-tjytimi@163.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-CM-TRANSID: _____wCHbWl3IqxpV5Y+Pw--.38359S2 X-Coremail-Antispam: 1Uf129KBjvJXoWfJrWfAFWrAr47Ww45WFy3Jwb_yoWDKw43pr 4DCryfur4fXrs7XF13tFWDZrn8uws7W3WrAry5CF90yFn0qr4fXr1vg34vvry5JryrXFW3 ZFyDGF15Ar4IyFUanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x0piUGYdUUUUU= X-Originating-IP: [2408:823d:2011:31d0:5258:f950:b600:dbf0] X-CM-SenderInfo: xwm13xlpl6il2tof0z/xtbC8RgspGmsInj8EwAA3R X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260307_050602_130392_02647F0F X-CRM114-Status: GOOD ( 16.12 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: KVM RISC-V triggers a TLB flush for every single stage-2 PTE modification (unmap or write-protect) now. Although KVM coalesces the hardware IPIs, the software overhead of executing the flush work for [...] Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.4 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. [117.135.210.2 listed in sa-trusted.bondedsender.org] 0.9 RCVD_IN_VALIDITY_SAFE_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. [117.135.210.2 listed in sa-accredit.habeas.com] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 0.8 RCVD_IN_VALIDITY_RPBL_BLOCKED RBL: ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. [117.135.210.2 listed in bl.score.senderscore.com] 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [tjytimi(at)163.com] 0.0 UNPARSEABLE_RELAY Informational: message has unparseable relay lines X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org KVM RISC-V triggers a TLB flush for every single stage-2 PTE modification (unmap or write-protect) now. Although KVM coalesces the hardware IPIs, the software overhead of executing the flush work for every 4K page is large, especially during dirty page tracking. Following the approach used in x86 and arm64, this patch optimizes the MMU logic by making the PTE manipulation functions return a boolean indicating if a leaf PTE was actually changed. The outer MMU functions bubble up this flag to batch the remote TLB flushes. This change makes the generic KVM MMU callbacks (e.g., `kvm_unmap_gfn_range()`) actually effective by returning the true flush requirement rather than `false`. Consequently, the flush operation is executed only once per batch. Moving it outside of the `mmu_lock` also reduces lock contention. For directory pages, we keep the synchronous flush to ensure safety. Tested with tools/testing/selftests/kvm on a 4-vCPU guest (Host environment: QEMU 10.2.1 RISC-V) 1. demand_paging_test (1GB memory) # time ./demand_paging_test -b 1G -v 4 - Total execution time reduced from ~2m33s to ~2m25s 2. dirty_log_perf_test (1GB memory) # ./dirty_log_perf_test -b 1G -v 4 - "Clear dirty log time" per iteration dropped significantly from ~3.02s to ~0.20s Signed-off-by: Jinyu Tang Reviewed-by: Nutty Liu --- arch/riscv/include/asm/kvm_gstage.h | 6 ++--- arch/riscv/kvm/gstage.c | 40 ++++++++++++++++++++--------- arch/riscv/kvm/mmu.c | 34 ++++++++++++++++++------ 3 files changed, 57 insertions(+), 23 deletions(-) diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/kvm_gstage.h index 595e21831..b003a07f1 100644 --- a/arch/riscv/include/asm/kvm_gstage.h +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -59,13 +59,13 @@ enum kvm_riscv_gstage_op { GSTAGE_OP_WP, /* Write-protect */ }; -void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, +bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op); -void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, +bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, gpa_t start, gpa_t size, bool may_block); -void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end); +bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end); void kvm_riscv_gstage_mode_detect(void); diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index b67d60d72..8e7a69d35 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -209,49 +209,59 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, return kvm_riscv_gstage_set_pte(gstage, pcache, out_map); } -void kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, +bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, pte_t *ptep, u32 ptep_level, enum kvm_riscv_gstage_op op) { int i, ret; pte_t old_pte, *next_ptep; u32 next_ptep_level; unsigned long next_page_size, page_size; + bool flush = false; ret = gstage_level_to_page_size(ptep_level, &page_size); if (ret) - return; + return false; WARN_ON(addr & (page_size - 1)); if (!pte_val(ptep_get(ptep))) - return; + return false; if (ptep_level && !gstage_pte_leaf(ptep)) { next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(next_ptep_level, &next_page_size); if (ret) - return; + return false; if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); for (i = 0; i < PTRS_PER_PTE; i++) - kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, + flush |= kvm_riscv_gstage_op_pte(gstage, addr + i * next_page_size, &next_ptep[i], next_ptep_level, op); - if (op == GSTAGE_OP_CLEAR) + if (op == GSTAGE_OP_CLEAR) { + gstage_tlb_flush(gstage, ptep_level, addr); + flush = false; put_page(virt_to_page(next_ptep)); + } } else { old_pte = *ptep; if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); - if (pte_val(*ptep) != pte_val(old_pte)) - gstage_tlb_flush(gstage, ptep_level, addr); + if (pte_val(*ptep) != pte_val(old_pte)) { + if (gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) + gstage_tlb_flush(gstage, ptep_level, addr); + else + flush = true; + } } + + return flush; } -void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, +bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, gpa_t start, gpa_t size, bool may_block) { int ret; @@ -260,6 +270,7 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, bool found_leaf; unsigned long page_size; gpa_t addr = start, end = start + size; + bool flush = false; while (addr < end) { found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level); @@ -271,7 +282,7 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, goto next; if (!(addr & (page_size - 1)) && ((end - addr) >= page_size)) - kvm_riscv_gstage_op_pte(gstage, addr, ptep, + flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep, ptep_level, GSTAGE_OP_CLEAR); next: @@ -284,9 +295,11 @@ void kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, if (!(gstage->flags & KVM_GSTAGE_FLAGS_LOCAL) && may_block && addr < end) cond_resched_lock(&gstage->kvm->mmu_lock); } + + return flush; } -void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end) +bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end) { int ret; pte_t *ptep; @@ -294,6 +307,7 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end bool found_leaf; gpa_t addr = start; unsigned long page_size; + bool flush = false; while (addr < end) { found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level); @@ -305,12 +319,14 @@ void kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end goto next; if (!(addr & (page_size - 1)) && ((end - addr) >= page_size)) - kvm_riscv_gstage_op_pte(gstage, addr, ptep, + flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep, ptep_level, GSTAGE_OP_WP); next: addr += page_size; } + + return flush; } void __init kvm_riscv_gstage_mode_detect(void) diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 0b75eb2a1..075cc433e 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -23,6 +23,7 @@ static void mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; gstage.kvm = kvm; gstage.flags = 0; @@ -30,9 +31,10 @@ static void mmu_wp_memory_region(struct kvm *kvm, int slot) gstage.pgd = kvm->arch.pgd; spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_wp_range(&gstage, start, end); + flush = kvm_riscv_gstage_wp_range(&gstage, start, end); spin_unlock(&kvm->mmu_lock); - kvm_flush_remote_tlbs_memslot(kvm, memslot); + if (flush) + kvm_flush_remote_tlbs_memslot(kvm, memslot); } int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, @@ -88,6 +90,7 @@ int kvm_riscv_mmu_ioremap(struct kvm *kvm, gpa_t gpa, phys_addr_t hpa, void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size) { struct kvm_gstage gstage; + bool flush; gstage.kvm = kvm; gstage.flags = 0; @@ -95,8 +98,11 @@ void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size) gstage.pgd = kvm->arch.pgd; spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); + flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); spin_unlock(&kvm->mmu_lock); + + if (flush) + kvm_flush_remote_tlbs_range(kvm, gpa >> PAGE_SHIFT, size >> PAGE_SHIFT); } void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, @@ -108,13 +114,18 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; gstage.kvm = kvm; gstage.flags = 0; gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); gstage.pgd = kvm->arch.pgd; - kvm_riscv_gstage_wp_range(&gstage, start, end); + flush = kvm_riscv_gstage_wp_range(&gstage, start, end); + if (flush) + kvm_flush_remote_tlbs_range(kvm, start >> PAGE_SHIFT, + (end - start) >> PAGE_SHIFT); + } void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) @@ -140,6 +151,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, gpa_t gpa = slot->base_gfn << PAGE_SHIFT; phys_addr_t size = slot->npages << PAGE_SHIFT; struct kvm_gstage gstage; + bool flush; gstage.kvm = kvm; gstage.flags = 0; @@ -147,8 +159,10 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, gstage.pgd = kvm->arch.pgd; spin_lock(&kvm->mmu_lock); - kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); + flush = kvm_riscv_gstage_unmap_range(&gstage, gpa, size, false); spin_unlock(&kvm->mmu_lock); + if (flush) + kvm_flush_remote_tlbs(kvm); } void kvm_arch_commit_memory_region(struct kvm *kvm, @@ -253,10 +267,9 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) gstage.flags = 0; gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); gstage.pgd = kvm->arch.pgd; - kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, + return kvm_riscv_gstage_unmap_range(&gstage, range->start << PAGE_SHIFT, (range->end - range->start) << PAGE_SHIFT, range->may_block); - return false; } bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) @@ -579,6 +592,7 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) { struct kvm_gstage gstage; void *pgd = NULL; + bool flush = false; spin_lock(&kvm->mmu_lock); if (kvm->arch.pgd) { @@ -586,13 +600,17 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) gstage.flags = 0; gstage.vmid = READ_ONCE(kvm->arch.vmid.vmid); gstage.pgd = kvm->arch.pgd; - kvm_riscv_gstage_unmap_range(&gstage, 0UL, kvm_riscv_gstage_gpa_size, false); + flush = kvm_riscv_gstage_unmap_range(&gstage, 0UL, + kvm_riscv_gstage_gpa_size, false); pgd = READ_ONCE(kvm->arch.pgd); kvm->arch.pgd = NULL; kvm->arch.pgd_phys = 0; } spin_unlock(&kvm->mmu_lock); + if (flush) + kvm_flush_remote_tlbs(kvm); + if (pgd) free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size)); }