From patchwork Fri Mar 11 00:25:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 1604182 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=ht+BOCFO; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=VNpyKBBd; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KF6Ck6M0sz9sGG for ; Fri, 11 Mar 2022 11:25:38 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=mJE+Xvr/Uhtw9GZEzxiVm0iWp4bSiCJWJwCTHRTRVN0=; b=ht+ BOCFOHCsG6GIcEJwmENn9sHdt75YSpinNJymOCVn5/u0M6HzbdJlx6UYyuIi68o21ystxdvIimXJg Z7K99p9gfCsIv2Y/cUeFaiccPhJWb7tqclTN2PbfVc+lfVJl4/gZ7RcfAPUitYBE87YQPVNUA+nmx K9e/nKzc6Wc2M6EOSRQN/RvGL367C8cqqv7f/i5cDN6W4U+OJg30XriorE2y5iPyAIoMRpcQu9iut Mm+7bUy/493PxeqGB5l0Z+6zqPbpJWfDHmKG9LvbA2G2x64gy1QBsHnlNkBPHOfiud2nNG3swMrWp DBCusCEkCMEgjAG2c3HNWDBmghZmgIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nST68-00EP8n-Jc; Fri, 11 Mar 2022 00:25:36 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nST65-00EP7R-8T for kvm-riscv@lists.infradead.org; Fri, 11 Mar 2022 00:25:35 +0000 Received: by mail-pf1-x449.google.com with SMTP id 127-20020a620685000000b004f6eaf868easo4156618pfg.22 for ; Thu, 10 Mar 2022 16:25:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=68gIue062XFpeNPWC+2GMGTpU9jB2fRiIaDhJyxenHk=; b=VNpyKBBd/v0UuvYgY/ZloH9oZ5k+QD5tcKd6YiYVquRAOM6SViBWGVvtGVuktxf9yD CuzF60g53qOcH8NszijmP4V2D2qCibXZBFTKOK33EBrUF9SgEGsPwMQcIvmlKUDJGYo7 iX1iekrhh94bn2ulyUflf0obMLH9Su5OVjWLEEU63I8NE4+GYoc19qqvgatfrrpkfAso gids+kT5gumjiGmHji7RVzYUOnQTwZn0TwAPYo2ewfje4MIWZDAawLZr2sRG2uZACKgZ +1pkELUe9fjewuMk7Tv+5E5mB8lTNVU6QJgsC6AsUo6bb2eBbzMTACejkwjqoTUZ5IFk ftxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=68gIue062XFpeNPWC+2GMGTpU9jB2fRiIaDhJyxenHk=; b=Jd+7nt3+SDYbXHi1n2DGFyWc81VEM4lNMpqICbDhlCw/r1WIh7ovLHHJjhF4+Igpi5 kBLiwSlOxlcZmLFWxfta5oraZ9C15/9VB8sJw+7QPruR0NuZ64Yv3UNmwlDnQD72hUSF t8trFx6fe1OuGjhcV77hLenZ48TZz9CsjAM9ITC23yCrHhnKSjnxgiUN9IUMKiBvf64k IqASCQkV+i7TcLTLixVPYrNejtptpwe3nKZgjyBVB5ZgNQLL4E7ibxv8MwG00kEtCnQl RnqDXXwn7j92kmJqP7jXDw0Q3WOVU0cAAK8Ymv+7LBsGx7YZD9/Tc8XlESFk68Q1IvDG FLDg== X-Gm-Message-State: AOAM531S+n8M3865tDZ2OEKFJFvF/PXMf9iPsVFYe0E+Xc+NAF8IK/VI zkdkM1KCGLAJcj7REYYdXkauDJj8iHwu5w== X-Google-Smtp-Source: ABdhPJxCSSsqSvR9E019LXHT4m59+2+TSlLclZ+WdnUH9Ghz6ysOEsXkR+QRVOEjrjxM6Te9nIBHRjCsO4pPHg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:887:b0:4f2:6d3f:5b53 with SMTP id q7-20020a056a00088700b004f26d3f5b53mr7613490pfj.21.1646958331377; Thu, 10 Mar 2022 16:25:31 -0800 (PST) Date: Fri, 11 Mar 2022 00:25:02 +0000 Message-Id: <20220311002528.2230172-1-dmatlack@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.1.723.g4982287a31-goog Subject: [PATCH v2 00/26] Extend Eager Page Splitting to the shadow MMU From: David Matlack To: Paolo Bonzini Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Sean Christopherson , Andrew Jones , Ben Gardon , Peter Xu , maciej.szmigiero@oracle.com, "moderated list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64)" , "open list:KERNEL VIRTUAL MACHINE FOR MIPS (KVM/mips)" , "open list:KERNEL VIRTUAL MACHINE FOR MIPS (KVM/mips)" , "open list:KERNEL VIRTUAL MACHINE FOR RISC-V (KVM/riscv)" , Peter Feiner , David Matlack X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220310_162533_350469_BAA44282 X-CRM114-Status: GOOD ( 23.97 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: [ NOTE: I would like to do more testing on this series than what is described in the Testing section below, but I will be out-of-office all next week so I wanted to share what I have so far. I fully e [...] Content analysis details: (-7.7 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org [ NOTE: I would like to do more testing on this series than what is described in the Testing section below, but I will be out-of-office all next week so I wanted to share what I have so far. I fully expect a v3 of this series anyway after collecting more reviews from Sean. ] This series extends KVM's Eager Page Splitting to also split huge pages mapped by the shadow MMU, i.e. huge pages present in the memslot rmaps. This will be useful for configurations that use Nested Virtualization, disable the TDP MMU, or disable/lack TDP hardware support. For background on Eager Page Splitting, see: - Proposal: https://lore.kernel.org/kvm/CALzav=dV_U4r1K9oDq4esb4mpBQDQ2ROQ5zH5wV3KpOaZrRW-A@mail.gmail.com/ - TDP MMU support: https://lore.kernel.org/kvm/20220119230739.2234394-1-dmatlack@google.com/ Splitting huge pages mapped by the shadow MMU is more complicated than the TDP MMU, but it is also more important for performance as the shadow MMU handles huge page write-protection faults under the write lock. See the Performance section for more details. The extra complexity of splitting huge pages mapped by the shadow MMU comes from a few places: (1) The shadow MMU has a limit on the number of shadow pages that are allowed to be allocated. So, as a policy, Eager Page Splitting refuses to split if there are KVM_MIN_FREE_MMU_PAGES or fewer pages available. (2) Huge pages may be mapped by indirect shadow pages. - Indirect shadow pages have the possibilty of being unsync. As a policy we opt not to split such pages as their translation may no longer be valid. - Huge pages on indirect shadow pages may have access permission constraints from the guest (unlike the TDP MMU which is ACC_ALL by default). (3) Splitting a huge page may end up re-using an existing lower level shadow page tables. This is unlike the TDP MMU which always allocates new shadow page tables when splitting. (4) When installing the lower level SPTEs, they must be added to the rmap which may require allocating additional pte_list_desc structs. In Google's internal implementation of Eager Page Splitting, we do not handle cases (3) and (4), and intstead opts to skip splitting entirely (case 3) or only partially splitting (case 4). This series handles the additional cases (patches 21-25), which comes with some additional complexity and an additional 4KiB of memory per VM to store the extra pte_list_desc cache. However it does also avoid the need for TLB flushes in most cases. About half of this series, patches 1-15, is just refactoring the existing MMU code in preparation for splitting. The bulk of the refactoring is to make it possible to operate on the MMU outside of a vCPU context. Motivation ---------- During dirty logging, VMs using the shadow MMU suffer from: (1) Write-protection faults on huge pages that take the MMU lock to unmap the huge page, map a 4KiB page, and update the dirty log. (2) Non-present faults caused by (1) that take the MMU lock to map in the missing page. (3) Write-protection faults on 4KiB pages that take the MMU lock to make the page writable and update the dirty log. [Note: These faults only take the MMU lock during shadow paging.] The lock contention from (1), (2) and (3) can severely degrade application performance to the point of failure. Eager page splitting eliminates (1) by moving the splitting of huge pages off the vCPU threads onto the thread invoking VM-ioctls to configure dirty logging, and eliminates (2) by fully splitting each huge page into its constituent small pages. (3) is still a concern for shadow paging workloads (e.g. nested virtualization) but is not addressed by this series. Splitting in the VM-ioctl thread is useful because it can run in the background without interrupting vCPU execution. However, it does take the MMU lock so it may introduce some extra contention if vCPUs are hammering the MMU lock. This is offset by the fact that eager page splitting drops the MMU lock after splitting each SPTE if there is any contention, and the fact that eager page splitting is reducing the MMU lock contention from (1) and (2) above. Even workloads that only write to 5% of their memory see massive MMU lock contention reduction during dirty logging thanks to Eager Page Splitting (see Performance data below). A downside of Eager Page Splitting is that it splits all huge pages, which may include ranges of memory that are never written to by the guest and thus could theoretically stay huge. Workloads that write to only a fraction of their memory may see higher TLB miss costs with Eager Page Splitting enabled. However, that is secondary to the application failure that otherwise may occur without Eager Page Splitting. Further work is necessary to improve the TLB miss performance for read-heavy workoads, such as dirty logging at 2M instead of 4K. Performance ----------- To measure the performance impact of Eager Page Splitting I ran dirty_log_perf_test with tdp_mmu=N, various virtual CPU counts, 1GiB per vCPU, and backed by 1GiB HugeTLB memory. The amount of memory that was written to versus read was controlled with the -f option. To measure the imapct of customer performance, we can look at the time it takes all vCPUs to dirty memory after dirty logging has been enabled. Without Eager Page Splitting enabled, such dirtying must take faults to split huge pages and bottleneck on the MMU lock. | 100% written / 0% read | | --------------------------------------------| | "Iteration 1 dirty memory time" (ept=Y) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.310786549s | 0.058731929s | 4 | 0.419165587s | 0.059615316s | 8 | 1.061233860s | 0.060945457s | 16 | 2.852955595s | 0.067069980s | 32 | 7.032750509s | 0.078623606s | 64 | 16.501287504s | 0.083914116s | | 5% written / 95% read | | --------------------------------------------| | "Iteration 1 dirty memory time" (ept=Y) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.325023846s | 0.006049684s | 4 | 0.398393318s | 0.006275966s | 8 | 1.242848347s | 0.006861012s | 16 | 2.724926895s | 0.010056859s | 32 | 7.134648637s | 0.012153849s | 64 | 16.804434189s | 0.017575228s | Eager Page Splitting does increase the time it takes to enable dirty logging when not using initially-all-set, since that's when KVM splits huge pages. However, this runs in parallel with vCPU execution and drops the MMU lock whenever there is contention. | 100% written / 0% read | | --------------------------------------------| | "Enabling dirty logging time" (ept=Y) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.001581619s | 0.025699730s | 4 | 0.003138664s | 0.051510208s | 8 | 0.006247177s | 0.102960379s | 16 | 0.012603892s | 0.206949435s | 32 | 0.026428036s | 0.435855597s | 64 | 0.103826796s | 1.199686530s | Similarly, Eager Page Splitting increases the time it takes to clear the dirty log for when using initially-all-set. The first time userspace clears the dirty log, KVM will split huge pages: | 100% written / 0% read | | --------------------------------------------| | "Iteration 1 clear dirty log time" (ept=Y) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.001544730s | 0.055327916s | 4 | 0.003145920s | 0.111887354s | 8 | 0.006306964s | 0.223920530s | 16 | 0.012681628s | 0.447849488s | 32 | 0.026827560s | 0.943874520s | 64 | 0.090461490s | 2.664388025s | Subsequent calls to clear the dirty log incur almost no additional cost since KVM can very quickly determine there are no more huge pages to split via the RMAP. This is unlike the TDP MMU which must re-traverse the entire page table to check for huge pages. | 100% written / 0% read | | --------------------------------------------| | "Iteration 2 clear dirty log time" (ept=Y) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.015613726s | 0.015771982s | 4 | 0.031456620s | 0.031911594s | 8 | 0.063341572s | 0.063837403s | 16 | 0.128409332s | 0.127484064s | 32 | 0.255635696s | 0.268837996s | 64 | 0.695572818s | 0.700420727s | Eager Page Splitting also improves the performance for shadow paging configurations, as measured with ept=N. Although the absolute gains are less for write-heavy workloads since KVM's shadow paging takes the write lock to track 4KiB writes (i.e. no fast_page_faut() or PML). However there are still major gains for read/write and read-heavy workloads. | 100% written / 0% read | | --------------------------------------------| | "Iteration 1 dirty memory time" (ept=N) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.373022770s | 0.348926043s | 4 | 0.563697483s | 0.453022037s | 8 | 1.588492808s | 1.524962010s | 16 | 3.988934732s | 3.369129917s | 32 | 9.470333115s | 8.292953856s | 64 | 20.086419186s | 18.531840021s | | 50% written / 50% read | | --------------------------------------------| | "Iteration 1 dirty memory time" (ept=N) | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.374301914s | 0.174864494s | 4 | 0.539841346s | 0.213246828s | 8 | 1.759793717s | 0.526697696s | 16 | 3.786053801s | 1.338638169s | 32 | 9.603927533s | 3.869825083s | 64 | 20.376757135s | 9.158492731s | | 5% written / 95% read | | --------------------------------------------| | "Iteration 1 dirty memory time" | | ------------------------------------------- | vCPU Count | eager_page_split=N | eager_page_split=Y | ------------ | -------------------- | -------------------- | 2 | 0.381538968s | 0.020396121s | 4 | 0.511922608s | 0.022625023s | 8 | 1.464410632s | 0.054727913s | 16 | 3.783471041s | 0.133412717s | 32 | 9.519432076s | 0.390443803s | 64 | 21.052654299s | 0.929496710s | Testing ------- - Ran all kvm-unit-tests and KVM selftests with all combinations of ept=[NY] and tdp_mmu=[NY]. - Booted a 32-bit non-PAE kernel with shadow paging to verify the quadrant change in patch 3. Version Log ----------- v2: - Add performance data for workloads that mix reads and writes [Peter] - Collect R-b tags from Ben and Sean. - Fix quadrant calculation when deriving role from parent [Sean] - Tweak new shadow page function names [Sean] - Move set_page_private() to allocation functions [Ben] - Only zap collapsible SPTEs up to MAX_LEVEL-1 [Ben] - Always top-up pte_list_desc cache to reduce complexity [Ben] - Require mmu cache capacity field to be initialized and add WARN() to reduce chance of programmer error [Marc] - Fix up kvm_mmu_memory_cache struct initialization in arm64 [Marc] v1: https://lore.kernel.org/kvm/20220203010051.2813563-1-dmatlack@google.com/ David Matlack (26): KVM: x86/mmu: Optimize MMU page cache lookup for all direct SPs KVM: x86/mmu: Use a bool for direct KVM: x86/mmu: Derive shadow MMU page role from parent KVM: x86/mmu: Decompose kvm_mmu_get_page() into separate functions KVM: x86/mmu: Rename shadow MMU functions that deal with shadow pages KVM: x86/mmu: Pass memslot to kvm_mmu_new_shadow_page() KVM: x86/mmu: Separate shadow MMU sp allocation from initialization KVM: x86/mmu: Link spt to sp during allocation KVM: x86/mmu: Move huge page split sp allocation code to mmu.c KVM: x86/mmu: Use common code to free kvm_mmu_page structs KVM: x86/mmu: Use common code to allocate kvm_mmu_page structs from vCPU caches KVM: x86/mmu: Pass const memslot to rmap_add() KVM: x86/mmu: Pass const memslot to init_shadow_page() and descendants KVM: x86/mmu: Decouple rmap_add() and link_shadow_page() from kvm_vcpu KVM: x86/mmu: Update page stats in __rmap_add() KVM: x86/mmu: Cache the access bits of shadowed translations KVM: x86/mmu: Pass access information to make_huge_page_split_spte() KVM: x86/mmu: Zap collapsible SPTEs at all levels in the shadow MMU KVM: x86/mmu: Refactor drop_large_spte() KVM: x86/mmu: Extend Eager Page Splitting to the shadow MMU KVM: Allow for different capacities in kvm_mmu_memory_cache structs KVM: Allow GFP flags to be passed when topping up MMU caches KVM: x86/mmu: Fully split huge pages that require extra pte_list_desc structs KVM: x86/mmu: Split huge pages aliased by multiple SPTEs KVM: x86/mmu: Drop NULL pte_list_desc_cache fallback KVM: selftests: Map x86_64 guest virtual memory with huge pages .../admin-guide/kernel-parameters.txt | 3 - arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/mmu.c | 13 +- arch/mips/include/asm/kvm_host.h | 2 +- arch/mips/kvm/mips.c | 2 + arch/riscv/include/asm/kvm_host.h | 2 +- arch/riscv/kvm/vcpu.c | 1 + arch/x86/include/asm/kvm_host.h | 19 +- arch/x86/include/asm/kvm_page_track.h | 2 +- arch/x86/kvm/mmu/mmu.c | 744 +++++++++++++++--- arch/x86/kvm/mmu/mmu_internal.h | 22 +- arch/x86/kvm/mmu/page_track.c | 4 +- arch/x86/kvm/mmu/paging_tmpl.h | 21 +- arch/x86/kvm/mmu/spte.c | 10 +- arch/x86/kvm/mmu/spte.h | 3 +- arch/x86/kvm/mmu/tdp_mmu.c | 48 +- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- include/linux/kvm_host.h | 1 + include/linux/kvm_types.h | 19 +- .../selftests/kvm/include/x86_64/processor.h | 6 + tools/testing/selftests/kvm/lib/kvm_util.c | 4 +- .../selftests/kvm/lib/x86_64/processor.c | 31 + virt/kvm/kvm_main.c | 19 +- 24 files changed, 768 insertions(+), 213 deletions(-) base-commit: ce41d078aaa9cf15cbbb4a42878cc6160d76525e