From patchwork Fri Sep 8 07:42:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 1831361 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=OLfvm5ZP; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.cz header.i=@suse.cz header.a=rsa-sha256 header.s=susede2_rsa header.b=0Tr1RiLs; dkim=fail reason="signature verification failed" header.d=suse.cz header.i=@suse.cz header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=r9UDzDCi; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rhp3y4N01z1yhh for ; Fri, 8 Sep 2023 17:42:38 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=UfpLpXS5xV7F5cFqU75OSIsWVR59gXSxDrEFGIpkd3w=; b=OLfvm5ZPxJi4cT kxT2f6Shloz6hBM6gtTtkUK4fk0kJv8SzKHCYXB/F4QC3j6jN2cw9qInP0JZrdGMnAn5YeRRb8NsH BLpke3sfaoaMrXI52bZ6C3DxSpNc2+l7a/c1u5ZJIQGg3qgC4JP3jWlRfU+vVY5oSSnUM/ZjI5ph4 6g0sw0oc5eVzGlh60eWYZvYmT/n2GelA8ePnP+ipF5RMvFKwQUBAZMCZq94qCPg/in1HcaB2/s7yi uGQLWLr1IuM+wQxkTPpD+O/hHYQp5LI6mP9wsaWpYaaHy5iI33yx5eQmVQdkd68aUJ8E9Hmjsd6vE I1HVxZRMkXh4F9ULor/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qeW8Q-00DG2s-03; Fri, 08 Sep 2023 07:42:34 +0000 Received: from smtp-out1.suse.de ([195.135.220.28]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qeW8M-00DG1Z-1s; Fri, 08 Sep 2023 07:42:32 +0000 Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2588421832; Fri, 8 Sep 2023 07:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1694158947; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=NHJKcpKlZ/+yii6ZB0cLfqW7JVxO+ELAe/deKUrmJ80=; b=0Tr1RiLsGK9TKLdxwSc9RwxLIVKmEae8PGUWUTGhYIhqkvcnxmKXE3HYt5+5SqUjjUpxEZ DCTZmPbwNt5ToqgWZJ00UT0I0Plhw1i14U2HJUY0Px379BJbaU56+RSKEu8z+V/xOMo2xd woPLeoFeQ7kp5dfAXicDw3PDice1ftM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1694158947; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=NHJKcpKlZ/+yii6ZB0cLfqW7JVxO+ELAe/deKUrmJ80=; b=r9UDzDCi5/bP4K3dBarcJVpQtk/FZHh8rlyYCrXu3EqfT7loeAs6cgqxhTivXkBssY2kKX lV8TgWuOzutVQyDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 96D2A132F2; Fri, 8 Sep 2023 07:42:26 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id WIYoJGLQ+mTfFQAAMHmgww (envelope-from ); Fri, 08 Sep 2023 07:42:26 +0000 From: Vlastimil Babka To: seanjc@google.com Cc: ackerleytng@google.com, akpm@linux-foundation.org, anup@brainfault.org, aou@eecs.berkeley.edu, chao.p.peng@linux.intel.com, chenhuacai@kernel.org, david@redhat.com, isaku.yamahata@gmail.com, jarkko@kernel.org, jmorris@namei.org, kirill.shutemov@linux.intel.com, kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, liam.merwick@oracle.com, linux-arm-kernel@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-security-module@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, mail@maciej.szmigiero.name, maz@kernel.org, michael.roth@amd.com, mpe@ellerman.id.au, oliver.upton@linux.dev, palmer@dabbelt.com, paul.walmsley@sifive.com, paul@paul-moore.com, pbonzini@redhat.com, qperret@google.com, serge@hallyn.com, tabba@google.com, vannapurve@google.com, vbabka@suse.cz, wei.w.wang@intel.com, willy@infradead.org, yu.c.zhang@linux.intel.com Subject: [PATCH gmem FIXUP v2] mm, compaction: make testing mapping_unmovable() safe Date: Fri, 8 Sep 2023 09:42:23 +0200 Message-ID: <20230908074222.28723-2-vbabka@suse.cz> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230908_004230_782164_5320C275 X-CRM114-Status: GOOD ( 18.85 ) X-Spam-Score: -2.5 (--) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: As Kirill pointed out, mapping can be removed under us due to truncation. Test it under folio lock as already done for the async compaction / dirty folio case. To prevent locking every folio with mapp [...] Content analysis details: (-2.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [195.135.220.28 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org As Kirill pointed out, mapping can be removed under us due to truncation. Test it under folio lock as already done for the async compaction / dirty folio case. To prevent locking every folio with mapping to do the test, do it only for unevictable folios, as we can expect the unmovable mapping folios are also unevictable. To enforce that expecation, make mapping_set_unmovable() also set AS_UNEVICTABLE. Also incorporate comment update suggested by Matthew. Fixes: 3424873596ce ("mm: Add AS_UNMOVABLE to mark mapping as completely unmovable") Signed-off-by: Vlastimil Babka --- v2: mapping_set_unmovable() sets also AS_UNEVICTABLE, as Sean suggested. include/linux/pagemap.h | 6 +++++ mm/compaction.c | 49 +++++++++++++++++++++++++++-------------- virt/kvm/guest_mem.c | 2 +- 3 files changed, 39 insertions(+), 18 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 931d2f1da7d5..4070c59e6f25 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -276,6 +276,12 @@ static inline int mapping_use_writeback_tags(struct address_space *mapping) static inline void mapping_set_unmovable(struct address_space *mapping) { + /* + * It's expected unmovable mappings are also unevictable. Compaction + * migrate scanner (isolate_migratepages_block()) relies on this to + * reduce page locking. + */ + set_bit(AS_UNEVICTABLE, &mapping->flags); set_bit(AS_UNMOVABLE, &mapping->flags); } diff --git a/mm/compaction.c b/mm/compaction.c index a3d2b132df52..e0e439b105b5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -862,6 +862,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* Time to isolate some pages for migration */ for (; low_pfn < end_pfn; low_pfn++) { + bool is_dirty, is_unevictable; if (skip_on_failure && low_pfn >= next_skip_pfn) { /* @@ -1047,10 +1048,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!mapping && (folio_ref_count(folio) - 1) > folio_mapcount(folio)) goto isolate_fail_put; - /* The mapping truly isn't movable. */ - if (mapping && mapping_unmovable(mapping)) - goto isolate_fail_put; - /* * Only allow to migrate anonymous pages in GFP_NOFS context * because those do not depend on fs locks. @@ -1062,8 +1059,10 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!folio_test_lru(folio)) goto isolate_fail_put; + is_unevictable = folio_test_unevictable(folio); + /* Compaction might skip unevictable pages but CMA takes them */ - if (!(mode & ISOLATE_UNEVICTABLE) && folio_test_unevictable(folio)) + if (!(mode & ISOLATE_UNEVICTABLE) && is_unevictable) goto isolate_fail_put; /* @@ -1075,26 +1074,42 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio)) goto isolate_fail_put; - if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_dirty(folio)) { - bool migrate_dirty; + is_dirty = folio_test_dirty(folio); + + if (((mode & ISOLATE_ASYNC_MIGRATE) && is_dirty) + || (mapping && is_unevictable)) { + bool migrate_dirty = true; + bool is_unmovable; /* - * Only pages without mappings or that have a - * ->migrate_folio callback are possible to migrate - * without blocking. However, we can be racing with - * truncation so it's necessary to lock the page - * to stabilise the mapping as truncation holds - * the page lock until after the page is removed - * from the page cache. + * Only folios without mappings or that have + * a ->migrate_folio callback are possible to migrate + * without blocking. + * + * Folios from unmovable mappings are not migratable. + * + * However, we can be racing with truncation, which can + * free the mapping that we need to check. Truncation + * holds the folio lock until after the folio is removed + * from the page so holding it ourselves is sufficient. + * + * To avoid this folio locking to inspect every folio + * with mapping for being unmovable, we assume every + * such folio is also unevictable, which is a cheaper + * test. If our assumption goes wrong, it's not a bug, + * just potentially wasted cycles. */ if (!folio_trylock(folio)) goto isolate_fail_put; mapping = folio_mapping(folio); - migrate_dirty = !mapping || - mapping->a_ops->migrate_folio; + if ((mode & ISOLATE_ASYNC_MIGRATE) && is_dirty) { + migrate_dirty = !mapping || + mapping->a_ops->migrate_folio; + } + is_unmovable = mapping && mapping_unmovable(mapping); folio_unlock(folio); - if (!migrate_dirty) + if (!migrate_dirty || is_unmovable) goto isolate_fail_put; } diff --git a/virt/kvm/guest_mem.c b/virt/kvm/guest_mem.c index c81d2bb9ae93..85903c32163f 100644 --- a/virt/kvm/guest_mem.c +++ b/virt/kvm/guest_mem.c @@ -390,7 +390,7 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, inode->i_size = size; mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER); mapping_set_large_folios(inode->i_mapping); - mapping_set_unevictable(inode->i_mapping); + /* this also sets the mapping as unevictable */ mapping_set_unmovable(inode->i_mapping); fd = get_unused_fd_flags(0);