From patchwork Thu Oct 6 21:34:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1687202 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OeArOu0I; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mk4X01H6hz23jd for ; Fri, 7 Oct 2022 08:36:46 +1100 (AEDT) Received: from localhost ([::1]:55712 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ogYXs-0003HV-4y for incoming@patchwork.ozlabs.org; Thu, 06 Oct 2022 17:36:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53712) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogYWe-00039H-Sh for qemu-devel@nongnu.org; Thu, 06 Oct 2022 17:35:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:56320) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ogYWX-0006mP-5w for qemu-devel@nongnu.org; Thu, 06 Oct 2022 17:35:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665092120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3mgJt1ynd7ZiWF7e8csGQVMXxZYrxlOZeWtbVF2Vfrg=; b=OeArOu0IJVF075m2p2gbS6lZDS1X/CJocmVP3JoFlDpMWTMqLKyBwNkuw5n6sy+VozSjMV EqhXw1Cjhnn2YuOJqR4IMfUtKq1PN8V+jvhUL2gaWIh8fchQMhe8k1iuPNiJvHXJztrim/ 3IG8hQOpQQNplrw1lLQtvfJ4ge26SLI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-286-e7EdbSK8NmW4VkVoGpc7Sg-1; Thu, 06 Oct 2022 17:35:19 -0400 X-MC-Unique: e7EdbSK8NmW4VkVoGpc7Sg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CCE8A811E67; Thu, 6 Oct 2022 21:35:18 +0000 (UTC) Received: from localhost (unknown [10.39.193.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 039152166B2E; Thu, 6 Oct 2022 21:35:17 +0000 (UTC) From: Stefan Hajnoczi To: qemu-devel@nongnu.org Cc: Yanan Wang , sgarzare@redhat.com, "Richard W.M. Jones" , Fam Zheng , Hanna Reitz , David Hildenbrand , integration@gluster.org, qemu-block@nongnu.org, Vladimir Sementsov-Ogievskiy , Paolo Bonzini , Kevin Wolf , afaria@redhat.com, Richard Henderson , Eric Blake , "Michael S. Tsirkin" , Thomas Huth , Xie Changlong , John Snow , Eduardo Habkost , Jeff Cody , Stefan Hajnoczi , "Denis V. Lunev" , Markus Armbruster , Laurent Vivier , Peter Xu , Raphael Norwitz , Marcel Apfelbaum , =?utf-8?q?Philippe_Mathieu-D?= =?utf-8?q?aud=C3=A9?= , Wen Congyang Subject: [PATCH v6 03/13] numa: call ->ram_block_removed() in ram_block_notifer_remove() Date: Thu, 6 Oct 2022 17:34:57 -0400 Message-Id: <20221006213507.645402-4-stefanha@redhat.com> In-Reply-To: <20221006213507.645402-1-stefanha@redhat.com> References: <20221006213507.645402-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Received-SPF: pass client-ip=170.10.133.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" When a RAMBlockNotifier is added, ->ram_block_added() is called with all existing RAMBlocks. There is no equivalent ->ram_block_removed() call when a RAMBlockNotifier is removed. The util/vfio-helpers.c code (the sole user of RAMBlockNotifier) is fine with this asymmetry because it does not rely on RAMBlockNotifier for cleanup. It walks its internal list of DMA mappings and unmaps them by itself. Future users of RAMBlockNotifier may not have an internal data structure that records added RAMBlocks so they will need ->ram_block_removed() callbacks. This patch makes ram_block_notifier_remove() symmetric with respect to callbacks. Now util/vfio-helpers.c needs to unmap remaining DMA mappings after ram_block_notifier_remove() has been called. This is necessary since users like block/nvme.c may create additional DMA mappings that do not originate from the RAMBlockNotifier. Reviewed-by: David Hildenbrand Signed-off-by: Stefan Hajnoczi --- hw/core/numa.c | 17 +++++++++++++++++ util/vfio-helpers.c | 5 ++++- 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/hw/core/numa.c b/hw/core/numa.c index 26d8e5f616..31e6fe1caa 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -822,6 +822,19 @@ static int ram_block_notify_add_single(RAMBlock *rb, void *opaque) return 0; } +static int ram_block_notify_remove_single(RAMBlock *rb, void *opaque) +{ + const ram_addr_t max_size = qemu_ram_get_max_length(rb); + const ram_addr_t size = qemu_ram_get_used_length(rb); + void *host = qemu_ram_get_host_addr(rb); + RAMBlockNotifier *notifier = opaque; + + if (host) { + notifier->ram_block_removed(notifier, host, size, max_size); + } + return 0; +} + void ram_block_notifier_add(RAMBlockNotifier *n) { QLIST_INSERT_HEAD(&ram_list.ramblock_notifiers, n, next); @@ -835,6 +848,10 @@ void ram_block_notifier_add(RAMBlockNotifier *n) void ram_block_notifier_remove(RAMBlockNotifier *n) { QLIST_REMOVE(n, next); + + if (n->ram_block_removed) { + qemu_ram_foreach_block(ram_block_notify_remove_single, n); + } } void ram_block_notify_add(void *host, size_t size, size_t max_size) diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c index 5ba01177bf..0d1520caac 100644 --- a/util/vfio-helpers.c +++ b/util/vfio-helpers.c @@ -847,10 +847,13 @@ void qemu_vfio_close(QEMUVFIOState *s) if (!s) { return; } + + ram_block_notifier_remove(&s->ram_notifier); + for (i = 0; i < s->nr_mappings; ++i) { qemu_vfio_undo_mapping(s, &s->mappings[i], NULL); } - ram_block_notifier_remove(&s->ram_notifier); + g_free(s->usable_iova_ranges); s->nb_iova_ranges = 0; qemu_vfio_reset(s);