{"id":2227392,"url":"http://patchwork.ozlabs.org/api/patches/2227392/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/patch/20260423154525.10292-1-jmarcin@redhat.com/","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":"","list_archive_url":"","list_archive_url_format":"","commit_url_format":""},"msgid":"<20260423154525.10292-1-jmarcin@redhat.com>","list_archive_url":null,"date":"2026-04-23T15:45:20","name":"[RFC] migration: Synchronize CPUs sooner in postcopy switchover phase","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"529a0434a1075c049448bd93224e5294a481843d","submitter":{"id":89058,"url":"http://patchwork.ozlabs.org/api/people/89058/?format=json","name":"Juraj Marcin","email":"jmarcin@redhat.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/patch/20260423154525.10292-1-jmarcin@redhat.com/mbox/","series":[{"id":501210,"url":"http://patchwork.ozlabs.org/api/series/501210/?format=json","web_url":"http://patchwork.ozlabs.org/project/qemu-devel/list/?series=501210","date":"2026-04-23T15:45:20","name":"[RFC] migration: Synchronize CPUs sooner in postcopy switchover phase","version":1,"mbox":"http://patchwork.ozlabs.org/series/501210/mbox/"}],"comments":"http://patchwork.ozlabs.org/api/patches/2227392/comments/","check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/2227392/checks/","tags":{},"related":[],"headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=N2mom7vT;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists1p.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4g1gT33KPsz1yD5\n\tfor ; Fri, 24 Apr 2026 01:47:17 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists1p.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1wFwGC-0005Sh-Gh; Thu, 23 Apr 2026 11:46:38 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1wFwFq-0005ST-7C\n for qemu-devel@nongnu.org; Thu, 23 Apr 2026 11:46:14 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1wFwFn-0007Kk-Uh\n for qemu-devel@nongnu.org; Thu, 23 Apr 2026 11:46:13 -0400","from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-595-aeA0Byb6Nd-Al1GiaGTttg-1; Thu,\n 23 Apr 2026 11:46:02 -0400","from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id CDCC7195608F; Thu, 23 Apr 2026 15:46:00 +0000 (UTC)","from fedora (unknown [10.43.3.182])\n by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP\n id DD714180047F; Thu, 23 Apr 2026 15:45:58 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1776959166;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding;\n bh=whbzFQ4VHP3kJIXELgoI1gtvCiwr1NhPhtpv2xU4JYM=;\n b=N2mom7vT9bX19+8pe+zCnCzb5Mk+B7WF1aI0XPXb6YL0/iXXJMY6WCjNPrMMpHXzHu2K+u\n 269DzawekedzkaQPOskCHf2WUnDyM69Z3CO9V+ajn9pftWW7bU3vEQ+PFfbb9rkRqFM+Rg\n bkiXoZsES/20qIgkVm31/EYVS3TzKkE=","X-MC-Unique":"aeA0Byb6Nd-Al1GiaGTttg-1","X-Mimecast-MFC-AGG-ID":"aeA0Byb6Nd-Al1GiaGTttg_1776959161","From":"Juraj Marcin ","To":"qemu-devel@nongnu.org","Cc":"Juraj Marcin , Jan Kiszka ,\n Peter Xu , Paolo Bonzini ,\n Fabiano Rosas ","Subject":"[RFC PATCH] migration: Synchronize CPUs sooner in postcopy switchover\n phase","Date":"Thu, 23 Apr 2026 17:45:20 +0200","Message-ID":"<20260423154525.10292-1-jmarcin@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=UTF-8","Content-Transfer-Encoding":"8bit","X-Scanned-By":"MIMEDefang 3.4.1 on 10.30.177.111","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=jmarcin@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"12","X-Spam_score":"1.2","X-Spam_bar":"+","X-Spam_report":"(1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001,\n SPF_PASS=-0.001 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"qemu development ","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"},"content":"From: Juraj Marcin \n\nPreviously, the post init CPU synchronization with the accelerator, like\nKVM, was performed in the bottom half of the POSTCOPY_RUN command\nhandler. However, this causes several problems.\n\nFirst issue is that if CPU synchronization fails, the destination QEMU\ncrashes. However, it is too late to recover the source side as the\nresponse to special PING has been already sent, and both sides are\nalready in the POSTCOPY_ACTIVE state.\n\nBy moving synchronization before responding, if the machine crashes, the\nresponse is never sent and the source side can resume from the\nPOSTCOPY_DEVICE state.\n\nSecond issue is caused when migration is paused due to a network\nfailure or user command right after transitioning to the POSTCOPY_ACTIVE\nstate and the CPU synchronization causes a page fault. This page fault\nblocks the CPU and the main QEMU threads and cannot be resolved until\npostcopy migration is recovered. However, as libvirt also tries to\nexecute 'cont' QMP command at this time (destination side transitions\nfrom POSTCOPY_DEVICE to POSTCOPY_ACTIVE), it will also halt waiting for\nthe response from the blocked main QEMU thread, unaware of the fact that\nthe migration is paused and needs to be recovered. Thus, it will wait\nindefinitely and never report postcopy migration error.\n\nWhen the CPU synchronization happens sooner, and the network fails\nduring it, the source side can transition from POSTCOPY_DEVICE to FAILED\nstate and resume safely. If migration is paused later, the main thread\nwon't be blocked by CPU synchronization and can respond to libvirt.\n\nSigned-off-by: Juraj Marcin \n---\nI am posting this as RFC to discuss the point at which the CPU\nsynchronization should happen.\n\nFor the POSTCOPY_DEVICE state to be effective, this synchronization must\nhappen before the destination machine responds to the special PING\ncommand. This leaves us with 2 options:\n\n1) In the PING command handler before responding to the specific\n request, as proposed in this patch.\n\n2) After loading CPU VMSD, for example in the post_load hook.\n\nThe first solution limits the number of places where synchronization\nneeds to happen, however, having it in the PING command handler feels\nsomewhat hacky.\n\nI have also tested the second solution, and while it seems natural to\nsynchronize the CPU after its data is loaded in the post_load, there are\nmultiple CPU types and VMSDs and each one would need to call\nsynchronization in its post_load hook. This would basically revert Jan\nKiszka's commit [1] which refactored CPU synchronization and united it\nto one place.\n\nKind of middle ground idea I have, is to add a special flag to VMSDs\nthat contain CPU data. After loading such VMSD the loadvm core would\nthen run CPU synchronization for that specific CPU. This solution would\nbring the synchronization close to the actual load from the migration\nstream but also keep it in one place.\n\n[1]: ea375f9ab8c7 (\"KVM: Rework VCPU state writeback API\")\n---\n migration/migration.c | 1 +\n migration/migration.h | 7 +++++++\n migration/savevm.c | 16 +++++++++++++---\n 3 files changed, 21 insertions(+), 3 deletions(-)","diff":"diff --git a/migration/migration.c b/migration/migration.c\nindex 5c9aaa6e58..9753f7dc26 100644\n--- a/migration/migration.c\n+++ b/migration/migration.c\n@@ -320,6 +320,7 @@ void migration_object_init(void)\n current_incoming->page_requested = g_tree_new(page_request_addr_cmp);\n \n current_incoming->exit_on_error = INMIGRATE_DEFAULT_EXIT_ON_ERROR;\n+ current_incoming->postcopy_device_cpu_synchronized = false;\n \n migration_object_check(current_migration, &error_fatal);\n \ndiff --git a/migration/migration.h b/migration/migration.h\nindex b6888daced..799a686a0b 100644\n--- a/migration/migration.h\n+++ b/migration/migration.h\n@@ -210,6 +210,13 @@ struct MigrationIncomingState {\n */\n QemuSemaphore postcopy_pause_sem_fast_load;\n \n+ /*\n+ * CPUs have been synchronized during POSTCOPY_DEVICE state before\n+ * responding to a special PING to source. This means, synchronization is\n+ * not required later during loadvm_postcopy_handle_run_bh().\n+ */\n+ bool postcopy_device_cpu_synchronized;\n+\n /* List of listening socket addresses */\n SocketAddressList *socket_address_list;\n \ndiff --git a/migration/savevm.c b/migration/savevm.c\nindex dd58f2a705..ad00c85887 100644\n--- a/migration/savevm.c\n+++ b/migration/savevm.c\n@@ -2143,9 +2143,10 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)\n /* TODO we should move all of this lot into postcopy_ram.c or a shared code\n * in migration.c\n */\n- cpu_synchronize_all_post_init();\n-\n- trace_vmstate_downtime_checkpoint(\"dst-postcopy-bh-cpu-synced\");\n+ if (!mis->postcopy_device_cpu_synchronized) {\n+ cpu_synchronize_all_post_init();\n+ trace_vmstate_downtime_checkpoint(\"dst-postcopy-bh-cpu-synced\");\n+ }\n \n qemu_announce_self(&mis->announce_timer, migrate_announce_params());\n \n@@ -2510,6 +2511,15 @@ static int loadvm_process_command(QEMUFile *f, Error **errp)\n tmp32);\n return -1;\n }\n+ if (tmp32 == QEMU_VM_PING_PACKAGED_LOADED) {\n+ /*\n+ * Try synchronizing CPU before responding. If it fails, QEMU exits\n+ * and source side can resume.\n+ */\n+ cpu_synchronize_all_post_init();\n+ mis->postcopy_device_cpu_synchronized = true;\n+ trace_vmstate_downtime_checkpoint(\"dst-postcopy-bh-cpu-synced\");\n+ }\n migrate_send_rp_pong(mis, tmp32);\n return 0;\n \n","prefixes":["RFC"]}