[{"id":3348344,"web_url":"http://patchwork.ozlabs.org/comment/3348344/","msgid":"","list_archive_url":null,"date":"2024-07-18T15:56:33","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"Steve,\n\nOn Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n> What?\n\nThanks for trying out with the cpr-transfer series. I saw that that series\nmissed most of the cc list here, so I'm attaching the link here:\n\nhttps://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n\nI think most of my previous questions for exec() solution still are there,\nI'll try to summarize them all in this reply as much as I can.\n\n> \n> This patch series adds the live migration cpr-exec mode, which allows\n> the user to update QEMU with minimal guest pause time, by preserving\n> guest RAM in place, albeit with new virtual addresses in new QEMU, and\n> by preserving device file descriptors.\n> \n> The new user-visible interfaces are:\n> * cpr-exec (MigMode migration parameter)\n> * cpr-exec-command (migration parameter)\n\nI really, really hope we can avoid this..\n\nIt's super cumbersome to pass in a qemu cmdline in a qemu migration\nparameter.. if we can do that with generic live migration ways, I hope we\nstick with the clean approach.\n\n> * anon-alloc (command-line option for -machine)\n\nIgor questioned this, and I second his opinion.. We can leave the\ndiscussion there for this one.\n\n> \n> The user sets the mode parameter before invoking the migrate command.\n> In this mode, the user issues the migrate command to old QEMU, which\n> stops the VM and saves state to the migration channels. Old QEMU then\n> exec's new QEMU, replacing the original process while retaining its PID.\n> The user specifies the command to exec new QEMU in the migration parameter\n> cpr-exec-command. The command must pass all old QEMU arguments to new\n> QEMU, plus the -incoming option. Execution resumes in new QEMU.\n> \n> Memory-backend objects must have the share=on attribute, but\n> memory-backend-epc is not supported. The VM must be started\n> with the '-machine anon-alloc=memfd' option, which allows anonymous\n> memory to be transferred in place to the new process.\n> \n> Why?\n> \n> This mode has less impact on the guest than any other method of updating\n> in place.\n\nSo I wonder whether there's comparison between exec() and transfer mode\nthat you recently proposed.\n\nI'm asking because exec() (besides all the rest of things that I dislike on\nit in this approach..) should be simply slower, logically, due to the\nserialized operation to (1) tearing down the old mm, (2) reload the new\nELF, then (3) runs through the QEMU init process.\n\nIf with a generic migration solution, the dest QEMU can start running (2+3)\nconcurrently without even need to run (1).\n\nIn this whole process, I doubt (2) could be relatively fast, (3) I donno,\nmaybe it could be slow but I never measured; Paolo may have good idea as I\nknow he used to work on qboot.\n\nFor (1), I also doubt in your test cases it's fast, but it may not always\nbe fast. Consider the guest has a huge TBs of shared mem, even if the\nmemory will be completely shared between src/dst QEMUs, the pgtable won't!\nIt means if the TBs are mapped in PAGE_SIZE tearing down the src QEMU\npgtable alone can even take time, and that will be accounted in step (1)\nand further in exec() request.\n\nAll these fuss will be avoided if you use a generic live migration model\nlike cpr-transfer you proposed. That's also cleaner.\n\n> The pause time is much lower, because devices need not be torn\n> down and recreated, DMA does not need to be drained and quiesced, and minimal\n> state is copied to new QEMU. Further, there are no constraints on the guest.\n> By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,\n> and suspending plus resuming vfio devices adds multiple seconds to the\n> guest pause time. Lastly, there is no loss of connectivity to the guest,\n> because chardev descriptors remain open and connected.\n\nAgain, I raised the question on why this would matter, as after all mgmt\napp will need to coop with reconnections due to the fact they'll need to\nsupport a generic live migration, in which case reconnection is a must.\n\nSo far it doesn't sound like a performance critical path, for example, to\ndo the mgmt reconnects on the ports. So this might be an optimization that\nmost mgmt apps may not care much?\n\n> \n> These benefits all derive from the core design principle of this mode,\n> which is preserving open descriptors. This approach is very general and\n> can be used to support a wide variety of devices that do not have hardware\n> support for live migration, including but not limited to: vfio, chardev,\n> vhost, vdpa, and iommufd. Some devices need new kernel software interfaces\n> to allow a descriptor to be used in a process that did not originally open it.\n\nYes, I still think this is a great idea. It just can also be built on top\nof something else than exec().\n\n> \n> In a containerized QEMU environment, cpr-exec reuses an existing QEMU\n> container and its assigned resources. By contrast, consider a design in\n> which a new container is created on the same host as the target of the\n> CPR operation. Resources must be reserved for the new container, while\n> the old container still reserves resources until the operation completes.\n\nNote that if we need to share RAM anyway, the resources consumption should\nbe minimal, as mem should IMHO be the major concern (except CPU, but CPU\nisn't a concern in this scenario) in container world and here the shared\nguest mem shouldn't be accounted to the dest container. So IMHO it's about\nthe metadata QEMU/KVM needs to do the hypervisor work, it seems to me, and\nthat should be relatively small.\n\nIn that case I don't yet see it a huge improvement, if the dest container\nis cheap to initiate.\n\n> Avoiding over commitment requires extra work in the management layer.\n\nSo it would be nice to know what needs to be overcommitted here. I confess\nI don't know much on containerized VMs, so maybe the page cache can be a\nproblem even if shared. But I hope we can spell that out. Logically IIUC\nmemcg shouldn't account those page cache if preallocated, because memcg\naccounting should be done at folio allocations, at least, where the page\ncache should miss first (so not this case..).\n\n> This is one reason why a cloud provider may prefer cpr-exec. A second reason\n> is that the container may include agents with their own connections to the\n> outside world, and such connections remain intact if the container is reused.\n> \n> How?\n> \n> All memory that is mapped by the guest is preserved in place. Indeed,\n> it must be, because it may be the target of DMA requests, which are not\n> quiesced during cpr-exec. All such memory must be mmap'able in new QEMU.\n> This is easy for named memory-backend objects, as long as they are mapped\n> shared, because they are visible in the file system in both old and new QEMU.\n> Anonymous memory must be allocated using memfd_create rather than MAP_ANON,\n> so the memfd's can be sent to new QEMU. Pages that were locked in memory\n> for DMA in old QEMU remain locked in new QEMU, because the descriptor of\n> the device that locked them remains open.\n> \n> cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag,\n> and by sending the unique name and value of each descriptor to new QEMU\n> via CPR state.\n> \n> For device descriptors, new QEMU reuses the descriptor when creating the\n> device, rather than opening it again. The same holds for chardevs. For\n> memfd descriptors, new QEMU mmap's the preserved memfd when a ramblock\n> is created.\n> \n> CPR state cannot be sent over the normal migration channel, because devices\n> and backends are created prior to reading the channel, so this mode sends\n> CPR state over a second migration channel that is not visible to the user.\n> New QEMU reads the second channel prior to creating devices or backends.\n\nOh, maybe this is the reason that cpr-transfer will need a separate uri..\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=TMtY9GAm;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WPy9k2HVtz20B2\n\tfor ; Fri, 19 Jul 2024 01:57:14 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sUTUr-0005yT-BR; Thu, 18 Jul 2024 11:56:45 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sUTUp-0005xb-IS\n for qemu-devel@nongnu.org; Thu, 18 Jul 2024 11:56:43 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sUTUn-0001BB-7x\n for qemu-devel@nongnu.org; Thu, 18 Jul 2024 11:56:43 -0400","from mail-qv1-f69.google.com (mail-qv1-f69.google.com\n [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-660-a6knK6ABNFWj3KqxAufKzQ-1; Thu, 18 Jul 2024 11:56:38 -0400","by mail-qv1-f69.google.com with SMTP id\n 6a1803df08f44-6b7678caf7dso2948526d6.1\n for ; Thu, 18 Jul 2024 08:56:38 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n af79cd13be357-7a19394f394sm37429185a.87.2024.07.18.08.56.35\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Thu, 18 Jul 2024 08:56:36 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1721318199;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=psdyXtxcjtiBBXhDetV+GB4YIl9/dNVKaz0XkWdAjgQ=;\n b=TMtY9GAmz1MxTuvMmTcC0V6C7rysLVnKqQu0i8zz8gaITjSUwmQ8Rp6OBuB5HRiYzBLe7N\n jigi5wLequQrgNFEW4EYo4ZPXDYdOtjkip6q/NFNsn2mWgT91Mt4KZPiKefudUm29Re3CG\n Z7mGOR+lt6XgGX6If1GmuchPPOvDOUM=","X-MC-Unique":"a6knK6ABNFWj3KqxAufKzQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1721318198; x=1721922998;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=psdyXtxcjtiBBXhDetV+GB4YIl9/dNVKaz0XkWdAjgQ=;\n b=DNCKfyi+DEfeFJ00c3rcCus77V1phpb8bfT/pFPUCep3Rk9SxXHIwdjQir5ntPum2g\n +NmPyibQsFeYXftgaSuDKQ0o445BM5+2zWP/uEAeMVWlVaNSYy8LtIa/uylUchIg0Mj6\n h7zpHsYd1OIzu6HAaAN5L2rQcrMC5cdf8ZL3dgDV0p8vQrGB3dXAI+Vs/NXiWHldZquB\n Gr9tgjLp/Nfnu+NVCGsqep8pW9YpxKzKUZ0zXrztcDi8GaDSQrGRYmpN2y+lc9k4Y+rY\n 70NdgnCppJcqwTz8Ypu9NzBby4TAyFV+/Hx5GCEDovrWzc4dG5Wf55BctfTc03M9gmHY\n +aDQ==","X-Gm-Message-State":"AOJu0YxF9XtVPUd+zUk/RoDOjmFZWg5/QV74EB1vfgQ5pLph81zN6yUG\n G4R4zdeuveM9s7TBB40oyezgAW8oT12cIawqQPvJjV6A0o4ABAqc8QaK1z+ljLTQzNAPyyMCfRz\n AlxNIi/QETWmUA9hMpYdb0sEW4Nc0elkQGTeig7eY0+5eko8ehq88","X-Received":["by 2002:a05:620a:450d:b0:79f:9fa:8222 with SMTP id\n af79cd13be357-7a196e16624mr449385a.6.1721318197431;\n Thu, 18 Jul 2024 08:56:37 -0700 (PDT)","by 2002:a05:620a:450d:b0:79f:9fa:8222 with SMTP id\n af79cd13be357-7a196e16624mr446685a.6.1721318196924;\n Thu, 18 Jul 2024 08:56:36 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IEKEKyz2SvFQq+ATqgvUg783HbLQOwEhrNH0klZWRPbnI9ftLirY3Uw8W1KNdyHOPFRnDkUQQ==","Date":"Thu, 18 Jul 2024 11:56:33 -0400","From":"Peter Xu ","To":"Steve Sistare ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini ,\n \"Daniel P. Berrange\" ,\n Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3349197,"web_url":"http://patchwork.ozlabs.org/comment/3349197/","msgid":"<4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>","list_archive_url":null,"date":"2024-07-20T21:26:07","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 7/18/2024 11:56 AM, Peter Xu wrote:\n> Steve,\n> \n> On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n>> What?\n> \n> Thanks for trying out with the cpr-transfer series. I saw that that series\n> missed most of the cc list here, so I'm attaching the link here:\n> \n> https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n> \n> I think most of my previous questions for exec() solution still are there,\n> I'll try to summarize them all in this reply as much as I can.\n> \n>>\n>> This patch series adds the live migration cpr-exec mode, which allows\n>> the user to update QEMU with minimal guest pause time, by preserving\n>> guest RAM in place, albeit with new virtual addresses in new QEMU, and\n>> by preserving device file descriptors.\n>>\n>> The new user-visible interfaces are:\n>> * cpr-exec (MigMode migration parameter)\n>> * cpr-exec-command (migration parameter)\n> \n> I really, really hope we can avoid this..\n> \n> It's super cumbersome to pass in a qemu cmdline in a qemu migration\n> parameter.. if we can do that with generic live migration ways, I hope we\n> stick with the clean approach.\n\nThis is no different than live migration, requiring a management agent to\nlaunch target qemu with all the arguments use to start source QEMU. Now that\nsame agent will send the arguments via cpr-exec-command.\n\n>> * anon-alloc (command-line option for -machine)\n> \n> Igor questioned this, and I second his opinion.. We can leave the\n> discussion there for this one.\n\nContinued on the other thread.\n\n>> The user sets the mode parameter before invoking the migrate command.\n>> In this mode, the user issues the migrate command to old QEMU, which\n>> stops the VM and saves state to the migration channels. Old QEMU then\n>> exec's new QEMU, replacing the original process while retaining its PID.\n>> The user specifies the command to exec new QEMU in the migration parameter\n>> cpr-exec-command. The command must pass all old QEMU arguments to new\n>> QEMU, plus the -incoming option. Execution resumes in new QEMU.\n>>\n>> Memory-backend objects must have the share=on attribute, but\n>> memory-backend-epc is not supported. The VM must be started\n>> with the '-machine anon-alloc=memfd' option, which allows anonymous\n>> memory to be transferred in place to the new process.\n>>\n>> Why?\n>>\n>> This mode has less impact on the guest than any other method of updating\n>> in place.\n> \n> So I wonder whether there's comparison between exec() and transfer mode\n> that you recently proposed.\n\nNot yet, but I will measure it.\n\n> I'm asking because exec() (besides all the rest of things that I dislike on\n> it in this approach..) should be simply slower, logically, due to the\n> serialized operation to (1) tearing down the old mm, (2) reload the new\n> ELF, then (3) runs through the QEMU init process.\n> \n> If with a generic migration solution, the dest QEMU can start running (2+3)\n> concurrently without even need to run (1).\n> \n> In this whole process, I doubt (2) could be relatively fast, (3) I donno,\n> maybe it could be slow but I never measured; Paolo may have good idea as I\n> know he used to work on qboot.\n\nWe'll see, but in any case these take < 100 msec, which is a wonderfully short\npause time unless your customer is doing high speed stock trading. If cpr-transfer\nis faster still, that's gravy, but cpr-exec is still great.\n\n> For (1), I also doubt in your test cases it's fast, but it may not always\n> be fast. Consider the guest has a huge TBs of shared mem, even if the\n> memory will be completely shared between src/dst QEMUs, the pgtable won't!\n> It means if the TBs are mapped in PAGE_SIZE tearing down the src QEMU\n> pgtable alone can even take time, and that will be accounted in step (1)\n> and further in exec() request.\n\nYes, there is an O(n) effect here, but it is a fast O(n) when the memory is\nbacked by huge pages. In UEK, we make it faster still by unmapping in parallel\nwith multiple threads. I don't have the data handy but can share after running\nsome experiments. Regardless, this time is negligible for small and medium\nsize guests, which form the majority of instances in a cloud.\n\n> All these fuss will be avoided if you use a generic live migration model\n> like cpr-transfer you proposed. That's also cleaner.\n> \n>> The pause time is much lower, because devices need not be torn\n>> down and recreated, DMA does not need to be drained and quiesced, and minimal\n>> state is copied to new QEMU. Further, there are no constraints on the guest.\n>> By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,\n>> and suspending plus resuming vfio devices adds multiple seconds to the\n>> guest pause time. Lastly, there is no loss of connectivity to the guest,\n>> because chardev descriptors remain open and connected.\n> \n> Again, I raised the question on why this would matter, as after all mgmt\n> app will need to coop with reconnections due to the fact they'll need to\n> support a generic live migration, in which case reconnection is a must.\n> \n> So far it doesn't sound like a performance critical path, for example, to\n> do the mgmt reconnects on the ports. So this might be an optimization that\n> most mgmt apps may not care much?\n\nPerhaps. I view the chardev preservation as nice to have, but not essential.\nIt does not appear in this series, other than in docs. It's easy to implement\ngiven the CPR foundation. I suggest we continue this discussion when I post\nthe chardev series, so we can focus on the core functionality.\n\n>> These benefits all derive from the core design principle of this mode,\n>> which is preserving open descriptors. This approach is very general and\n>> can be used to support a wide variety of devices that do not have hardware\n>> support for live migration, including but not limited to: vfio, chardev,\n>> vhost, vdpa, and iommufd. Some devices need new kernel software interfaces\n>> to allow a descriptor to be used in a process that did not originally open it.\n> \n> Yes, I still think this is a great idea. It just can also be built on top\n> of something else than exec().\n> \n>>\n>> In a containerized QEMU environment, cpr-exec reuses an existing QEMU\n>> container and its assigned resources. By contrast, consider a design in\n>> which a new container is created on the same host as the target of the\n>> CPR operation. Resources must be reserved for the new container, while\n>> the old container still reserves resources until the operation completes.\n> \n> Note that if we need to share RAM anyway, the resources consumption should\n> be minimal, as mem should IMHO be the major concern (except CPU, but CPU\n> isn't a concern in this scenario) in container world and here the shared\n> guest mem shouldn't be accounted to the dest container. So IMHO it's about\n> the metadata QEMU/KVM needs to do the hypervisor work, it seems to me, and\n> that should be relatively small.\n> \n> In that case I don't yet see it a huge improvement, if the dest container\n> is cheap to initiate.\n\nIt's about reserving memory and CPUs, and transferring those reservations from\nthe old instance to the new, and fiddling with the OS mechanisms that enforce\nreservations and limits. The devil is in the details, and with the exec model,\nthe management agent can ignore all of that.\n\nYou don't see it as a huge improvement because you don't need to write the\nmanagement code. I do!\n\nBoth modes are valid and useful - exec in container, or launch a new container.\nI have volunteered to implement the cpr-transfer mode for the latter, a mode\nI do not use. Please don't reward me by dropping the mode I care about :)\nBoth modes can co-exist. The presence of the cpr-exec specific code in qemu\nwill not hinder future live migration development.\n\n>> Avoiding over commitment requires extra work in the management layer.\n> \n> So it would be nice to know what needs to be overcommitted here. I confess\n> I don't know much on containerized VMs, so maybe the page cache can be a\n> problem even if shared. But I hope we can spell that out. Logically IIUC\n> memcg shouldn't account those page cache if preallocated, because memcg\n> accounting should be done at folio allocations, at least, where the page\n> cache should miss first (so not this case..).\n> \n>> This is one reason why a cloud provider may prefer cpr-exec. A second reason\n>> is that the container may include agents with their own connections to the\n>> outside world, and such connections remain intact if the container is reused.\n>>\n>> How?\n\nchardev preservation. The qemu socket chardevs to these agents are preserved,\nand the agent connections to the outside world do not change, so no one sees\nany interruption of traffic.\n\n>> All memory that is mapped by the guest is preserved in place. Indeed,\n>> it must be, because it may be the target of DMA requests, which are not\n>> quiesced during cpr-exec. All such memory must be mmap'able in new QEMU.\n>> This is easy for named memory-backend objects, as long as they are mapped\n>> shared, because they are visible in the file system in both old and new QEMU.\n>> Anonymous memory must be allocated using memfd_create rather than MAP_ANON,\n>> so the memfd's can be sent to new QEMU. Pages that were locked in memory\n>> for DMA in old QEMU remain locked in new QEMU, because the descriptor of\n>> the device that locked them remains open.\n>>\n>> cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag,\n>> and by sending the unique name and value of each descriptor to new QEMU\n>> via CPR state.\n>>\n>> For device descriptors, new QEMU reuses the descriptor when creating the\n>> device, rather than opening it again. The same holds for chardevs. For\n>> memfd descriptors, new QEMU mmap's the preserved memfd when a ramblock\n>> is created.\n>>\n>> CPR state cannot be sent over the normal migration channel, because devices\n>> and backends are created prior to reading the channel, so this mode sends\n>> CPR state over a second migration channel that is not visible to the user.\n>> New QEMU reads the second channel prior to creating devices or backends.\n> \n> Oh, maybe this is the reason that cpr-transfer will need a separate uri..\n\nIndeed.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=WMXqbGoG;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=Ox7F7mZ+;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WRKPs4Crqz1xrQ\n\tfor ; Sun, 21 Jul 2024 07:27:25 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sVHaz-0008Vz-Qd; Sat, 20 Jul 2024 17:26:25 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sVHax-0008U6-6v\n for qemu-devel@nongnu.org; Sat, 20 Jul 2024 17:26:23 -0400","from mx0b-00069f02.pphosted.com ([205.220.177.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sVHau-0007o2-BO\n for qemu-devel@nongnu.org; Sat, 20 Jul 2024 17:26:22 -0400","from pps.filterd (m0246630.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 46KLFYHX023636;\n Sat, 20 Jul 2024 21:26:16 GMT","from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com\n (phxpaimrmta01.appoci.oracle.com [138.1.114.2])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gn05g07p-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sat, 20 Jul 2024 21:26:16 +0000 (GMT)","from pps.filterd\n (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 46KIMhJX024450; Sat, 20 Jul 2024 21:26:15 GMT","from nam02-dm3-obe.outbound.protection.outlook.com\n (mail-dm3nam02lp2041.outbound.protection.outlook.com [104.47.56.41])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id\n 40g3p609mb-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sat, 20 Jul 2024 21:26:15 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by DM3PR10MB7946.namprd10.prod.outlook.com (2603:10b6:0:4b::7) with\n Microsoft\n SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id\n 15.20.7784.17; Sat, 20 Jul 2024 21:26:12 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%6]) with mapi id 15.20.7784.016; Sat, 20 Jul 2024\n 21:26:11 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=G9vBRQEPxZ+QtlCo0Y1jjYdkkcyukjP+5CeUaLyZZ9Y=; b=\n WMXqbGoG8zxwszWLATWDxp5MGq+GikpCo4Z42baJoMKXO07/61i/VWqU3MXmObjH\n NS2hinl3e/ETwzRxOi1YpUfBLTpZLcgpNAnEN2V+EFNGFhLsJkYV1yoaKYOgE1cr\n WUDWwCZYAC6KsSsPfYEKiPeNthEg1H83tTSZxbuszC/yrmXX//sm7ShW866xDCgV\n Q+YWyDizYgbOFP+iy/PXZsU0KqgTkL/FW2w3Xf93DesijoKUwcTWtfuKEp+1/V70\n TMJqAb/VjE6q4r8ld3y+fpFpfuz8ddoCrBSk7FkXHdOpKqgNC8UyQpe9Jspbt2hF\n Ijfh7Ji/wlwKQW4iy88f3Q==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=G9vBRQEPxZ+QtlCo0Y1jjYdkkcyukjP+5CeUaLyZZ9Y=;\n b=Ox7F7mZ+xUfwdmwEWGO3V3xyX9iynQj57D1l74YNiO77GktyqUqmNDksE4pseNSTUjiw7f7GpG6xjVu6xApnYLcnYw/YWyTA9KEHanUVQA1Hsnga+n6e1zjJ9rmp++pMpvt7MNbpJmzqbcEowbPSBTim7aNs6z1whikfdqli7RU="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=Nz2yI+M7PVu4Dm8I1e3hFBrXTtaczXDCMFNg5gaKo1v68yN4jPd676jNr0yGDr9e4bMvDrv68okOAyqlj2vBEE8Djk0OupJHX4I643pQGOoMGaqypHI6KU0C55jP9icWVGpph/LwVnC90FmzVV/ojw6+otdUdGxgu3HJ52gZyYcXHqeaN8T6Ny0ufYgfw8ER8DRbNQy4nrY1GkoZqtcjPvsj+ZQdBHYFkFGHgbG40Qgf+52NkOVwgtdjxQxSECWwKp65XdLvtxCtxp3Ey63yRGreZvC1AJCfOnS22gx6K/5Bd2SguC2Djzx3QNT0unE4qD6UOh31x3CztW0pWC3Ynw==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=G9vBRQEPxZ+QtlCo0Y1jjYdkkcyukjP+5CeUaLyZZ9Y=;\n b=QsECwIaOpbpaRvD/ch3eHGxqD8b4ovrNY82/JcgIon63BagtnEAxGen+REO9r54yew++QhTQkfaHVCvHpQrceVNyy8d1DJsCPTsNs5RJLP2JkBP9Hyh9mvLOMHd7p60ihMlh/ORvr2S2h0VBSTJAcnOeQnqbWDVumQlN9Lpgds5KbwO5Xh4YOH1Gkma4Uk9XcA836UDLu0BEMG2sPYs3IkMZHSIrUQ95NqA3lNJeF8DV5BsO5+8jgW6jJMIogysLdoQQbrWfXaklwSgJmKHTtYcWAuO/gtE1l3sfNFU8zAmQviBHrs54PID6smM8EyXRtXPt2X/IPc2AZ+Febbp4Lg==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"<4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>","Date":"Sat, 20 Jul 2024 17:26:07 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-ClientProxiedBy":"BYAPR11CA0076.namprd11.prod.outlook.com\n (2603:10b6:a03:f4::17) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|DM3PR10MB7946:EE_","X-MS-Office365-Filtering-Correlation-Id":"409d9996-6333-46ed-0d12-08dca90293fd","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|366016|1800799024|7416014|376014;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?qV+/kxwGlVOyt1fr536PQ3IFdPapxEJ?=\n\t=?utf-8?q?K6578HwcSXfoBrfjTSDPa3AgIck0IYooFRSl7tL0IBe9o9d6lrFAAMKKaZKbiyp3i?=\n\t=?utf-8?q?ymH/eR97ZWHmMzOZfmrgvtmbqyhRyWkiEhIggRrZ9QaWBCUZ+r6cAdmorB8A34vHi?=\n\t=?utf-8?q?QW3Rrc5H/pNSqtqmTmVl0dXvknAvYwXlhedPII3eIiNqCFCsCFs+2KEB+yp6JuiTj?=\n\t=?utf-8?q?xPry/EOZuICk/fAuWacq2oxKGzpXWqidjHcmwlmh8JpY9g1ky+VQ5wrjMwD4eeWVD?=\n\t=?utf-8?q?1iZsM/myFYC/M+rMaRzgDnPdAg3nsUUMgs+4/lhTLjpnQlcrqUw40gDK18bQoALGE?=\n\t=?utf-8?q?LGEtJcnUNf8gQbKV+XpzafoMeJKgwmdVHfOZfN73IdjsOmrBmutWWOclzu0edjUmQ?=\n\t=?utf-8?q?wtTnM5wp1NnUlseNBKcvo4NhJ7/QKEy4adUDg4oXdHxmpy4Yb+q1Ro5QXjUKKPw0F?=\n\t=?utf-8?q?hv9iAZvmGJPDcSxpkCetdSvizwFwRpVmcVt5XZbMEqNDGFNdnaRN8cxjZvGhai9cY?=\n\t=?utf-8?q?jpNdfC1HvxEw/bwit2gJoj/H5joFBMm7+NEBTVdAZ/haMybcMY0IgJtf3Wqto6BWq?=\n\t=?utf-8?q?rv1QI8NtMybQDyXWkZxTbhMAT4FrhyrNYoWKx8P5s2gJ1ZcqrmQkn3nKcMgvN1Vpy?=\n\t=?utf-8?q?IHHSaxeKnp2PqHzHrktJztjQg19B5o/Rn+Tdu+O8i3x4CoN+XAGvHmXlzFSGD8lW8?=\n\t=?utf-8?q?bUUMv/MUDcTwkQVQspDvleKNbGl+cU3Q7NK2rMhxarrE+OqLFddwSYm0PSSnWpR1n?=\n\t=?utf-8?q?42ciXrE14Lwh//v6KCYSN+W0af9zSXjzIFT/ppsXCi38Gc6BUj+yHDH4iEJsj0/og?=\n\t=?utf-8?q?6sTKSVWaqMVa+A8FFDKqQ2MndiWlzjyCoeUwpO5OftAUR3npUvt1ZLt6LdUub5vnF?=\n\t=?utf-8?q?4iSn3NX180ePfPthH1HuKMwQsFuuinaWgBjychhoAIDxlG/GgxSkCgyNasXH5HxaI?=\n\t=?utf-8?q?wutx4nFQFzo6/BSGPHgkeDZAvcZ4CzUItsVwpTQBmZ0oxmq+ihl2sjfVvQdEKSoRV?=\n\t=?utf-8?q?fno14zb391S0QHb6MUxyC5emt53fgoeXelhZDj6EnMkWTd+UE1SFe6DRES7NpZV77?=\n\t=?utf-8?q?55gKTAjrzxzMOa3jdRkOGj78BaKMIojnQD5eEo1MZZZuc/ZFBX1RwG9iivysBCoUP?=\n\t=?utf-8?q?ZFH6sZnoqhfMuy/8ibf6oWtnH4pCroFO/SKHRo/MCetO79KZ+bigWfqo+4M8hgckI?=\n\t=?utf-8?q?CIsmMZIcB7DYStTDXu+lBUemGgrjZOzInv1WCQwNXaGi39DyXxrZjFVxQ=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(1800799024)(7416014)(376014); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?yUNW2sc5JUnYXwnBKLTBUpbYVxZM?=\n\t=?utf-8?q?Q66SWfKaGFVnTyUIm1pxkT8uRhHEG21U7eUiSfKmXxh0VOMvZ+N5uu5+jEePc64F6?=\n\t=?utf-8?q?Z+2PU5jrgsTKYftLcVLxHSxt244h9wOM1VUS/gcUJjIi4EeEBXQFHa/41liD86ZM3?=\n\t=?utf-8?q?+v0geGaK0cJ1RauyzuG4XpB6wMINaJIhEsBjQfqiM3y+cvcHnmgeAZ2AjtA/nZc+C?=\n\t=?utf-8?q?mt0uedO+L/9/+axL/ppaGmGeJF+qGV1GgJ2V8iRftSS0WpmFjjtvzXQGDjkb9nxLB?=\n\t=?utf-8?q?pTay+6tmiyhvJtE+fy6+JazpV+FseYbr6HmSM6MLz4Zw3F2UED3PqgpROzw4pD04L?=\n\t=?utf-8?q?LG6vdhP7qAetVvdozH/1C+tIZwgKb7cEwiOd+j3WiRSfbHSPdaUv71SIvg9cNSwSE?=\n\t=?utf-8?q?oOTMauhrhSVeg/xReDo89zxery/iwKs+fNpz+DXdKebgmY/vc5S2HuVlCBceuLkmh?=\n\t=?utf-8?q?tLLSbBhN3xI3qK0BSJBb6SfmuuMbiljV12uoD9n0Vv37SzSBycXojb1DYxM+V3N49?=\n\t=?utf-8?q?ACC+em6q4MBQHDWVAdNOVxB05auh7g6pzzkgqAtFgGIXi2Rz2rPWCIsme9AZVU4bW?=\n\t=?utf-8?q?3Ycr9LE5mT+RfGHdwgTSozB0ALXbkUW/hPLOqMnDkTDDhAhaigQnItpamWUSZ2kLF?=\n\t=?utf-8?q?s83JkZjEzaKcCtyiEZFkBzM6/tJSpkLj2BYNOdLJi+ZjI2pzlynLMNKJTLZmnFaEc?=\n\t=?utf-8?q?nuxTfFbvcueXMgzN1MdkH4FE48Rw06YUiI7h6v952jA8ZDTqxrd0ccCBSC8MrUAPZ?=\n\t=?utf-8?q?Xm5uIHMLa4F70d6tyWU8O82N+OvSKBueazjAcuZw6nYZ45KFFsgpi9vTmty3iR9GP?=\n\t=?utf-8?q?asuAIdgDgmReNmG2c6KlXoGTalvBpGRXTHu+rhzuLt1XXVrdcYEEYRySWPsebsa1H?=\n\t=?utf-8?q?8BuEeKbVo8ymc9EdKaXWmNnHnuclRcHYGdqWUCkqxgcBz5/vW6EeKDhHed90+ssHS?=\n\t=?utf-8?q?U+0nKJPfbiyhs2qauB8oXhFaNGmQOBNQbVolIr1Xfxd+Ob/ej0mU1ilzcJP+3dfpn?=\n\t=?utf-8?q?i0S7QyY1X6SwsLX+hu5i6CP7nmN2y97QfmhXwBETA33GnNFrMogAiU/ZwmyLSPqlC?=\n\t=?utf-8?q?GpGDZekTknwS0LPyNLt7UQETDxFYaeSL115mAoMhWZQAEwF36fKwj8ImL29lkGHxn?=\n\t=?utf-8?q?cKX1d++yuEiB5U0OBig22OiwJbCHEEh514+bEITbWA1OMx0xULMjnqnmfvrsaAq7i?=\n\t=?utf-8?q?gurWrQ/pMYdBgheZ5bxczhXZnMiWU82oUzMw9BuRuz5kCNjSYo16DvQSDSqZ20Wgb?=\n\t=?utf-8?q?jByJWHR0Ex3rTse7qykVISPuMt3PyLEqEHO4A27P3m9Jp+QnwfV1kjldtHQlf8I40?=\n\t=?utf-8?q?6lYhHF5VZaWmj81pdXRzyScUoyRMkqplE88VOWisTSYUD+t3bpMOMCNk0ujZgDAzu?=\n\t=?utf-8?q?nvHa1aE03EC3aRiheJQbpiAT/wwRM8aYy5ZTIvVqnGVDKKEOha8EwRDp8cBbr+yr/?=\n\t=?utf-8?q?y7FBHdFxCoKYy+Ug18UgOkyuP+9ATMXLph7yQURl6t/1uQOagpPZ6kyoHSpIrxMTq?=\n\t=?utf-8?q?5x0L+hgqZ3U9C/rGs35vJWolH7U9BNCxPQ=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n izIGGNmUJGfDnHEWwTLIpXfUfJ/HkW0eAZeYTzvqhx8t5KZ1z4NlIsIPaT3DMdEhozfEldW1kwILjvxhoveOr+P7hqzHlsW082YVvWH3RRDB2ckT73AVPysa0WnDtF5E/mIey9R4I32hmPu7XJjbwKgJawtlw2UiHPLFcJzHjXYePGsCXWlWyUbbSbOtSwsbj+7zsxz9fg6DCyIWe1MCLGd8BA60EjhUGRFvU8Hx09SLZRG2JkLrLA7rZtskHSvBaKn1SdhZVKplly3Td/nXjfgdQs/cMmt1cSY1+FcTIk8PZSK0niW1Dg56Uwk8/vMiEHpKT+A5G/25H7TTgVQgBvHUx6wBz53Rw/JGh3zJsdVJHMlmdqRU9LtAjAqr+2oeNQye6J8yA2SPCLIUrKWtFI+aMKF9EX5MFBlDZsUpTZ7jXJM+g9g35QupGqWk/TVPE651DXHt7UZsKP3i//nYPq5ON6pRAe6wuo5Nz8uL8YYLekHxGRIzY8DEXVOlWFjCEbLcZh/MIaC7dwyo3uPhMFNtwZKoFzyMoJv5FMTm8TZYDoe+WEsYci3k2dvqu04mFyogu5QVb/9288nmypAEe9gePo5q9YQW6w0/av7/ms0=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 409d9996-6333-46ed-0d12-08dca90293fd","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"20 Jul 2024 21:26:11.9204 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n ykGsoLWxUAYEBr4UlZ91SiQ9jDOKMeJzxu+6A9/wHnxYjIbAAV7KEHAPwU6vB4Ul2nBBN5PAAkBpKVhInjSdRZVsRVd5NXpQhi25hhDgu4g=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"DM3PR10MB7946","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-07-20_19,2024-07-18_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 phishscore=0\n suspectscore=0 bulkscore=0\n mlxlogscore=999 adultscore=0 malwarescore=0 mlxscore=0 spamscore=0\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2407200157","X-Proofpoint-ORIG-GUID":"j0O-Xy4b1T69YdBtsQUB2KOOSezFHdKD","X-Proofpoint-GUID":"j0O-Xy4b1T69YdBtsQUB2KOOSezFHdKD","Received-SPF":"pass client-ip=205.220.177.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com","X-Spam_score_int":"-27","X-Spam_score":"-2.8","X-Spam_bar":"--","X-Spam_report":"(-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3349471,"web_url":"http://patchwork.ozlabs.org/comment/3349471/","msgid":"<37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>","list_archive_url":null,"date":"2024-07-22T08:59:47","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":70402,"url":"http://patchwork.ozlabs.org/api/people/70402/","name":"David Hildenbrand","email":"david@redhat.com"},"content":"On 18.07.24 17:56, Peter Xu wrote:\n> Steve,\n> \n> On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n>> What?\n> \n> Thanks for trying out with the cpr-transfer series. I saw that that series\n> missed most of the cc list here, so I'm attaching the link here:\n> \n> https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n> \n> I think most of my previous questions for exec() solution still are there,\n> I'll try to summarize them all in this reply as much as I can.\n> \n>>\n>> This patch series adds the live migration cpr-exec mode, which allows\n>> the user to update QEMU with minimal guest pause time, by preserving\n>> guest RAM in place, albeit with new virtual addresses in new QEMU, and\n>> by preserving device file descriptors.\n>>\n>> The new user-visible interfaces are:\n>> * cpr-exec (MigMode migration parameter)\n>> * cpr-exec-command (migration parameter)\n> \n> I really, really hope we can avoid this..\n> \n> It's super cumbersome to pass in a qemu cmdline in a qemu migration\n> parameter.. if we can do that with generic live migration ways, I hope we\n> stick with the clean approach.\n> \n>> * anon-alloc (command-line option for -machine)\n> \n> Igor questioned this, and I second his opinion.. We can leave the\n> discussion there for this one.\n> \n>>\n>> The user sets the mode parameter before invoking the migrate command.\n>> In this mode, the user issues the migrate command to old QEMU, which\n>> stops the VM and saves state to the migration channels. Old QEMU then\n>> exec's new QEMU, replacing the original process while retaining its PID.\n>> The user specifies the command to exec new QEMU in the migration parameter\n>> cpr-exec-command. The command must pass all old QEMU arguments to new\n>> QEMU, plus the -incoming option. Execution resumes in new QEMU.\n>>\n>> Memory-backend objects must have the share=on attribute, but\n>> memory-backend-epc is not supported. The VM must be started\n>> with the '-machine anon-alloc=memfd' option, which allows anonymous\n>> memory to be transferred in place to the new process.\n>>\n>> Why?\n>>\n>> This mode has less impact on the guest than any other method of updating\n>> in place.\n> \n> So I wonder whether there's comparison between exec() and transfer mode\n> that you recently proposed.\n> \n> I'm asking because exec() (besides all the rest of things that I dislike on\n> it in this approach..) should be simply slower, logically, due to the\n> serialized operation to (1) tearing down the old mm, (2) reload the new\n> ELF, then (3) runs through the QEMU init process.\n> \n> If with a generic migration solution, the dest QEMU can start running (2+3)\n> concurrently without even need to run (1).\n\nI'll note (not sure if already discussed) that with the \"async-teardown\" \noption we have a way to move the MM teardown to a separate process, such \nthat it will happen asynchronously.","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=ewelpZUq;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WSDl96MjCz1yZ7\n\tfor ; Mon, 22 Jul 2024 19:00:37 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sVotj-0001De-Qd; Mon, 22 Jul 2024 04:59:59 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sVoti-0001B4-J8\n for qemu-devel@nongnu.org; Mon, 22 Jul 2024 04:59:58 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sVotg-0003aK-Ju\n for qemu-devel@nongnu.org; Mon, 22 Jul 2024 04:59:58 -0400","from mail-wm1-f70.google.com (mail-wm1-f70.google.com\n [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-615-vtOlOUsyMJmi_eQ7jWYGKw-1; Mon, 22 Jul 2024 04:59:50 -0400","by mail-wm1-f70.google.com with SMTP id\n 5b1f17b1804b1-427e9c2f82cso2598325e9.1\n for ; Mon, 22 Jul 2024 01:59:50 -0700 (PDT)","from ?IPV6:2003:cb:c727:7000:c050:e303:f8a7:6ed9?\n (p200300cbc7277000c050e303f8a76ed9.dip0.t-ipconnect.de.\n [2003:cb:c727:7000:c050:e303:f8a7:6ed9])\n by smtp.gmail.com with ESMTPSA id\n ffacd0b85a97d-36878695b35sm7873295f8f.65.2024.07.22.01.59.47\n (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n Mon, 22 Jul 2024 01:59:48 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1721638793;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references:autocrypt:autocrypt;\n bh=/y7xSCwGc1J5Ir7ZI+sNuRPBjTMKYjecoutAJVBNdOo=;\n b=ewelpZUqPtZm45vNg6eUOzAVIivT9OjGh49upZ6xOq9a6SrV47eEQYfPzsarfhdjkIcjvp\n l5PZT0se7+qVCrbK3SJ7jarvpyaqG7UY6bsMChixf8Aby0lyr4hS/EIcYAsA3rwBimts2p\n HJkufw88RqPJoLWRFH/LuAQX9GS2Cw0=","X-MC-Unique":"vtOlOUsyMJmi_eQ7jWYGKw-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1721638789; x=1722243589;\n h=content-transfer-encoding:in-reply-to:organization:autocrypt:from\n :content-language:references:cc:to:subject:user-agent:mime-version\n :date:message-id:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=/y7xSCwGc1J5Ir7ZI+sNuRPBjTMKYjecoutAJVBNdOo=;\n b=MMRzd23Wcijjn0lQsZguhcla1kdsE1Uze8Iy5mCE90Cq+6RFfDAfRFwb6awy6qRKOy\n dIoG1DzCUA1aOCM6QAHPJpiYS54vug3nOq6b3Ah4tIeUAetYnYVbhawdMM8c3JVWHqu5\n d/jeeW7fswdG3nBruy4kYiVVkYjQUHss47ahCEtSQJoVqn1yjontLC0iTXwMBEDUhX/9\n kmI2a7MoikljBl5LjiFg2hBApMFiYhnJNpOlD5t8D4FFzGOulkAKp0Hbwy0BS566os8i\n nNr2ZUykiw1hIAdsS/gQRBGBUaFU4M1rzRZsBInQ25beqkHBXvSUG5laNT3SEk+iVvuh\n L7aQ==","X-Gm-Message-State":"AOJu0YwDnO7hHL8llsXRy+8CQTk57fdnCxSZH8TWcrbif77uvj8r35ER\n urfbFn6HCZj1CPwTLqevxQ4dJ84wVbXmt7LHCW+hI9k58hniPPQWjF7QD0fkhxhLGj24o7ugJcH\n tM9fQKM5mNB3FDsK0JJBRnidIMJ8QIvv++dXQ0BWKpW/bSE/ztQDI","X-Received":["by 2002:a5d:440f:0:b0:368:4626:1327 with SMTP id\n ffacd0b85a97d-369bae3f958mr3250396f8f.23.1721638789023;\n Mon, 22 Jul 2024 01:59:49 -0700 (PDT)","by 2002:a5d:440f:0:b0:368:4626:1327 with SMTP id\n ffacd0b85a97d-369bae3f958mr3250383f8f.23.1721638788595;\n Mon, 22 Jul 2024 01:59:48 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IFdg6wHGYW3VHOCdZF3nJw4Qj8Lg5wy1LBgo1WnaPAfSMIwLV2OEKtlwdiYdGaH/8NZ+4Z02g==","Message-ID":"<37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>","Date":"Mon, 22 Jul 2024 10:59:47 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu , Steve Sistare ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n ","Content-Language":"en-US","From":"David Hildenbrand ","Autocrypt":"addr=david@redhat.com; keydata=\n xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ\n dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL\n QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp\n XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK\n Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9\n PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt\n WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc\n UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv\n jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb\n B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk\n ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW\n AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q\n 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp\n rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf\n wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4\n 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l\n pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd\n KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE\n BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs\n 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF\n 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9\n M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz\n Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb\n T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A\n 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk\n CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G\n NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75\n 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx\n 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS\n lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv\n AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa\n N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3\n AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB\n boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq\n 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f\n XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ\n a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq\n Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6\n 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8\n kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E\n th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr\n jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt\n WNyWQQ==","Organization":"Red Hat","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=david@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.141,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3356716,"web_url":"http://patchwork.ozlabs.org/comment/3356716/","msgid":"","list_archive_url":null,"date":"2024-08-04T15:43:13","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Mon, Jul 22, 2024 at 10:59:47AM +0200, David Hildenbrand wrote:\n> > So I wonder whether there's comparison between exec() and transfer mode\n> > that you recently proposed.\n> > \n> > I'm asking because exec() (besides all the rest of things that I dislike on\n> > it in this approach..) should be simply slower, logically, due to the\n> > serialized operation to (1) tearing down the old mm, (2) reload the new\n> > ELF, then (3) runs through the QEMU init process.\n> > \n> > If with a generic migration solution, the dest QEMU can start running (2+3)\n> > concurrently without even need to run (1).\n> \n> I'll note (not sure if already discussed) that with the \"async-teardown\"\n> option we have a way to move the MM teardown to a separate process, such\n> that it will happen asynchronously.\n\nI just had a look, maybe it won't trivially work, as it relies on QEMU\nprocess to quit first..\n\nasync_teardown_fn():\n if (the_ppid == getppid()) {\n pause();\n }\n\nWhile if we stick with exec(), then PID shouldn't change, so the teardown\nprocess can hold the mm and pause until the VM is destroyed..\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=Cftr7vnd;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WcP523hbxz1yY3\n\tfor ; Mon, 5 Aug 2024 01:44:17 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sadOH-0005Ga-IT; Sun, 04 Aug 2024 11:43:25 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sadOF-0005F3-Q1\n for qemu-devel@nongnu.org; Sun, 04 Aug 2024 11:43:23 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sadOD-0005OJ-L3\n for qemu-devel@nongnu.org; Sun, 04 Aug 2024 11:43:22 -0400","from mail-ot1-f71.google.com (mail-ot1-f71.google.com\n [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-267-cBohN2xkNEKcZKLOzWu7hw-1; Sun, 04 Aug 2024 11:43:17 -0400","by mail-ot1-f71.google.com with SMTP id\n 46e09a7af769-70954de6142so1727165a34.0\n for ; Sun, 04 Aug 2024 08:43:17 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n 6a1803df08f44-6bb9c87c487sm26759996d6.131.2024.08.04.08.43.14\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Sun, 04 Aug 2024 08:43:15 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1722786198;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=LPBShP6gLkVLC4k9oCbDnZ/GHvgHuKYiNh1jOXPc5yE=;\n b=Cftr7vnd1Uog2KfappOsTFO+fxH8oAEvZ5BNxDZ/7cqCIs6mC45L0fYlVeVTSTxA+Tjis1\n 9+oWXXvZif2DYxZ1kwitB1ctdGEl6aiNhnIhCLF6e+IBHFyfeJyruLvvQ7iQ+bTzJMwq7T\n jt1T76bf+PreMVXcMt4Xsl3eMo/oef4=","X-MC-Unique":"cBohN2xkNEKcZKLOzWu7hw-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1722786196; x=1723390996;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=LPBShP6gLkVLC4k9oCbDnZ/GHvgHuKYiNh1jOXPc5yE=;\n b=ktkFXK04OZBZsN/u+m0yOGPbIsoHDS/ClD8B05QdRLasDLR9E0IWiratumsZUMR+s7\n 5BSOxf12nzk2Hld2YnplhsOAs52ULgoCY24+PtJiAb46GJAT/B3S9FEuNZ83zWS97xUs\n InueFtvUOS6i+miKClM4Yd6TLZ6WhtlMUg/YoSd8SgQqgWvjm2/7XeHAqNFQiCFFp7NC\n rHbthX5BOZ9P7f9Yw2RWfKKAYGBc2tOQsZRLf8Z3AnrByP/BR1PjRt0kG4iSWVcRFtL7\n dbsN72hZihyKJTM8DAh7xqVmsVExEICWJQ/a4NyKblPkog3qvULjCOmsAIjcd3Qh8E/b\n GOQQ==","X-Forwarded-Encrypted":"i=1;\n AJvYcCUOvc8WohLpOfiOeVdizk7viQra84+7oGyCumDsm75sPaY2YaNNuN5NHIIg+VKCR5oKTD8MAsl8/dsw@nongnu.org","X-Gm-Message-State":"AOJu0YzzGrrd37hW8OxLS5WWZTVCUWQFRdn/pAGI/l3s4+KFSrCC9TNY\n V8UFOEquyUoK1BGWYCv5ZJiFmyc7uthSb8inClyx9Hpo0D4ihgIDl5vUDLBYCZw0gXVOSCREElC\n DEIEGB+3jUNJjTyA5saaCk2LgUaCssm/Y0+5FjUQYcqJlJ+nkjNI7","X-Received":["by 2002:a05:6830:6188:b0:708:b80d:f3f with SMTP id\n 46e09a7af769-709b996b07fmr6408337a34.4.1722786196177;\n Sun, 04 Aug 2024 08:43:16 -0700 (PDT)","by 2002:a05:6830:6188:b0:708:b80d:f3f with SMTP id\n 46e09a7af769-709b996b07fmr6408323a34.4.1722786195827;\n Sun, 04 Aug 2024 08:43:15 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IHBJ2jfbD9XAW6u0h1bV4NdLix+NkmEFNg/TV8p9dP/irC0EaA7+yNhh+AAaPz8IWJ4UziKyg==","Date":"Sun, 4 Aug 2024 11:43:13 -0400","From":"Peter Xu ","To":"David Hildenbrand ","Cc":"Steve Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini ,\n \"Daniel P. Berrange\" ,\n Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.145,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3356720,"web_url":"http://patchwork.ozlabs.org/comment/3356720/","msgid":"","list_archive_url":null,"date":"2024-08-04T16:10:52","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Sat, Jul 20, 2024 at 05:26:07PM -0400, Steven Sistare wrote:\n> On 7/18/2024 11:56 AM, Peter Xu wrote:\n> > Steve,\n> > \n> > On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n> > > What?\n> > \n> > Thanks for trying out with the cpr-transfer series. I saw that that series\n> > missed most of the cc list here, so I'm attaching the link here:\n> > \n> > https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n> > \n> > I think most of my previous questions for exec() solution still are there,\n> > I'll try to summarize them all in this reply as much as I can.\n> > \n> > > \n> > > This patch series adds the live migration cpr-exec mode, which allows\n> > > the user to update QEMU with minimal guest pause time, by preserving\n> > > guest RAM in place, albeit with new virtual addresses in new QEMU, and\n> > > by preserving device file descriptors.\n> > > \n> > > The new user-visible interfaces are:\n> > > * cpr-exec (MigMode migration parameter)\n> > > * cpr-exec-command (migration parameter)\n> > \n> > I really, really hope we can avoid this..\n> > \n> > It's super cumbersome to pass in a qemu cmdline in a qemu migration\n> > parameter.. if we can do that with generic live migration ways, I hope we\n> > stick with the clean approach.\n> \n> This is no different than live migration, requiring a management agent to\n> launch target qemu with all the arguments use to start source QEMU. Now that\n> same agent will send the arguments via cpr-exec-command.\n\nIt's still a bit different.\n\nThere we append \"-incoming defer\" only, which makes sense because we're\ninstructing a QEMU to take an incoming stream to load. Now we append the\ncomplete qemu cmdline within the QEMU itself, that was booted with exactly\nthe same cmdline.. :-( I would at least start to ask why we need to pass\nthe same thing twice..\n\nNot saying that this is no-go, but really looks unpretty to me from this\npart.. especially if a cleaner solution seems possible.\n\n> \n> > > * anon-alloc (command-line option for -machine)\n> > \n> > Igor questioned this, and I second his opinion.. We can leave the\n> > discussion there for this one.\n> \n> Continued on the other thread.\n> \n> > > The user sets the mode parameter before invoking the migrate command.\n> > > In this mode, the user issues the migrate command to old QEMU, which\n> > > stops the VM and saves state to the migration channels. Old QEMU then\n> > > exec's new QEMU, replacing the original process while retaining its PID.\n> > > The user specifies the command to exec new QEMU in the migration parameter\n> > > cpr-exec-command. The command must pass all old QEMU arguments to new\n> > > QEMU, plus the -incoming option. Execution resumes in new QEMU.\n> > > \n> > > Memory-backend objects must have the share=on attribute, but\n> > > memory-backend-epc is not supported. The VM must be started\n> > > with the '-machine anon-alloc=memfd' option, which allows anonymous\n> > > memory to be transferred in place to the new process.\n> > > \n> > > Why?\n> > > \n> > > This mode has less impact on the guest than any other method of updating\n> > > in place.\n> > \n> > So I wonder whether there's comparison between exec() and transfer mode\n> > that you recently proposed.\n> \n> Not yet, but I will measure it.\n\nThanks.\n\n> \n> > I'm asking because exec() (besides all the rest of things that I dislike on\n> > it in this approach..) should be simply slower, logically, due to the\n> > serialized operation to (1) tearing down the old mm, (2) reload the new\n> > ELF, then (3) runs through the QEMU init process.\n> > \n> > If with a generic migration solution, the dest QEMU can start running (2+3)\n> > concurrently without even need to run (1).\n> > \n> > In this whole process, I doubt (2) could be relatively fast, (3) I donno,\n> > maybe it could be slow but I never measured; Paolo may have good idea as I\n> > know he used to work on qboot.\n> \n> We'll see, but in any case these take < 100 msec, which is a wonderfully short\n\nI doubt whether it keeps <100ms when the VM is large. Note that I think we\nshould cover the case where the user does 4k mapping for a large guest.\n\nSo I agree that 4k mapping over e.g. 1T without hugetlb may not be the\nideal case, but the question is I suspect there're indeed serious users\nusing QEMU like that, and if we have most exactly a parallel solution that\ndoes cover this case, it is definitely preferrable to consider the other\nfrom this POV, simply because there's nothing to lose there..\n\n> pause time unless your customer is doing high speed stock trading. If cpr-transfer\n> is faster still, that's gravy, but cpr-exec is still great.\n> \n> > For (1), I also doubt in your test cases it's fast, but it may not always\n> > be fast. Consider the guest has a huge TBs of shared mem, even if the\n> > memory will be completely shared between src/dst QEMUs, the pgtable won't!\n> > It means if the TBs are mapped in PAGE_SIZE tearing down the src QEMU\n> > pgtable alone can even take time, and that will be accounted in step (1)\n> > and further in exec() request.\n> \n> Yes, there is an O(n) effect here, but it is a fast O(n) when the memory is\n> backed by huge pages. In UEK, we make it faster still by unmapping in parallel\n> with multiple threads. I don't have the data handy but can share after running\n> some experiments. Regardless, this time is negligible for small and medium\n> size guests, which form the majority of instances in a cloud.\n\nPossible. It's just that it sounds like a good idea to avoid having the\ndowntime taking any pgtable tearing down into account here for the old mm,\nirrelevant of how much time it'll take. It's just that I suspect some use\ncase can take fair amount of time.\n\nSo I think this is \"one point less\" for exec() solution, while the issue\ncan be big or small on its own. What matters is IMHO where exec() is\nsuperior so that we'd like to pay for this. I'll try to stop saying \"let's\ntry to avoid using exec() as it sounds risky\", but we still need to compare\nwith solid pros and cons.\n\n> \n> > All these fuss will be avoided if you use a generic live migration model\n> > like cpr-transfer you proposed. That's also cleaner.\n> > \n> > > The pause time is much lower, because devices need not be torn\n> > > down and recreated, DMA does not need to be drained and quiesced, and minimal\n> > > state is copied to new QEMU. Further, there are no constraints on the guest.\n> > > By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,\n> > > and suspending plus resuming vfio devices adds multiple seconds to the\n> > > guest pause time. Lastly, there is no loss of connectivity to the guest,\n> > > because chardev descriptors remain open and connected.\n> > \n> > Again, I raised the question on why this would matter, as after all mgmt\n> > app will need to coop with reconnections due to the fact they'll need to\n> > support a generic live migration, in which case reconnection is a must.\n> > \n> > So far it doesn't sound like a performance critical path, for example, to\n> > do the mgmt reconnects on the ports. So this might be an optimization that\n> > most mgmt apps may not care much?\n> \n> Perhaps. I view the chardev preservation as nice to have, but not essential.\n> It does not appear in this series, other than in docs. It's easy to implement\n> given the CPR foundation. I suggest we continue this discussion when I post\n> the chardev series, so we can focus on the core functionality.\n\nIt's just that it can affect our decision on choosing the way to go.\n\nFor example, do we have someone from Libvirt or any mgmt layer can help\njustify this point?\n\nAs I said, I thought most facilities for reconnection should be ready, but\nI could miss important facts in mgmt layers..\n\n> \n> > > These benefits all derive from the core design principle of this mode,\n> > > which is preserving open descriptors. This approach is very general and\n> > > can be used to support a wide variety of devices that do not have hardware\n> > > support for live migration, including but not limited to: vfio, chardev,\n> > > vhost, vdpa, and iommufd. Some devices need new kernel software interfaces\n> > > to allow a descriptor to be used in a process that did not originally open it.\n> > \n> > Yes, I still think this is a great idea. It just can also be built on top\n> > of something else than exec().\n> > \n> > > \n> > > In a containerized QEMU environment, cpr-exec reuses an existing QEMU\n> > > container and its assigned resources. By contrast, consider a design in\n> > > which a new container is created on the same host as the target of the\n> > > CPR operation. Resources must be reserved for the new container, while\n> > > the old container still reserves resources until the operation completes.\n> > \n> > Note that if we need to share RAM anyway, the resources consumption should\n> > be minimal, as mem should IMHO be the major concern (except CPU, but CPU\n> > isn't a concern in this scenario) in container world and here the shared\n> > guest mem shouldn't be accounted to the dest container. So IMHO it's about\n> > the metadata QEMU/KVM needs to do the hypervisor work, it seems to me, and\n> > that should be relatively small.\n> > \n> > In that case I don't yet see it a huge improvement, if the dest container\n> > is cheap to initiate.\n> \n> It's about reserving memory and CPUs, and transferring those reservations from\n> the old instance to the new, and fiddling with the OS mechanisms that enforce\n> reservations and limits. The devil is in the details, and with the exec model,\n> the management agent can ignore all of that.\n> \n> You don't see it as a huge improvement because you don't need to write the\n> management code. I do!\n\nHeh, possibly true.\n\nCould I ask what management code you're working on? Why that management\ncode doesn't need to already work out these problems with reconnections\n(like pre-CPR ways of live upgrade)?\n\n> \n> Both modes are valid and useful - exec in container, or launch a new container.\n> I have volunteered to implement the cpr-transfer mode for the latter, a mode\n> I do not use. Please don't reward me by dropping the mode I care about :)\n> Both modes can co-exist. The presence of the cpr-exec specific code in qemu\n> will not hinder future live migration development.\n\nI'm trying to remove some of my \"prejudices\" on exec() :). Hopefully that\nproved more or less that I simply wanted to be fair on making a design\ndecision. I don't think I have a strong opinion, but it looks to me not\nideal to merge two solutions if both modes share the use case.\n\nOr if you think both modes should service different purpose, we might\nconsider both, but that needs to be justified - IOW, we shouldn't merge\nanything that will never be used.\n\nThanks,\n\n> \n> > > Avoiding over commitment requires extra work in the management layer.\n> > \n> > So it would be nice to know what needs to be overcommitted here. I confess\n> > I don't know much on containerized VMs, so maybe the page cache can be a\n> > problem even if shared. But I hope we can spell that out. Logically IIUC\n> > memcg shouldn't account those page cache if preallocated, because memcg\n> > accounting should be done at folio allocations, at least, where the page\n> > cache should miss first (so not this case..).\n> > \n> > > This is one reason why a cloud provider may prefer cpr-exec. A second reason\n> > > is that the container may include agents with their own connections to the\n> > > outside world, and such connections remain intact if the container is reused.\n> > > \n> > > How?\n> \n> chardev preservation. The qemu socket chardevs to these agents are preserved,\n> and the agent connections to the outside world do not change, so no one sees\n> any interruption of traffic.\n> \n> > > All memory that is mapped by the guest is preserved in place. Indeed,\n> > > it must be, because it may be the target of DMA requests, which are not\n> > > quiesced during cpr-exec. All such memory must be mmap'able in new QEMU.\n> > > This is easy for named memory-backend objects, as long as they are mapped\n> > > shared, because they are visible in the file system in both old and new QEMU.\n> > > Anonymous memory must be allocated using memfd_create rather than MAP_ANON,\n> > > so the memfd's can be sent to new QEMU. Pages that were locked in memory\n> > > for DMA in old QEMU remain locked in new QEMU, because the descriptor of\n> > > the device that locked them remains open.\n> > > \n> > > cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag,\n> > > and by sending the unique name and value of each descriptor to new QEMU\n> > > via CPR state.\n> > > \n> > > For device descriptors, new QEMU reuses the descriptor when creating the\n> > > device, rather than opening it again. The same holds for chardevs. For\n> > > memfd descriptors, new QEMU mmap's the preserved memfd when a ramblock\n> > > is created.\n> > > \n> > > CPR state cannot be sent over the normal migration channel, because devices\n> > > and backends are created prior to reading the channel, so this mode sends\n> > > CPR state over a second migration channel that is not visible to the user.\n> > > New QEMU reads the second channel prior to creating devices or backends.\n> > \n> > Oh, maybe this is the reason that cpr-transfer will need a separate uri..\n> \n> Indeed.\n> \n> - Steve\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=CL9sHWta;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WcPjF1Y4lz1yY3\n\tfor ; Mon, 5 Aug 2024 02:12:17 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sadpI-0002fr-A8; Sun, 04 Aug 2024 12:11:20 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sadp0-0002eI-Ut\n for qemu-devel@nongnu.org; Sun, 04 Aug 2024 12:11:08 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sadox-0001nI-Uk\n for qemu-devel@nongnu.org; Sun, 04 Aug 2024 12:11:02 -0400","from mail-qv1-f72.google.com (mail-qv1-f72.google.com\n [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-343-Rx2hCwExPZG9muSP3-G6bw-1; Sun, 04 Aug 2024 12:10:56 -0400","by mail-qv1-f72.google.com with SMTP id\n 6a1803df08f44-6b798c6b850so17577116d6.0\n for ; Sun, 04 Aug 2024 09:10:56 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n d75a77b69052e-4518a6ef2f0sm23063221cf.57.2024.08.04.09.10.53\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Sun, 04 Aug 2024 09:10:54 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1722787858;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=o93WBA3jHeImCBM87jdRdnJ1qW/xL/Oa4NWz5waBgJc=;\n b=CL9sHWtarA5L50X9IpmdcynlYosujiM8Sdueyr6nBLtwHv7H9S7gEjf4tyQJBQqUMCFMQx\n UCb7bwhx1I4XHVR4baJ3xQ3gd/q94SXzBrf+fqIPCIR87FJPB9/ghKzeIFxJhavQ+y0WEc\n Di3iFH4QLpJMYeHfHyjxRJzVJxb3vAo=","X-MC-Unique":"Rx2hCwExPZG9muSP3-G6bw-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1722787856; x=1723392656;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=o93WBA3jHeImCBM87jdRdnJ1qW/xL/Oa4NWz5waBgJc=;\n b=Kivjg9lrCmYwZ8CppjPQWoxHFXAIaEH35OvnZs0Z+98Uq0CQQ0jwHNDlEzLwCmQJi6\n zGJAyAl+G8EhBxh7b94vN6+M1m6Zx43N7TfFmgejK2JlROJDLPpZJmFdS6ujiVT2F5Xk\n lVfhguzsU1TLhU6ZEWoPlL2szArLd6jhbX/WtwQs4T5Rb6u25nbU3lvaie1To0bTgqtT\n P4vn6wmcSd4r+NuHyx5b2qoWy9UfyWv1gSsIOoCCBXo30sXg3lhzzWNcw793rORQV1kE\n HUYKgXolEwtoN3QAWqtLLE07dCdeiYIFXH+AP7TRJBwv5WKYz/pcMwZcs23YZbY/Zz9P\n kIkw==","X-Gm-Message-State":"AOJu0YxXSMTfWAxCDVdn0j+Ypy2ZmsIzd3wN17DittHDkwMsbbotJFP/\n WSWokksUaHb6HzuSq3gRa6OB+zMjKq4gOgMolP/2xFXG+I1BV/oGJEvqoutMmY+ij6IeksuIqh7\n CHsLya+S3QXoIj7Mq1yzLQNzZ6xkk1B6J5WmWiY+bhvx07A+XIRur","X-Received":["by 2002:ac8:5a82:0:b0:44f:d899:148c with SMTP id\n d75a77b69052e-4518919e11fmr74408821cf.0.1722787855723;\n Sun, 04 Aug 2024 09:10:55 -0700 (PDT)","by 2002:ac8:5a82:0:b0:44f:d899:148c with SMTP id\n d75a77b69052e-4518919e11fmr74408521cf.0.1722787855194;\n Sun, 04 Aug 2024 09:10:55 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IEWrB6FJeFSTzXfJlvzgPcP1Zw+xuKOB4MO6Qannw9iwwPnExY19/aCQNx8cAQqgmn7BJFtVw==","Date":"Sun, 4 Aug 2024 12:10:52 -0400","From":"Peter Xu ","To":"Steven Sistare ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini ,\n \"Daniel P. Berrange\" ,\n Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.145,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3356994,"web_url":"http://patchwork.ozlabs.org/comment/3356994/","msgid":"<636cc66b-95bf-4003-9e35-9547dfbcbfd1@redhat.com>","list_archive_url":null,"date":"2024-08-05T09:52:35","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":70402,"url":"http://patchwork.ozlabs.org/api/people/70402/","name":"David Hildenbrand","email":"david@redhat.com"},"content":"On 04.08.24 17:43, Peter Xu wrote:\n> On Mon, Jul 22, 2024 at 10:59:47AM +0200, David Hildenbrand wrote:\n>>> So I wonder whether there's comparison between exec() and transfer mode\n>>> that you recently proposed.\n>>>\n>>> I'm asking because exec() (besides all the rest of things that I dislike on\n>>> it in this approach..) should be simply slower, logically, due to the\n>>> serialized operation to (1) tearing down the old mm, (2) reload the new\n>>> ELF, then (3) runs through the QEMU init process.\n>>>\n>>> If with a generic migration solution, the dest QEMU can start running (2+3)\n>>> concurrently without even need to run (1).\n>>\n>> I'll note (not sure if already discussed) that with the \"async-teardown\"\n>> option we have a way to move the MM teardown to a separate process, such\n>> that it will happen asynchronously.\n> \n> I just had a look, maybe it won't trivially work, as it relies on QEMU\n> process to quit first..\n> \n> async_teardown_fn():\n> if (the_ppid == getppid()) {\n> pause();\n> }\n> \n> While if we stick with exec(), then PID shouldn't change, so the teardown\n> process can hold the mm and pause until the VM is destroyed..\n\nRight, the mechanism would have to be extended to realize that exec() \nhappened. Notifying the child before exec() would be undesired, so it \nwould have to happen after exec() from the changed parent.\n\nSounds doable, but certainly doesn't come for free!","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=LgnDgjTc;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WcsFn2lQkz1yYD\n\tfor ; Mon, 5 Aug 2024 19:53:33 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sauOU-0006La-Jz; Mon, 05 Aug 2024 05:52:46 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sauOT-0006L4-6I\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 05:52:45 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sauOR-0008LR-Mr\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 05:52:44 -0400","from mail-wr1-f72.google.com (mail-wr1-f72.google.com\n [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-674-3N79N5o4OpSSeRPZH3KrKA-1; Mon, 05 Aug 2024 05:52:39 -0400","by mail-wr1-f72.google.com with SMTP id\n ffacd0b85a97d-36873a449dfso2605167f8f.0\n for ; Mon, 05 Aug 2024 02:52:39 -0700 (PDT)","from ?IPV6:2a09:80c0:192:0:5dac:bf3d:c41:c3e7?\n ([2a09:80c0:192:0:5dac:bf3d:c41:c3e7])\n by smtp.gmail.com with ESMTPSA id\n ffacd0b85a97d-36bbd0597f6sm9332197f8f.66.2024.08.05.02.52.36\n (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n Mon, 05 Aug 2024 02:52:36 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1722851561;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references:autocrypt:autocrypt;\n bh=uGCGO/DB+EfmOabKIkXEOknhCnwtduq7gANaKCS5tNc=;\n b=LgnDgjTcUm2gPuVy+tSW+VFa8icWonu1BORk3wwDjj5C6MJJstOak6D8rqKkph5f8zuuwx\n 2BFA72UajGAEJKXvvJwdgsfgigD0yg6HbsLuorFFP19fsZsiYVjlQP06oI3LAf252QDoa/\n x5YgvIQPTRLbqhUzoXY2TGEEtTaj/+E=","X-MC-Unique":"3N79N5o4OpSSeRPZH3KrKA-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1722851558; x=1723456358;\n h=content-transfer-encoding:in-reply-to:organization:autocrypt\n :content-language:from:references:cc:to:subject:user-agent\n :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject\n :date:message-id:reply-to;\n bh=uGCGO/DB+EfmOabKIkXEOknhCnwtduq7gANaKCS5tNc=;\n b=Zo6v9szl4+Z9GMYO4B5VrktcgGqxq2EdB3HUjxi69Ypr4IZ4lTIN7KLooG0HX8pJGN\n Q2qjRHhzmI7dYL2jopa195C8IJNywBgvXuwemICG3/6EQcbnI1AgKUyOk6jaOEKXBUnj\n ILXriDP9HLTT3ntUnNkY6uwlYZCEzrSWUTFx+9/U72oWuzHbGvfg68T++CfiUDD5E5Iy\n gt9KNwcvPnvQX3/agNi+wRSoRF1OGxR5D95VFReLttWC7OnDBELj0Aue/xL6Jvh33hZe\n WRcr7MnudFWgPmvA5fhZjeHWebq3ImYMuGaHvqpwndvMIVdGpoX8ktQZ5XxOYafNyvsR\n rggg==","X-Forwarded-Encrypted":"i=1;\n AJvYcCU5KXoHLiMRz3knZI392VSzkNbPNRYOJGy5taadXXPizQdBuR81CP4yWbjlRzyByaPRrhNQj8LVYNgM0bUEMtTjD5wJ83E=","X-Gm-Message-State":"AOJu0YzQF2GxpunVeOwSB2sWoJXLBY4zUnSt0cfBhJ18TLsh4e8A+fqY\n TnnA1kq1AzemAqhcV3QL85dBtfie6yTozUMJ82oFPLtQ/iagR9N3GoBfWocg3mzYVvHGqIsYm03\n jo4hcaqGquUeKgaWIMHmm1y8HmowsKO3AgnYNqZY2xd9AjD61o1mW","X-Received":["by 2002:adf:f60b:0:b0:36b:aa96:d1e5 with SMTP id\n ffacd0b85a97d-36bbbe5b841mr9400995f8f.18.1722851557883;\n Mon, 05 Aug 2024 02:52:37 -0700 (PDT)","by 2002:adf:f60b:0:b0:36b:aa96:d1e5 with SMTP id\n ffacd0b85a97d-36bbbe5b841mr9400970f8f.18.1722851557365;\n Mon, 05 Aug 2024 02:52:37 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IFquxaKtlJa/tG0mqIEQQ4SgliO4vtb04PJiLlWvPdSZ6UpWj2TeCjkVlDC1dNY8HIsiYmCxw==","Message-ID":"<636cc66b-95bf-4003-9e35-9547dfbcbfd1@redhat.com>","Date":"Mon, 5 Aug 2024 11:52:35 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ","Cc":"Steve Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>\n ","From":"David Hildenbrand ","Content-Language":"en-US","Autocrypt":"addr=david@redhat.com; keydata=\n xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ\n dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL\n QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp\n XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK\n Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9\n PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt\n WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc\n UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv\n jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb\n B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk\n ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW\n AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q\n 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp\n rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf\n wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4\n 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l\n pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd\n KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE\n BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs\n 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF\n 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9\n M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz\n Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb\n T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A\n 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk\n CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G\n NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75\n 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx\n 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS\n lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv\n AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa\n N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3\n AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB\n boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq\n 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f\n XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ\n a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq\n Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6\n 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8\n kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E\n th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr\n jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt\n WNyWQQ==","Organization":"Red Hat","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=david@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.145,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3356995,"web_url":"http://patchwork.ozlabs.org/comment/3356995/","msgid":"","list_archive_url":null,"date":"2024-08-05T10:01:20","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Thu, Jul 18, 2024 at 11:56:33AM -0400, Peter Xu wrote:\n> Steve,\n> \n> On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n> > What?\n> \n> Thanks for trying out with the cpr-transfer series. I saw that that series\n> missed most of the cc list here, so I'm attaching the link here:\n> \n> https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n> \n> I think most of my previous questions for exec() solution still are there,\n> I'll try to summarize them all in this reply as much as I can.\n> \n> > \n> > This patch series adds the live migration cpr-exec mode, which allows\n> > the user to update QEMU with minimal guest pause time, by preserving\n> > guest RAM in place, albeit with new virtual addresses in new QEMU, and\n> > by preserving device file descriptors.\n> > \n> > The new user-visible interfaces are:\n> > * cpr-exec (MigMode migration parameter)\n> > * cpr-exec-command (migration parameter)\n> \n> I really, really hope we can avoid this..\n> \n> It's super cumbersome to pass in a qemu cmdline in a qemu migration\n> parameter.. if we can do that with generic live migration ways, I hope we\n> stick with the clean approach.\n\nA further issue I have is that it presumes the QEMU configuration is\nfully captured by the command line. We have a long term design goal\nin QEMU to get away from specifying configuration on the command\nline, and move entirely to configuring QEMU via a series of QMP\ncommands.\n\nThis proposed command is introducing the concept of command line argv\nas a formal part of the QEMU API and IMHO that is undesirable. Even\ntoday we have backend configuration steps only done via QMP, and I'm\nwondering how it would fit in with how mgmt apps currently doing live\nmigration.\n\nThe flipside, however, is that localhost migration via 2 separate QEMU\nprocesses has issues where both QEMUs want to be opening the very same\nfile, and only 1 of them can ever have them open.\n\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=WDWuzT14;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WcsRW1sj4z1yZb\n\tfor ; Mon, 5 Aug 2024 20:01:59 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sauX4-0005SG-O4; Mon, 05 Aug 2024 06:01:39 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sauX3-0005D3-9V\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 06:01:37 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sauX1-00026l-8Y\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 06:01:36 -0400","from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-634-AHo1aTqeMzSbUXPWusr3lg-1; Mon,\n 05 Aug 2024 06:01:29 -0400","from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id AD6191955D4E; Mon, 5 Aug 2024 10:01:27 +0000 (UTC)","from redhat.com (unknown [10.42.28.36])\n by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id C9F6219560AE; Mon, 5 Aug 2024 10:01:23 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1722852092;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:in-reply-to:in-reply-to: references:references;\n bh=ztgT8ra/T2koBE+RWMqzRZZj3ngMYY96BS9nQ4dkgLw=;\n b=WDWuzT146Eh/l5YLJBOp2CCjsx6J1nVV5RH45H4E8/7kJTI9GyQ1LXN7bxgXjlEpA8j5zN\n UUztiPOzw2uaAwCqDT/vZajaSEr5bUOy9AvCWFHmF/+nfiJ6Z6iUwQEXomGULrI+RuTWoZ\n gEf0p819XnbSuwanT6sAXmsz7JL3wVw=","X-MC-Unique":"AHo1aTqeMzSbUXPWusr3lg-1","Date":"Mon, 5 Aug 2024 11:01:20 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Peter Xu ","Cc":"Steve Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.12","Received-SPF":"pass client-ip=170.10.133.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.145,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3357001,"web_url":"http://patchwork.ozlabs.org/comment/3357001/","msgid":"<44a34534-993d-4352-966f-c0c2fd4008d7@redhat.com>","list_archive_url":null,"date":"2024-08-05T10:06:42","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":70402,"url":"http://patchwork.ozlabs.org/api/people/70402/","name":"David Hildenbrand","email":"david@redhat.com"},"content":"On 05.08.24 11:52, David Hildenbrand wrote:\n> On 04.08.24 17:43, Peter Xu wrote:\n>> On Mon, Jul 22, 2024 at 10:59:47AM +0200, David Hildenbrand wrote:\n>>>> So I wonder whether there's comparison between exec() and transfer mode\n>>>> that you recently proposed.\n>>>>\n>>>> I'm asking because exec() (besides all the rest of things that I dislike on\n>>>> it in this approach..) should be simply slower, logically, due to the\n>>>> serialized operation to (1) tearing down the old mm, (2) reload the new\n>>>> ELF, then (3) runs through the QEMU init process.\n>>>>\n>>>> If with a generic migration solution, the dest QEMU can start running (2+3)\n>>>> concurrently without even need to run (1).\n>>>\n>>> I'll note (not sure if already discussed) that with the \"async-teardown\"\n>>> option we have a way to move the MM teardown to a separate process, such\n>>> that it will happen asynchronously.\n>>\n>> I just had a look, maybe it won't trivially work, as it relies on QEMU\n>> process to quit first..\n>>\n>> async_teardown_fn():\n>> if (the_ppid == getppid()) {\n>> pause();\n>> }\n>>\n>> While if we stick with exec(), then PID shouldn't change, so the teardown\n>> process can hold the mm and pause until the VM is destroyed..\n> \n> Right, the mechanism would have to be extended to realize that exec()\n> happened. Notifying the child before exec() would be undesired, so it\n> would have to happen after exec() from the changed parent.\n> \n> Sounds doable, but certainly doesn't come for free!\n\nI did not look deeply into this, but possibly using a pipe between both \nprocesses created with O_CLOEXEC might do. Anyhow, something to look \ninto if really required :)","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=RcasI8Zo;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WcsYf1Z02z1yYD\n\tfor ; Mon, 5 Aug 2024 20:07:18 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sauc7-0003HR-W0; Mon, 05 Aug 2024 06:06:52 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sauc6-0003Bs-F4\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 06:06:50 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sauc4-0002qU-M6\n for qemu-devel@nongnu.org; Mon, 05 Aug 2024 06:06:50 -0400","from mail-wm1-f71.google.com (mail-wm1-f71.google.com\n [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-259-ROaXQo49PBO2eu7L5uShTQ-1; Mon, 05 Aug 2024 06:06:45 -0400","by mail-wm1-f71.google.com with SMTP id\n 5b1f17b1804b1-42816096cb8so99397045e9.0\n for ; Mon, 05 Aug 2024 03:06:45 -0700 (PDT)","from ?IPV6:2a09:80c0:192:0:5dac:bf3d:c41:c3e7?\n ([2a09:80c0:192:0:5dac:bf3d:c41:c3e7])\n by smtp.gmail.com with ESMTPSA id\n 5b1f17b1804b1-4282bb6403bsm194483035e9.35.2024.08.05.03.06.43\n (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);\n Mon, 05 Aug 2024 03:06:43 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1722852407;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references:autocrypt:autocrypt;\n bh=qqqarilDeosWdla69lCnBN0D4oBX8B2KiMpKAsxfoyg=;\n b=RcasI8ZokZJyx1tOzRMqHt+ihhuKDF3Ghy5nuNcdMqy/rC6AVTDoGuMF8af6MYAORsre/8\n mBUIyllx29jLmM7YiS+V6dUBwchBQ8pM5z7WBmCLlEr6iUa4ntfZRlnfbqKb+n3u34iUbw\n bJHyXGikQiNaWBIGIZ5/mM9AoSKM4Xg=","X-MC-Unique":"ROaXQo49PBO2eu7L5uShTQ-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1722852404; x=1723457204;\n h=content-transfer-encoding:in-reply-to:organization:autocrypt\n :content-language:references:cc:to:from:subject:user-agent\n :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject\n :date:message-id:reply-to;\n bh=qqqarilDeosWdla69lCnBN0D4oBX8B2KiMpKAsxfoyg=;\n b=SXgx6FxDK0r/jx1BoWWwfWSas1ypOMDlpzwE3hO+ncfVu6Bs4HaJ7APaq4rMbwYjgD\n AlwPUwqvdwrvvSTweXpAaOd/gyyviGoAURYgGb9QNrnhmtlbWbLzGGJjzYJVrSEb9Qfc\n f14DjeCh14TMDOW27cHqKVxmT8z9IEh41TmBlB5ICr5PM/bP3PXLGMagzapXPxapwG9x\n rUF2VFbERaRQDo0UelsgSrmuTQMybnqFftYCXNGlotmkQvUpS3ndk02rmDdqwbz5zo5B\n wvRtagouKj/PU5XR35kFlPfazs0bfb7swLC8gKpRs1VBo1NUjhU9UQi4aOTTER/AtQV7\n Nexg==","X-Forwarded-Encrypted":"i=1;\n AJvYcCVe+1UpPXxqehcCIsLmpoXN8jH6FAO/C9ozOaRcTd0j3TBFUevsR8/Ht5up4rPMt3/oaupxRf+8Hdg3/1H3jdtr50CvpPc=","X-Gm-Message-State":"AOJu0Yzp943IN5PvmtRR7jbNzzgHR/WgH0ufW046JrzTOxSHgCk8vKXN\n jw9oJD2mGH1Bm8UwP7c7herndbVyReTBYMBroumMxFWJ8z6liw3H+7wzG07ZuWuaIrjQwu3lDk9\n 2dEf2OGO4axmdjrfof/Ux5Y9YZ0rruVQAic7HhK6CEqG8wP0vfKMV","X-Received":["by 2002:a05:600c:314c:b0:426:6857:3156 with SMTP id\n 5b1f17b1804b1-428e6b7e949mr97055225e9.27.1722852404523;\n Mon, 05 Aug 2024 03:06:44 -0700 (PDT)","by 2002:a05:600c:314c:b0:426:6857:3156 with SMTP id\n 5b1f17b1804b1-428e6b7e949mr97054925e9.27.1722852404125;\n Mon, 05 Aug 2024 03:06:44 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IFvBlqAg1optDQWZ4Brpnj59DgvSI0hZxFN9HNKO6x8jKResqApP/TNWLPe52tcXJg3WotMKA==","Message-ID":"<44a34534-993d-4352-966f-c0c2fd4008d7@redhat.com>","Date":"Mon, 5 Aug 2024 12:06:42 +0200","MIME-Version":"1.0","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","From":"David Hildenbrand ","To":"Peter Xu ","Cc":"Steve Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <37d9e725-1be7-48e1-b621-3f657192f124@redhat.com>\n <636cc66b-95bf-4003-9e35-9547dfbcbfd1@redhat.com>","Content-Language":"en-US","Autocrypt":"addr=david@redhat.com; keydata=\n xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ\n dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL\n QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp\n XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK\n Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9\n PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt\n WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc\n UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv\n jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb\n B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk\n ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW\n AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q\n 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp\n rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf\n wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4\n 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l\n pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd\n KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE\n BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs\n 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF\n 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9\n M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz\n Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb\n T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A\n 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk\n CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G\n NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75\n 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx\n 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS\n lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv\n AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa\n N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3\n AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB\n boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq\n 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f\n XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ\n a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq\n Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6\n 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8\n kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E\n th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr\n jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt\n WNyWQQ==","Organization":"Red Hat","In-Reply-To":"<636cc66b-95bf-4003-9e35-9547dfbcbfd1@redhat.com>","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=david@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-22","X-Spam_score":"-2.3","X-Spam_bar":"--","X-Spam_report":"(-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.145,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3358091,"web_url":"http://patchwork.ozlabs.org/comment/3358091/","msgid":"<5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>","list_archive_url":null,"date":"2024-08-06T20:56:18","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/5/2024 6:01 AM, Daniel P. Berrangé wrote:\n> On Thu, Jul 18, 2024 at 11:56:33AM -0400, Peter Xu wrote:\n>> Steve,\n>>\n>> On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n>>> What?\n>>\n>> Thanks for trying out with the cpr-transfer series. I saw that that series\n>> missed most of the cc list here, so I'm attaching the link here:\n>>\n>> https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n>>\n>> I think most of my previous questions for exec() solution still are there,\n>> I'll try to summarize them all in this reply as much as I can.\n>>\n>>>\n>>> This patch series adds the live migration cpr-exec mode, which allows\n>>> the user to update QEMU with minimal guest pause time, by preserving\n>>> guest RAM in place, albeit with new virtual addresses in new QEMU, and\n>>> by preserving device file descriptors.\n>>>\n>>> The new user-visible interfaces are:\n>>> * cpr-exec (MigMode migration parameter)\n>>> * cpr-exec-command (migration parameter)\n>>\n>> I really, really hope we can avoid this..\n>>\n>> It's super cumbersome to pass in a qemu cmdline in a qemu migration\n>> parameter.. if we can do that with generic live migration ways, I hope we\n>> stick with the clean approach.\n> \n> A further issue I have is that it presumes the QEMU configuration is\n> fully captured by the command line. We have a long term design goal\n> in QEMU to get away from specifying configuration on the command\n> line, and move entirely to configuring QEMU via a series of QMP\n> commands.\n> \n> This proposed command is introducing the concept of command line argv\n> as a formal part of the QEMU API and IMHO that is undesirable. \n\nActually cpr-exec-command does not presume anything; it is an arbitrary\ncommand with arbitrary arguments. If in the future QEMU takes no command-line\narguments, then mgmt will pass a simple launcher command as cpr-exec-command,\nand the launcher will issue QMP commands. Or the launcher will send a message\nto another mgmt agent to do so. It is very flexible. Regardless, the API\ndefinition of cpr-exec-command will not change.\n\nAs another example, in our cloud environment, when the mgmt agent starts QEMU,\nit saves the QEMU args in a file. My cpr-exec-command is just \"/bin/qemu-exec\"\nwith a few simple arguments. That command reads QEMU args from the file and\nexec's new QEMU.\n\n> Even\n> today we have backend configuration steps only done via QMP, and I'm\n> wondering how it would fit in with how mgmt apps currently doing live\n> migration.\n\nSure, and that still works. For live migration, mgmt starts new QEMU with\nits static arguments plus -S plus -incoming, then mgmt detects QEMU has reached\nthe prelaunch state, then it issues QMP commands. For live update, cpr-exec-command\nhas the static arguments plus -S, then mgmt detects QEMU has reached the prelaunch\nstate, then it issues QMP commands.\n\n> The flipside, however, is that localhost migration via 2 separate QEMU\n> processes has issues where both QEMUs want to be opening the very same\n> file, and only 1 of them can ever have them open.\n\nIndeed, and \"files\" includes unix domain sockets. Network ports also conflict.\ncpr-exec avoids such problems, and is one of the advantages of the method that\nI forgot to promote.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=XyUyY8DA;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=cx9GPu+8;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wdlws0KSGz1yfC\n\tfor ; Wed, 7 Aug 2024 06:57:01 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sbRET-0001tA-Oo; Tue, 06 Aug 2024 16:56:37 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sbRES-0001sc-Lb\n for qemu-devel@nongnu.org; Tue, 06 Aug 2024 16:56:36 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sbREQ-0001f1-MQ\n for qemu-devel@nongnu.org; Tue, 06 Aug 2024 16:56:36 -0400","from pps.filterd (m0246629.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 476Em5Vf030468;\n Tue, 6 Aug 2024 20:56:30 GMT","from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta02.appoci.oracle.com [147.154.18.20])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40sce966mg-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Tue, 06 Aug 2024 20:56:29 +0000 (GMT)","from pps.filterd\n (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 476JPZqT017007; Tue, 6 Aug 2024 20:56:28 GMT","from nam10-mw2-obe.outbound.protection.outlook.com\n (mail-mw2nam10lp2049.outbound.protection.outlook.com [104.47.55.49])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id\n 40sb0agbks-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Tue, 06 Aug 2024 20:56:28 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by CY5PR10MB5964.namprd10.prod.outlook.com (2603:10b6:930:2b::8) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7828.27; Tue, 6 Aug\n 2024 20:56:25 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7828.023; Tue, 6 Aug 2024\n 20:56:25 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=3Oq3CoLEdcfEI2KIzWMXKyIWtzRQcGJGcp9X2COykt0=; b=\n XyUyY8DA0GD1FyUoEV03UEdJxAJgsRDBJXIdOpME5e2AXfumd3anXiZUH1dUwg7e\n U5Nv//FaYvEsTF23ULWJ4f3zHQtDFygHjsSX57piPuGvi9uCLBYZq1PNVyMmI9mh\n V8ic//jEWqtOekEhy/xQK+p+iYwXs2YQNGFhBZaz6G+SGuVwLzO+L2TKb25OsGJi\n QwRpXZ9ixWmd/8GbHRblYi6de24q5CeiR8KO+EUVowX/3QHRQiKF1XYW8ZH+1h8g\n L+vCXNzEk7BBqvTZvq70UN+RGLOG6qR0vZxv53zEyhIzjEKoFhIntpOkw+jtgimT\n Ni+M29TISoB+T03Cq2wecA==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=3Oq3CoLEdcfEI2KIzWMXKyIWtzRQcGJGcp9X2COykt0=;\n b=cx9GPu+8awu5uetbJID+qcBGBBjKmTHERU6ypLaC7AhNVvx5Fj7dyu4EF4GfkEv5KreiZxMcUEXOlON4sKOZsYhb+GATfzDhKexS4piQn+vK1QG6ZpOQ8FBehlsEGPNScWzzCsIzT3q6JoeV5z+d6Q0jtKi6rD+zA6Y8lqU6FY8="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=EkgTCwOV9cB4B5o8cE1WNSK1QWpUa4XY7d2l3L/mYMbKncNuqfpZvl/Obxy7CQs+0JO2n8ieOpAkSNhrRoLP23n9u7sn63Bi9EEeo8G3ADrJH+6DZrlao55gPgOpg/AcuULW7P4TC+9bYKwqHdvtyQV6WQejsPPV6ekseOt2l+4HUfkl4eXuhNif2qLnbwaD2CIlxavHy/171pKxi63pt24INsB84A/InREHe5nEFi09MDRcr2eKTmu2Pjg7s14Y6mm2/y0C39AD70WyA6niaBTQzZlwwMp9vt/4tzJmMhcc6nFidG6KmW093/f+x6/jkDU7GLIGCWKPwpKKJRnV/w==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=3Oq3CoLEdcfEI2KIzWMXKyIWtzRQcGJGcp9X2COykt0=;\n b=otBG6YxZjg0LcHafiNc2ddek4gsP4i758bhCYzkMLpni9uEICqkgwDQ0586Szxc+Ixf12HGAtvbqo0zk833hU1MQlUF9GTXm9DyLJ46/FTsgBRxKJZ29M/PoajZOpfaDHqr9nZoANP6htRU+TvlUxbgPmOY2a8s92OfwqeVWAiba9EfYA7wn0pAtFu6SQC4rlaSaZ2WexWVXHLBX70nMflRR/JxUcz+uEE5u/s4l9p3Zjjm9xv8uwoypU+ZZ4DJCD4kJH65yk3TJUH7DWsQMMFbFApFMKEfco/I1p+7GViSE8eUiOwm3DQPw+552br6qCC3xefH9+1sV3cPPg59ihA==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"<5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>","Date":"Tue, 6 Aug 2024 16:56:18 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"=?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= ,\n Peter Xu ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-ClientProxiedBy":"SJ0P220CA0014.NAMP220.PROD.OUTLOOK.COM\n (2603:10b6:a03:41b::22) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|CY5PR10MB5964:EE_","X-MS-Office365-Filtering-Correlation-Id":"0e19d5c6-b5ed-4d1f-69f6-08dcb65a3c45","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|366016|7416014|376014|1800799024;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?KjU/nBZIBiALshEr5Cn4TJnQPiB48rt?=\n\t=?utf-8?q?R1ic3se2AokfoM7uRNQ9YgS35USIEmufbbC4GmPXOPFuDXlgDyM6kyJYpVaFRVxTw?=\n\t=?utf-8?q?NNQkA1FbIiM60UYccVtAGFYYIBpgqfnZ5qfN9IyC5pQ/R2agr0ZYZrqQMTajbpdov?=\n\t=?utf-8?q?Ic/uGlmZtnZtetJBsNu0FsK/ZZZUJmyD4WAX6EyrJmHMPVBBCwJEYJCc0hkTUFHG3?=\n\t=?utf-8?q?9NLLyjpFD817pmk+sywIuVpzryj3zJCGUUpGJvwUP90r9dmmisdUVhqyE2cLKrZeC?=\n\t=?utf-8?q?GXTnP9t2Y/CTy4ZzPf1wabFLH670whPZrUSNvNgMq+1qC/jHkJSqTkKIc7xW4XNIP?=\n\t=?utf-8?q?J+QmvgMMQfQKKq5++9/KNiHsaS681MgrHTIjA2YfP4DlcOb6L7R6XvmIeJDt9Lu7x?=\n\t=?utf-8?q?FJ2dGCbJpKDphQJQqtWeEtkiIK7xY/JEz2ephcgSqfuEQerFfgpIn4uqyvc4zoDaR?=\n\t=?utf-8?q?faP4mub1RgnrsFPrMAoEf5MT1TuejvsYSf0CbQ9+ukReZaPDAkSmQlqr6N6TBRR5r?=\n\t=?utf-8?q?hqFhHlKGlTUY5FXULk4jcYIqU84MuMO3qPsk3E914JpS71XYnoxAVp4mjNOjC6iRM?=\n\t=?utf-8?q?pHm9ZE5sxLd0KCuS+EanEcES28lacRl6QJWXIoThHAWAPOVVxYDQBN/jM6VKDI89l?=\n\t=?utf-8?q?+sDJb3IXAT7hX7gZ1jhFkK7/J5XnoZf/p7ecUdc3f8fWX2GOPP9B2ovOwZ1+/uKTy?=\n\t=?utf-8?q?fEFlVV8/aRJWWKtULm24OR2u4uCBXtRajDEju4+Zh7YZtHZuPoTPKO+oKVTuNCRvk?=\n\t=?utf-8?q?biM8bXVtZyLy/nh5/IpewiHOpa4lXVIW1kFNeu6zMX9YIFIA+WiJdrh6i1rZjxqJZ?=\n\t=?utf-8?q?Nq1XnAHRs3YHFnH+D7X+1/M8iUBuhzh0mGKZSrbbm6rFdglHj/nAFWD4JhH+gDfar?=\n\t=?utf-8?q?E95PKdiyIKG+nnQaNSa41es1Z3KUNp6zZ1ShC6Fbwv7JXYO6i0RBgJKKxUc3fjham?=\n\t=?utf-8?q?il8OtUUPYwGKBqLWQypdCYLxWseDMS7xw4g4ffrGEscUIpbZKYD4CiIjuMv1nqJtC?=\n\t=?utf-8?q?DNM/g8AaQT+WOfjXiKdGp1KPBbkYlioJ+Z6SFRTMyPbqWOZ0lvqpqi0d4C2K5/UwV?=\n\t=?utf-8?q?Ty/57B5Wh/Udq1/3Tuq7FNOP4Ro8MM4dPPC9feaETsKz1jmiDdiJdGUgPikTk9SZf?=\n\t=?utf-8?q?DsNia1wd6YiEmOe7BJHAlrq40ocndzvKGjkgYB6fB1pk5e9svX9vCNvL5qYmibKTn?=\n\t=?utf-8?q?BkFmiwffDkea6eLAst6Wc9sTHO5SsrbBZMwsRPcqkGSnPSOhNyIRZma7M=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(7416014)(376014)(1800799024); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?DQ8a6Q2lxM5g/eT60nLwQwdzybMq?=\n\t=?utf-8?q?tLG08hxBoBieu+mk216P61pDkcIeDmkK124/kiCFa0ulIjAWHDIqUguAMtXt8nUaz?=\n\t=?utf-8?q?4LaamFVmLo9OnXBw0RY2ccIXJnR78jGDzt7aqFJ8FCG77TfmQlJ6p/ecdsovXWn+b?=\n\t=?utf-8?q?TJmag7pVjXfnFcC6n2zgQ/VHnJwY1v1MuQ8HnuFLG+V65ivWEHnbwph9hkWd3TK0Q?=\n\t=?utf-8?q?PpBpxMZ9Gm2O837uY75whELA2wenaOuL/eLc+w8tkkmwyig8tn43cHx6u6e1cXoJe?=\n\t=?utf-8?q?nAMbVlv/a6TUx+PAguPfHwCTZmKHxlvwafXaNylMf3YrqzCPRj0KdS6y7tIVsgXG9?=\n\t=?utf-8?q?npukLX+ujf0SQ8oaQBcI4qk8QTYAUYYjJFr85Va3opi7R3TKXUPhE1qNqLiD6fY6G?=\n\t=?utf-8?q?9SeR5qMCf8CUI7ZrGWY4//GxJ4uKQgklvyX4l0BQapG55HoBMtMjOYWj0zk7yeLVe?=\n\t=?utf-8?q?SQ0+CjbJ4JuxB2LsCphAcbvHuqfqK8TuieTQoKMdZh5ePTvmPICcHuOWTv0mNGSps?=\n\t=?utf-8?q?eNacX1baCrX+lrp4eBPhrexrqa7izIWMmwnvVKbXVweqnrVd4D/C8s0dbnMjYPymG?=\n\t=?utf-8?q?Z3lceh7U99wbHc8OlRtBPsLMF7uYbVuSN4hYGPDy+iFNQ/a5eYF8u+HNx30HL88p7?=\n\t=?utf-8?q?+Irdj+JjuVgnoAD4mE3sw7EqwxiVn7JTHvMZD2mTZzUiJYdknYZRob7y9ybDtz2ik?=\n\t=?utf-8?q?t3poo+T+vJMTsL0IvQkRQeDjhH3ATf+IygJyVD90MDCL5E28FxHVrds83gdKvzDOj?=\n\t=?utf-8?q?h246NCQopfvDUDBYM/BGN83JTyxirnnxBMtlXX3L8FSKEutIntHhoBWTtJbgudxql?=\n\t=?utf-8?q?cjsOKz+rqMzyEuK9qY9hNsshEwrkeqcdRZk9RhMg8QgKXQ4D/SnGfitgDoiLm7WSZ?=\n\t=?utf-8?q?2RX/BKibra05ovOkXJHg0yC8IOX+E1ZDNl0/0DR9OBiUrY0d86GwC/QU4Z0tDEmmQ?=\n\t=?utf-8?q?f3+h2wFPZM41edL9ufhAx4svpe4ocG6djIRHdi9dNkPnquChq/INtWRGY68JlAafJ?=\n\t=?utf-8?q?2j1+ytp108I7gEN0f880l5jDoa/nr+xTJLzEWKX9tOPKrj8OjF4G9Ds6M+aOUjSt2?=\n\t=?utf-8?q?CZfMUNPtfanJhvX6DW6Wh8KdRdTTA6xzYkHwhAZYFU42lw373xgOUQKC3ATrlWmaU?=\n\t=?utf-8?q?VyMeFca1cwxMlGLYvoRn4HzYykQpxssIBZv0785SYBBF3seY4dcAQi/HHYY9yGRML?=\n\t=?utf-8?q?MX5VF7ywRxbUbZMyGWfsHK6yMSImtIWTk5gieFfRCrhXiqtmk7xdHO9KJ2y4Xq5hA?=\n\t=?utf-8?q?DDiN6/RIUZAYB8fXbxOHoxqV+XHBg9n/9pE6Jk5h/hFu66vsbTK//+qXCahBADRLW?=\n\t=?utf-8?q?UOPuN0nyohLFB2QmC8uDPg3WKTygxT3wLrsFcR32TF3NowKPBVY03GLgKrrdL5pcw?=\n\t=?utf-8?q?T65ay6ICWGRm93cMkcOVxa0J0PiCY9S0dVuGK2ISk9AxToce1O35/u8AQCjg86d7O?=\n\t=?utf-8?q?3EETgtwuqs6r1SLH765MfZFAu9auNaZP+UA5xSKrKm6opVDOXf/ZQXBDcOiY0UGql?=\n\t=?utf-8?q?GKsJ7QTaCU9hvACskM2MijMfNRQJAsmqZg=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n 8toHB+yPdoPnrH5uhcRgh8yoU1bUj2d5URSv8zoKDmQI3TKLTaPNdyE5s5vR3ucswv1g2bCgOiDX3BcdbWdKAELUE9AQxier08b2ecNsgH3iaW8k83YyoukByN1po9/7IRRu3LITbSHnzgaZyR1dyzS+pUNKwNMvbSUROKXK8pjF4IuoMeKSyMXxyrwhDXtI34CAFeaoqTxParKkfswSUaDTzqpI0LHIhLS/PWjPTqLfzOBdol3OA71wXHpGUtnbH8PFBJC3LRAOh9h++8uL8iSlo2ZxgnFz8aN7q5RF+5LGhux1ZHXxNBvFjzta7yrjEy3PzdidluG3Sez9uplI4REvLDanmlk82SSU1YfwNO2hE4IYvIolE3Yva/agFKe5IR6v1bPzuWLMRB43gDZO52uOzZ54a7BHJyzI0qMtQQhHXWr0OG9NamAxMMM5VcYaBd5gQn84pJQRc+f8Kjt9ggU+rGC7amTjtQvPNIAtuEHM4szp76a9hJQ9lfBvnAYJgHosg4ZN9CWJbCPgVukzSRLKCx/lj12xMahuBItpOQ0+WzrJ92rHhTAhACxu+Gdx9XpyvW53wCfZi5KMYjQWzGZPcOYM7pEnmKD4fiwg88c=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 0e19d5c6-b5ed-4d1f-69f6-08dcb65a3c45","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"06 Aug 2024 20:56:25.5675 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n wIkRhhPFJxnC+K5ZWhb/owTRRzwAvvLtTetCXST25Vsh765eyob9F0ui2WmyeEkEyFmv6l8iRr0hVGIZKmLrc+PsuIYRAA7fX9wd5oUIyEA=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CY5PR10MB5964","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-06_17,2024-08-06_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 phishscore=0\n suspectscore=0\n adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 mlxscore=0\n spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1\n engine=8.12.0-2407110000 definitions=main-2408060146","X-Proofpoint-GUID":"NeC4wUqpmC5xqukK-yI4RztF9CEf9-3Q","X-Proofpoint-ORIG-GUID":"NeC4wUqpmC5xqukK-yI4RztF9CEf9-3Q","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001,\n RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3358834,"web_url":"http://patchwork.ozlabs.org/comment/3358834/","msgid":"","list_archive_url":null,"date":"2024-08-07T19:47:47","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/4/2024 12:10 PM, Peter Xu wrote:\n> On Sat, Jul 20, 2024 at 05:26:07PM -0400, Steven Sistare wrote:\n>> On 7/18/2024 11:56 AM, Peter Xu wrote:\n>>> Steve,\n>>>\n>>> On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n>>>> What?\n>>>\n>>> Thanks for trying out with the cpr-transfer series. I saw that that series\n>>> missed most of the cc list here, so I'm attaching the link here:\n>>>\n>>> https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n>>>\n>>> I think most of my previous questions for exec() solution still are there,\n>>> I'll try to summarize them all in this reply as much as I can.\n>>>\n>>>>\n>>>> This patch series adds the live migration cpr-exec mode, which allows\n>>>> the user to update QEMU with minimal guest pause time, by preserving\n>>>> guest RAM in place, albeit with new virtual addresses in new QEMU, and\n>>>> by preserving device file descriptors.\n>>>>\n>>>> The new user-visible interfaces are:\n>>>> * cpr-exec (MigMode migration parameter)\n>>>> * cpr-exec-command (migration parameter)\n>>>\n>>> I really, really hope we can avoid this..\n>>>\n>>> It's super cumbersome to pass in a qemu cmdline in a qemu migration\n>>> parameter.. if we can do that with generic live migration ways, I hope we\n>>> stick with the clean approach.\n>>\n>> This is no different than live migration, requiring a management agent to\n>> launch target qemu with all the arguments use to start source QEMU. Now that\n>> same agent will send the arguments via cpr-exec-command.\n> \n> It's still a bit different.\n> \n> There we append \"-incoming defer\" only, which makes sense because we're\n> instructing a QEMU to take an incoming stream to load. Now we append the\n> complete qemu cmdline within the QEMU itself, that was booted with exactly\n> the same cmdline.. :-( I would at least start to ask why we need to pass\n> the same thing twice..\n\nSometimes one must modify the command line arguments passed to new QEMU.\nThis interface allows for that possibility.\n\nIn an earlier patch series, I proposed a cpr-exec command that took no arguments,\nand reused the old QEMU argv, which was remembered in main. A reviewer pointed out\nhow inflexible that was. See my response to Daniel yesterday for more on the value\nof this flexibility.\n\nThis is not a burden for the mgmt agent. It already knows the arguments because\nit can launch new qemu with the arguments for live migration. Passing the arguments\nto cpr-exec-command is trivial.\n\n> Not saying that this is no-go, but really looks unpretty to me from this\n> part.. especially if a cleaner solution seems possible.\n> \n>>\n>>>> * anon-alloc (command-line option for -machine)\n>>>\n>>> Igor questioned this, and I second his opinion.. We can leave the\n>>> discussion there for this one.\n>>\n>> Continued on the other thread.\n>>\n>>>> The user sets the mode parameter before invoking the migrate command.\n>>>> In this mode, the user issues the migrate command to old QEMU, which\n>>>> stops the VM and saves state to the migration channels. Old QEMU then\n>>>> exec's new QEMU, replacing the original process while retaining its PID.\n>>>> The user specifies the command to exec new QEMU in the migration parameter\n>>>> cpr-exec-command. The command must pass all old QEMU arguments to new\n>>>> QEMU, plus the -incoming option. Execution resumes in new QEMU.\n>>>>\n>>>> Memory-backend objects must have the share=on attribute, but\n>>>> memory-backend-epc is not supported. The VM must be started\n>>>> with the '-machine anon-alloc=memfd' option, which allows anonymous\n>>>> memory to be transferred in place to the new process.\n>>>>\n>>>> Why?\n>>>>\n>>>> This mode has less impact on the guest than any other method of updating\n>>>> in place.\n>>>\n>>> So I wonder whether there's comparison between exec() and transfer mode\n>>> that you recently proposed.\n>>\n>> Not yet, but I will measure it.\n> \n> Thanks.\n> \n>>\n>>> I'm asking because exec() (besides all the rest of things that I dislike on\n>>> it in this approach..) should be simply slower, logically, due to the\n>>> serialized operation to (1) tearing down the old mm, (2) reload the new\n>>> ELF, then (3) runs through the QEMU init process.\n>>>\n>>> If with a generic migration solution, the dest QEMU can start running (2+3)\n>>> concurrently without even need to run (1).\n>>>\n>>> In this whole process, I doubt (2) could be relatively fast, (3) I donno,\n>>> maybe it could be slow but I never measured; Paolo may have good idea as I\n>>> know he used to work on qboot.\n>>\n>> We'll see, but in any case these take < 100 msec, which is a wonderfully short\n> \n> I doubt whether it keeps <100ms when the VM is large. Note that I think we\n> should cover the case where the user does 4k mapping for a large guest.\n> \n> So I agree that 4k mapping over e.g. 1T without hugetlb may not be the\n> ideal case, but the question is I suspect there're indeed serious users\n> using QEMU like that, and if we have most exactly a parallel solution that\n> does cover this case, it is definitely preferrable to consider the other\n> from this POV, simply because there's nothing to lose there..\n> \n>> pause time unless your customer is doing high speed stock trading. If cpr-transfer\n>> is faster still, that's gravy, but cpr-exec is still great.\n>>\n>>> For (1), I also doubt in your test cases it's fast, but it may not always\n>>> be fast. Consider the guest has a huge TBs of shared mem, even if the\n>>> memory will be completely shared between src/dst QEMUs, the pgtable won't!\n>>> It means if the TBs are mapped in PAGE_SIZE tearing down the src QEMU\n>>> pgtable alone can even take time, and that will be accounted in step (1)\n>>> and further in exec() request.\n>>\n>> Yes, there is an O(n) effect here, but it is a fast O(n) when the memory is\n>> backed by huge pages. In UEK, we make it faster still by unmapping in parallel\n>> with multiple threads. I don't have the data handy but can share after running\n>> some experiments. Regardless, this time is negligible for small and medium\n>> size guests, which form the majority of instances in a cloud.\n> \n> Possible. It's just that it sounds like a good idea to avoid having the\n> downtime taking any pgtable tearing down into account here for the old mm,\n> irrelevant of how much time it'll take. It's just that I suspect some use\n> case can take fair amount of time.\n\nHere is the guest pause time, measured as the interval from the start\nof the migrate command to the new QEMU guest reaching the running state.\nThe average over 10 runs is shown, in msecs.\nHuge pages are enabled.\nGuest memory is memfd.\nThe kernel is 6.9.0 (not UEK, so no parallel unmap)\nThe system is old and slow: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz\n\n cpr-exec cpr-transfer\n256M 180 148\n16G 190 150\n128G 250 159\n1T 300 ? 159 ? // extrapolated\n\nAt these scales, the difference between exec and transfer is not significant.\nA provider would choose one vs the other based on ease of implementation in\ntheir mgmt agent and container environment.\n\nFor small pages and large memory, cpr-exec can take multiple seconds, and\nthe UEK parallel unmap reduces that further. But, that is the exception,\nnot the rule. Providers strive to back huge memories with huge pages. It\nmakes no sense to use such a valuable resource in the crappiest way possible\n(ie with small pages).\n\n> So I think this is \"one point less\" for exec() solution, while the issue\n> can be big or small on its own. What matters is IMHO where exec() is\n> superior so that we'd like to pay for this. I'll try to stop saying \"let's\n> try to avoid using exec() as it sounds risky\", but we still need to compare\n> with solid pros and cons.\n> \n>>\n>>> All these fuss will be avoided if you use a generic live migration model\n>>> like cpr-transfer you proposed. That's also cleaner.\n>>>\n>>>> The pause time is much lower, because devices need not be torn\n>>>> down and recreated, DMA does not need to be drained and quiesced, and minimal\n>>>> state is copied to new QEMU. Further, there are no constraints on the guest.\n>>>> By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,\n>>>> and suspending plus resuming vfio devices adds multiple seconds to the\n>>>> guest pause time. Lastly, there is no loss of connectivity to the guest,\n>>>> because chardev descriptors remain open and connected.\n>>>\n>>> Again, I raised the question on why this would matter, as after all mgmt\n>>> app will need to coop with reconnections due to the fact they'll need to\n>>> support a generic live migration, in which case reconnection is a must.\n>>>\n>>> So far it doesn't sound like a performance critical path, for example, to\n>>> do the mgmt reconnects on the ports. So this might be an optimization that\n>>> most mgmt apps may not care much?\n>>\n>> Perhaps. I view the chardev preservation as nice to have, but not essential.\n>> It does not appear in this series, other than in docs. It's easy to implement\n>> given the CPR foundation. I suggest we continue this discussion when I post\n>> the chardev series, so we can focus on the core functionality.\n> \n> It's just that it can affect our decision on choosing the way to go.\n> \n> For example, do we have someone from Libvirt or any mgmt layer can help\n> justify this point?\n> \n> As I said, I thought most facilities for reconnection should be ready, but\n> I could miss important facts in mgmt layers..\n\nI will more deeply study reconnects in the mgmt layer, run some experiments to\nsee if it is seamless for the end user, and get back to you, but it will take\nsome time.\n>>>> These benefits all derive from the core design principle of this mode,\n>>>> which is preserving open descriptors. This approach is very general and\n>>>> can be used to support a wide variety of devices that do not have hardware\n>>>> support for live migration, including but not limited to: vfio, chardev,\n>>>> vhost, vdpa, and iommufd. Some devices need new kernel software interfaces\n>>>> to allow a descriptor to be used in a process that did not originally open it.\n>>>\n>>> Yes, I still think this is a great idea. It just can also be built on top\n>>> of something else than exec().\n>>>\n>>>>\n>>>> In a containerized QEMU environment, cpr-exec reuses an existing QEMU\n>>>> container and its assigned resources. By contrast, consider a design in\n>>>> which a new container is created on the same host as the target of the\n>>>> CPR operation. Resources must be reserved for the new container, while\n>>>> the old container still reserves resources until the operation completes.\n>>>\n>>> Note that if we need to share RAM anyway, the resources consumption should\n>>> be minimal, as mem should IMHO be the major concern (except CPU, but CPU\n>>> isn't a concern in this scenario) in container world and here the shared\n>>> guest mem shouldn't be accounted to the dest container. So IMHO it's about\n>>> the metadata QEMU/KVM needs to do the hypervisor work, it seems to me, and\n>>> that should be relatively small.\n>>>\n>>> In that case I don't yet see it a huge improvement, if the dest container\n>>> is cheap to initiate.\n>>\n>> It's about reserving memory and CPUs, and transferring those reservations from\n>> the old instance to the new, and fiddling with the OS mechanisms that enforce\n>> reservations and limits. The devil is in the details, and with the exec model,\n>> the management agent can ignore all of that.\n>>\n>> You don't see it as a huge improvement because you don't need to write the\n>> management code. I do!\n> \n> Heh, possibly true.\n> \n> Could I ask what management code you're working on? Why that management\n> code doesn't need to already work out these problems with reconnections\n> (like pre-CPR ways of live upgrade)?\n\nOCI - Oracle Cloud Infrastructure.\nMgmt needs to manage reconnections for live migration, and perhaps I could\nleverage that code for live update, but happily I did not need to. Regardless,\nreconnection is the lesser issue. The bigger issue is resource management and\nthe container environment. But I cannot justify that statement in detail without\nactually trying to implement cpr-transfer in OCI.\n\n>> Both modes are valid and useful - exec in container, or launch a new container.\n>> I have volunteered to implement the cpr-transfer mode for the latter, a mode\n>> I do not use. Please don't reward me by dropping the mode I care about :)\n>> Both modes can co-exist. The presence of the cpr-exec specific code in qemu\n>> will not hinder future live migration development.\n> \n> I'm trying to remove some of my \"prejudices\" on exec() :). Hopefully that\n> proved more or less that I simply wanted to be fair on making a design\n> decision. I don't think I have a strong opinion, but it looks to me not\n> ideal to merge two solutions if both modes share the use case.\n> \n> Or if you think both modes should service different purpose, we might\n> consider both, but that needs to be justified - IOW, we shouldn't merge\n> anything that will never be used.\n\nThe use case is the same for both modes, but they are simply different\ntransport methods for moving descriptors from old QEMU to new. The developer\nof the mgmt agent should be allowed to choose.\n\n- Steve\n\n>>>> Avoiding over commitment requires extra work in the management layer.\n>>>\n>>> So it would be nice to know what needs to be overcommitted here. I confess\n>>> I don't know much on containerized VMs, so maybe the page cache can be a\n>>> problem even if shared. But I hope we can spell that out. Logically IIUC\n>>> memcg shouldn't account those page cache if preallocated, because memcg\n>>> accounting should be done at folio allocations, at least, where the page\n>>> cache should miss first (so not this case..).\n>>>\n>>>> This is one reason why a cloud provider may prefer cpr-exec. A second reason\n>>>> is that the container may include agents with their own connections to the\n>>>> outside world, and such connections remain intact if the container is reused.\n>>>>\n>>>> How?\n>>\n>> chardev preservation. The qemu socket chardevs to these agents are preserved,\n>> and the agent connections to the outside world do not change, so no one sees\n>> any interruption of traffic.\n>>\n>>>> All memory that is mapped by the guest is preserved in place. Indeed,\n>>>> it must be, because it may be the target of DMA requests, which are not\n>>>> quiesced during cpr-exec. All such memory must be mmap'able in new QEMU.\n>>>> This is easy for named memory-backend objects, as long as they are mapped\n>>>> shared, because they are visible in the file system in both old and new QEMU.\n>>>> Anonymous memory must be allocated using memfd_create rather than MAP_ANON,\n>>>> so the memfd's can be sent to new QEMU. Pages that were locked in memory\n>>>> for DMA in old QEMU remain locked in new QEMU, because the descriptor of\n>>>> the device that locked them remains open.\n>>>>\n>>>> cpr-exec preserves descriptors across exec by clearing the CLOEXEC flag,\n>>>> and by sending the unique name and value of each descriptor to new QEMU\n>>>> via CPR state.\n>>>>\n>>>> For device descriptors, new QEMU reuses the descriptor when creating the\n>>>> device, rather than opening it again. The same holds for chardevs. For\n>>>> memfd descriptors, new QEMU mmap's the preserved memfd when a ramblock\n>>>> is created.\n>>>>\n>>>> CPR state cannot be sent over the normal migration channel, because devices\n>>>> and backends are created prior to reading the channel, so this mode sends\n>>>> CPR state over a second migration channel that is not visible to the user.\n>>>> New QEMU reads the second channel prior to creating devices or backends.\n>>>\n>>> Oh, maybe this is the reason that cpr-transfer will need a separate uri..\n>>\n>> Indeed.\n>>\n>> - Steve\n>>\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=nhuO6SIh;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=T4JIXNTy;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WfLN30YlPz1yfC\n\tfor ; Thu, 8 Aug 2024 05:49:04 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sbmdm-00075F-Eu; Wed, 07 Aug 2024 15:48:10 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sbmdk-00074A-5I\n for qemu-devel@nongnu.org; Wed, 07 Aug 2024 15:48:08 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sbmdh-0001Mm-0d\n for qemu-devel@nongnu.org; Wed, 07 Aug 2024 15:48:07 -0400","from pps.filterd (m0333521.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 477IteBX011708;\n Wed, 7 Aug 2024 19:47:57 GMT","from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com\n (phxpaimrmta01.appoci.oracle.com [138.1.114.2])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40sbfarff2-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Wed, 07 Aug 2024 19:47:56 +0000 (GMT)","from pps.filterd\n (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 477J53hq018361; Wed, 7 Aug 2024 19:47:56 GMT","from nam12-dm6-obe.outbound.protection.outlook.com\n (mail-dm6nam12lp2169.outbound.protection.outlook.com [104.47.59.169])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id\n 40sb0adx6q-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Wed, 07 Aug 2024 19:47:56 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by CH0PR10MB7533.namprd10.prod.outlook.com (2603:10b6:610:183::6)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7849.13; Wed, 7 Aug\n 2024 19:47:53 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7828.023; Wed, 7 Aug 2024\n 19:47:53 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=iitJFWZ6Cz1ulRg/QX97n991CUG423qa2rhdAhKsJyI=; b=\n nhuO6SIhD4p6VLdfRlxZz30XQXWCD0HvEvsR3HoP465BwDTmC1TeG2HmuMRIiJRu\n SH71BX3PqgN7n9QY9gTiBl0b9Oxrr1Hn51m7nZWobtb3ogVtahhdNFRbiKwHiHiH\n wW9sJpoY2+AezSDjxb9J/NPOXh0wzpMzLSE4qXzbDAak9LY5kTG/MWwqVQvCqYIx\n rnyng0RUBJQHt3DvUi4nLwhOa12GKxJBmw8zyzBPJ4+SDN9u6+tl8/8WyWaCP5gO\n QaQBTKlN7kbwVME79f75PWFZpWF/BGlmLKHNbsNgXBRqPz19BtrL7+IAmoj2/Br3\n iaAMwFYooM/0B7g8i4LZVQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=iitJFWZ6Cz1ulRg/QX97n991CUG423qa2rhdAhKsJyI=;\n b=T4JIXNTyeWVMvNZVKEnrMoHdmuCRSo/dadInyjqvL96uLD1oJ5/NKT+xGAReTToC2uwDSF8XJzaSumwZx4KUoAdWOLCaYDkPrGFsW15RofPePWk4gQ7e2LEpVGzeIaNMFhlYlz0lejUW/ZWlwrHHa7a1jNSrMhRuBpk3yLvjODM="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=HzO7hJIFL2voQ1/rV53RQuPmwkVymL6Y4v9xO7MnZ4/gpXo5Vn+IhNaDmf+RwC4os7CG/tTdDnWESMPc8d1Cf2F4EFXljzD84t5PgspQ3lWwgWrl6mT9NOz6voBec9NbhwGYIaaIPK3/lFoYmOJsKY05kQr81FIkfL7Pdu9dBT5K/9W91FquZAI4f1i9SeOvn0iAaNbnCzE33S74QSXIa6kbxqO7vV7gpAohJy7MJPfcgKVJzPQB5WSqWsKWYVfOjAT+7LtLzLKP1onWFKDrXeCmzsMKaBOCP3IICzuWjFk+odQeDH3fe9HsCUgBm+0Gy2X/b/F6NtZ8YyzZKmDdCg==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=iitJFWZ6Cz1ulRg/QX97n991CUG423qa2rhdAhKsJyI=;\n b=O8MXDBO2qewmhFcGzh3WPig29zFaCEBYTUk3bphk4yN0Rjs5/HCCa/DeN/aUvg7suFCIALxlYGHQZTapjBh42bS9wE/4XfubQRB84HtQSGWzpSymIMO81cwuaCTxF0nfJ0FjgBcjw0jhftuk5JrYvg4Ly/2GwA37G57F6YucmrLbX2NwY92j7/vMqz8MKe7PmnJ14XEOKxIos7lU6UWhOmxJO9xphTQPFyP1ZgmOsXhgXSgfUvdcAzJ+AfZItoki1ejFWByAgcToTYjFj0znLp8WhVAuRjP0A5DVkkq8YgBwMyqFHbBJCqdG0o35piOO9iB4qvaAMyVoYKCNtesGnA==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"","Date":"Wed, 7 Aug 2024 15:47:47 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>\n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-ClientProxiedBy":"BYAPR02CA0002.namprd02.prod.outlook.com\n (2603:10b6:a02:ee::15) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|CH0PR10MB7533:EE_","X-MS-Office365-Filtering-Correlation-Id":"1899f031-863e-4d04-9d39-08dcb719d376","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|366016|1800799024|376014|7416014;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?/tYEbbEHUYF3PiIf4mZ+/nz/iL6iurS?=\n\t=?utf-8?q?oLda5ZAoqV87j9ddZ0uCMcuCBuem8SqkCwP44kjhvs2VGYHOgbSQptMWTwaN2vQNY?=\n\t=?utf-8?q?MvrJYCBHHVt0F7SYr79YmdRp5sr4vUlCk1ln0/DGGNz+JHT5QrREu+a8R1HPumfnf?=\n\t=?utf-8?q?38IOa6WJtv0omGHE8xS3YJpYxeaabbTGvhBb+7raSh98/sAQz3CT6qV64XYgAQEbs?=\n\t=?utf-8?q?m+FTkq9GpoMMzLepEnKGmReEa6r7AOoizNVYPMqsFIHxzH6ATlcx0XBDSUfqVgD1a?=\n\t=?utf-8?q?Rb/VP2NRcvaC8MggTXlniG+uFmXk+iRgCXuevxb01sD7XkUj2LZMPs/Z7Rwu/sJ9w?=\n\t=?utf-8?q?Rk0IyShVT9JEXpFIJYguXaRzNKHA66wGIpRwlTMz1lJUrICbWmtAXC35Wa2JPBw6w?=\n\t=?utf-8?q?gKN1W7vGcobNtTYzqnkcSIzgxHICCcO3FIakAnduNfSuEpzNYtqTpb16NKlMt46TE?=\n\t=?utf-8?q?28lHMWtNtHFnmzlu6MCBJciXo9YL2X6+3sGj36PVO1oIEMsO2FRHahJtVUutQe02A?=\n\t=?utf-8?q?emYUFb2e3ByeElfUrC/NOSN6o3hM2M2DoySDNaiGQbu9gUp2WjYCfgcDjAo2Dq5P+?=\n\t=?utf-8?q?E298AHeQtJqp5rKtkbv3lmDRZEN7vwC7pNX1JfiKK92D7X6lFSE5fRFIgqtTzb5xR?=\n\t=?utf-8?q?4XBJlex6vD4NZJagGyGhi63OYne5GUdFRabWtPdhyQvKgk3HOQNjinfU2/auHC3c8?=\n\t=?utf-8?q?u89PnQf4llr/R8bu+/qwRWHdKY6k+I2bjCkZo4EikDiSKOXpQTwzaAk5SH/oIrSMc?=\n\t=?utf-8?q?OQRSCghDttgenZU2kvo/BFaJMK29JTxOp/8GVo1G7QJ3fGDUi+BCNR8wcoTpOrdRw?=\n\t=?utf-8?q?zI9z6yE/lhey+k613zcOVsK/Cw3kxwHG+3vXkOi5AV4t7Ij1eIE3M/iXK+jFxhwDq?=\n\t=?utf-8?q?UO98apKiPCWqBETIooz3rjC1oQkCBUAWJmpubfIfKmNMmIgSbFAwQI9HfOlPBsurl?=\n\t=?utf-8?q?5HjlVyPqg8DtD3eMUPGUM+xajrtyQrRP5062DR60yn8BkJ01jDk3RzVI26/bJD/hQ?=\n\t=?utf-8?q?7jVI2oFQs/Nn7smFgUTy5KHMCb/I7U68qb4NpAgIbay1E8+fW/U76KNtu9ZX7fRnn?=\n\t=?utf-8?q?MHyJj1AwLYGZAz8XN9hVLceTOzKCEm4SdNnBGmhHK7UcM3NfEaVJE9Hd9X++xkqd1?=\n\t=?utf-8?q?oo9x4GbJNvbD8SIxeTFzpYFqUEgcok1kPagN5ytqW0KwTh3AxpK+xEx7MPxCor2KG?=\n\t=?utf-8?q?28PaFl6qUYKoYB0nZcYMcvaRQaZh3PE4sz1qTGQ0SJM98ym7LSrxQKG9w=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(1800799024)(376014)(7416014); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?F9X33ChWmddvskF8QiiflRzrU7me?=\n\t=?utf-8?q?8/SKEToSGPw/agZAqwwiFJHtS8CIYFQjCwSQmGc9JC904C6ijNdww0DuZid9XPryJ?=\n\t=?utf-8?q?4utybCSmBypeLBOtGmbv/yVUFjxAQAHzQTHqQA6tIrUXnV/f7M2Jx7UeVAIKT348D?=\n\t=?utf-8?q?wew9tH78HK9/YR12m2L9WtUAWVX/LUNH4ZC2XDTbrECEUATxSZGlt4nlqRDCqN81V?=\n\t=?utf-8?q?2GyW4nuu2pZBI1knDs/kimATwd9rlinv4xNO27hTI40/6OBZq2w8oL8fJ36Q+XlLP?=\n\t=?utf-8?q?yq1s2dI25b2cI4d7IP89r2Cli8wX/fQJ/3eWgTLHjWPZoDUIO6w1Sp6vfKTBgDiNZ?=\n\t=?utf-8?q?CkyUMNYBpecbK7RrfTjyrllynlPXPg/6cRE4zmfeuLzuqTB1SjQiASDJga7755Fi3?=\n\t=?utf-8?q?UpHTiNvgRtcjViMgKIj/RFLs+M5+uH5JtBGgrwTz6VXnbyj7+O4Bx57oYmSDCOi5x?=\n\t=?utf-8?q?xHOrj/3E8f3yuh6z186xrqX4K0sMXxqhb43rBXI80HkQILLDxadnhpNRxOPQ88NRd?=\n\t=?utf-8?q?Hv9iwLxWukqiYCmdO/j61GMzX+E4SyNl6CsNk1PB1OL1MzdpoAbl7KlH8nJQLSdI2?=\n\t=?utf-8?q?bxNp8Evi6FlThLT02wo9f17m3tmrdEnXw23nInvEIRMeLVMvs4tCBk5VTFaWcxJVz?=\n\t=?utf-8?q?Xy71vkoKy29a+AOrfRpAZwahQVZnWK8gYguffWepflAgij9qD7MdVRi4EhSDzWTll?=\n\t=?utf-8?q?NYhcn3bh3mKoZvzCxdfFtq8KrXKu/DBLjlzm2P1HxdjEeJhVw32MQXzxXAREwknA/?=\n\t=?utf-8?q?nq6kUudk+g3sc3CrEOowkOHcyUFCkkmeVbB7HHqfiy5ZOlmP87PYDNachtz7b4+eb?=\n\t=?utf-8?q?X8TKyCJnxxmKXnQnywPi0A690vidMRz0MhXh3NatdCJhbEI+JqEJ/8R1DW/4iKTDF?=\n\t=?utf-8?q?vR+9eOUI41Haj96OI8YqnkMYjp141ZHcvwjJJzKO7Lav3IW0/S8pLE5tVHuxSHP53?=\n\t=?utf-8?q?1QzN/raA+VBhqnr6kZMIMqczRUGmliGfAdbEIYfzgbN18rwbz1DZ8JoZszM2ePyqW?=\n\t=?utf-8?q?NtdLCP9lYSMDemTLWyIOwOKOV+xFroOdFnWIBpWFtYFfZOdVxcFPD3ELWhO03LmUk?=\n\t=?utf-8?q?8o06aQXHJ7HJ9e+qFbIbocVKVB04bCGiuLjmeaSqWYxV9xlz6as0Gy7NKnB/cZgJQ?=\n\t=?utf-8?q?pts1wjs8ysAL3+sTfNhRR9dHXUUE1Wf74W2kauQF76gltxdwFIJaQinyCMq2vT8/c?=\n\t=?utf-8?q?TMnhL5Xsot8pHNWxkTypBRsrgCHL2XsPR+zF6Umm2/IjZuwNDhFDHG1ob+XwrUVhR?=\n\t=?utf-8?q?4auwHLa2MF3t+nWGHKycgbx4GgjLi70IQ+f/kO5+JKWX0ZYP3v/ufm+q1BdJ4exDy?=\n\t=?utf-8?q?4lPF26gIU626vLuzozbCWvKDIoW4UzQCdC4w2A+8hqgcvLSHDzBEYniawcw5pUfdc?=\n\t=?utf-8?q?uIZGOh0MdktugmNwbOWMMXemZIKMT5JDpyzcCGzOJA8mWBmqZucUeKyRzUkz9uMq7?=\n\t=?utf-8?q?8rObRg8hPpZ5+fZO/HMQ1K/vNXHe65JKDaOJgAxd181hHn6yeS91z7PBnQl2KTUf8?=\n\t=?utf-8?q?1kX5Iy6bEJHpXC30IPmoFPVv0h80t+OSMA=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n SeE0zDfZq7Um9AWVy1RVCCvKCrkY2mATJVSavB4q2ai4Oao5ZxIvi8vW83kbOre0+taTVvv/gv/MNgqhURX3wr6EsfxJsOI39DzTUEcJNbSXE276pwQ1sZ8b23h2ZQdByQg+x+MzpwHDEoEhBr3PU8ZeeoNrMz/zKVbYvSLuFcdF8ziLEKwbffnRt3UGaY//t1swuZJsSD1nHmCNE/IdQZQSlnDkM35UsFajMVELvDZcz9LvwjZ3MGc0x6dHbHp8rtks6I7I39JG7y57sAgJZR2FefoZGAT+/FFp2f7KuiSqyFl8Eymt1OZLrRYCs4z9xgiUk8nG8xySpFVEWfMxZ9TMyGIacBad+CGrl3846Y+wYVaRWh/8kNFZuI5pJdbpnPgryPhOvdTj5gdS7nkminMikfv3UIRJcfxYxRDnuYRJ1KLRNqJ6g5EO9nuy7ckkloLze8BQ1wwQEvNMcjaBV9QBdiDw3cLeGi4JlO9mVu6VW0w7hY4pGqEzX8gKKPx+UK2c8e04UFsMMGS1rG4i3vQnqb5t48Hah1A2mx18jxi0VlE6pOYTG3+Mq6Yz7zz0LeqyMvMQkitYol1lUnezm1zeZjq+yFhVpYz3dEzXTFQ=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 1899f031-863e-4d04-9d39-08dcb719d376","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"07 Aug 2024 19:47:53.1682 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n Qt9j+fVF0l8DcKXGsoaq6w8pjmgIfUxhdkvbUmsVsxEhSFCHM6BcTM4iy+g3t/aZrhB7gWiEuXFJplVCXZXD92YQOeMIbXBjVt/24bUjeKE=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CH0PR10MB7533","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-07_11,2024-08-07_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 suspectscore=0\n spamscore=0 adultscore=0\n mlxscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2408070138","X-Proofpoint-GUID":"p-_qjcAnaHrUxC_qAt6w1b-xYa36Yn6Y","X-Proofpoint-ORIG-GUID":"p-_qjcAnaHrUxC_qAt6w1b-xYa36Yn6Y","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3361810,"web_url":"http://patchwork.ozlabs.org/comment/3361810/","msgid":"","list_archive_url":null,"date":"2024-08-13T19:46:09","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > The flipside, however, is that localhost migration via 2 separate QEMU\n> > processes has issues where both QEMUs want to be opening the very same\n> > file, and only 1 of them can ever have them open.\n\nI thought we used to have similar issue with block devices, but I assume\nit's solved for years (and whoever owns it will take proper file lock,\nIIRC, and QEMU migration should properly serialize the time window on who's\ngoing to take the file lock).\n\nMaybe this is about something else?\n\n> \n> Indeed, and \"files\" includes unix domain sockets. Network ports also conflict.\n> cpr-exec avoids such problems, and is one of the advantages of the method that\n> I forgot to promote.\n\nI was thinking that's fine, as the host ports should be the backend of the\nVM ports only anyway so they don't need to be identical on both sides?\n\nIOW, my understanding is it's the guest IP/ports/... which should still be\nstable across migrations, where the host ports can be different as long as\nthe host ports can forward guest port messages correctly?\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=aGqPNGJG;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wk23D5d4sz1yfP\n\tfor ; Wed, 14 Aug 2024 05:47:20 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sdxTH-0005ou-3q; Tue, 13 Aug 2024 15:46:19 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sdxTG-0005oP-0C\n for qemu-devel@nongnu.org; Tue, 13 Aug 2024 15:46:18 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sdxTE-00015K-I2\n for qemu-devel@nongnu.org; Tue, 13 Aug 2024 15:46:17 -0400","from mail-ot1-f71.google.com (mail-ot1-f71.google.com\n [209.85.210.71]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-175-njqxElm_PYaOGz9ElA2Tfw-1; Tue, 13 Aug 2024 15:46:13 -0400","by mail-ot1-f71.google.com with SMTP id\n 46e09a7af769-7093752a9f5so1063083a34.1\n for ; Tue, 13 Aug 2024 12:46:13 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n d75a77b69052e-4531c1c532asm35029581cf.30.2024.08.13.12.46.10\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Tue, 13 Aug 2024 12:46:11 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723578374;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=hOQW1bo3800LACsO6zbK6WIEbmkoNuyCxGIAh7tVsKU=;\n b=aGqPNGJGdUYviuRcQuoTDHxFE3WFSaAqLeaHotmNK7KufJnBqsmEtMkZmdwyJPyP4uYo+E\n 3AjVcK1DYTDcnSGVgdzAyHhgrDX0zSWCZP1/s5SY6cL50E5JSAI9PdrgzDMotTazFMrTc6\n AfEfmdA/c83k16drXgzDZk/bvbNWGsQ=","X-MC-Unique":"njqxElm_PYaOGz9ElA2Tfw-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1723578372; x=1724183172;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=hOQW1bo3800LACsO6zbK6WIEbmkoNuyCxGIAh7tVsKU=;\n b=cSIDx40UPtkJsRWPTmW5kze3rqZO1Z302w+FtXZu5tgQoeTyJdZDkzUCVe8NG9Fxhq\n u4nW+/fh66Vrd8ythbsCAWnV8fdJtiXC1pfa2DtqgaahZgnojVPsO/AOutmWk9XthDPM\n NMbKCvdj2Etz7s0EgMApONWZaISJWdnwEX/GLVkpf0mbs+xkd7F8Z+L350TEWAvt0uFW\n 41glkycwp6gTpQ/nIO4vuTY/x2koIY+43ryEYkLFsZLr7OPreLVq35KcI7L/ffEdVGmB\n MY8292oWbiL3iqKMa2zOpqUV/oc8Vp6ab52cs/pHhX7WFnhj+O5E6y1Ei1HY/y5H7Unu\n GO7g==","X-Forwarded-Encrypted":"i=1;\n AJvYcCUc0f0dGM9W2+GBrQSdaqoM1oL5+UyTZBLXjqxiHFCe8xWhjowse9JiLBYqfsruFGcWWzzL8XcSfavz@nongnu.org","X-Gm-Message-State":"AOJu0YwC7544dJOJRahy/2sQRX35N+r7GbjIxHr+COYe1mcYMuqVIzQv\n ssc+YVoG7qkduwXxVaUGolmNKA/BOFKTDiioU0bmquxa3lAVGheI7oGmaCbybyUW5LirCjYw9c/\n ylsuBqh22bTFStSNizT4CJyN72VMOR4kQ2ex9abk/pEA4IMLcZtpt","X-Received":["by 2002:a05:6808:3713:b0:3db:427d:7da0 with SMTP id\n 5614622812f47-3dd297de0eemr229509b6e.2.1723578372279;\n Tue, 13 Aug 2024 12:46:12 -0700 (PDT)","by 2002:a05:6808:3713:b0:3db:427d:7da0 with SMTP id\n 5614622812f47-3dd297de0eemr229494b6e.2.1723578371884;\n Tue, 13 Aug 2024 12:46:11 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IFWSHjY12Avlz9OxZQBUTx+UbdvuVihwRint/Tz/tkok8U0qQ5i46z999mYKpaE//Qhsh4C/A==","Date":"Tue, 13 Aug 2024 15:46:09 -0400","From":"Peter Xu ","To":"Steven Sistare ","Cc":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ,\n qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.125,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3361828,"web_url":"http://patchwork.ozlabs.org/comment/3361828/","msgid":"","list_archive_url":null,"date":"2024-08-13T20:12:16","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Wed, Aug 07, 2024 at 03:47:47PM -0400, Steven Sistare wrote:\n> On 8/4/2024 12:10 PM, Peter Xu wrote:\n> > On Sat, Jul 20, 2024 at 05:26:07PM -0400, Steven Sistare wrote:\n> > > On 7/18/2024 11:56 AM, Peter Xu wrote:\n> > > > Steve,\n> > > > \n> > > > On Sun, Jun 30, 2024 at 12:40:23PM -0700, Steve Sistare wrote:\n> > > > > What?\n> > > > \n> > > > Thanks for trying out with the cpr-transfer series. I saw that that series\n> > > > missed most of the cc list here, so I'm attaching the link here:\n> > > > \n> > > > https://lore.kernel.org/r/1719776648-435073-1-git-send-email-steven.sistare@oracle.com\n> > > > \n> > > > I think most of my previous questions for exec() solution still are there,\n> > > > I'll try to summarize them all in this reply as much as I can.\n> > > > \n> > > > > \n> > > > > This patch series adds the live migration cpr-exec mode, which allows\n> > > > > the user to update QEMU with minimal guest pause time, by preserving\n> > > > > guest RAM in place, albeit with new virtual addresses in new QEMU, and\n> > > > > by preserving device file descriptors.\n> > > > > \n> > > > > The new user-visible interfaces are:\n> > > > > * cpr-exec (MigMode migration parameter)\n> > > > > * cpr-exec-command (migration parameter)\n> > > > \n> > > > I really, really hope we can avoid this..\n> > > > \n> > > > It's super cumbersome to pass in a qemu cmdline in a qemu migration\n> > > > parameter.. if we can do that with generic live migration ways, I hope we\n> > > > stick with the clean approach.\n> > > \n> > > This is no different than live migration, requiring a management agent to\n> > > launch target qemu with all the arguments use to start source QEMU. Now that\n> > > same agent will send the arguments via cpr-exec-command.\n> > \n> > It's still a bit different.\n> > \n> > There we append \"-incoming defer\" only, which makes sense because we're\n> > instructing a QEMU to take an incoming stream to load. Now we append the\n> > complete qemu cmdline within the QEMU itself, that was booted with exactly\n> > the same cmdline.. :-( I would at least start to ask why we need to pass\n> > the same thing twice..\n> \n> Sometimes one must modify the command line arguments passed to new QEMU.\n> This interface allows for that possibility.\n> \n> In an earlier patch series, I proposed a cpr-exec command that took no arguments,\n> and reused the old QEMU argv, which was remembered in main. A reviewer pointed out\n> how inflexible that was. See my response to Daniel yesterday for more on the value\n> of this flexibility.\n> \n> This is not a burden for the mgmt agent. It already knows the arguments because\n> it can launch new qemu with the arguments for live migration. Passing the arguments\n> to cpr-exec-command is trivial.\n\nRight, trivial as-is. To me it's not a major blocker yet so far, but it's\nstill about being hackish, and I have this unpleasant feeling that we're\ndigging holes for our future.\n\n> \n> > Not saying that this is no-go, but really looks unpretty to me from this\n> > part.. especially if a cleaner solution seems possible.\n> > \n> > > \n> > > > > * anon-alloc (command-line option for -machine)\n> > > > \n> > > > Igor questioned this, and I second his opinion.. We can leave the\n> > > > discussion there for this one.\n> > > \n> > > Continued on the other thread.\n> > > \n> > > > > The user sets the mode parameter before invoking the migrate command.\n> > > > > In this mode, the user issues the migrate command to old QEMU, which\n> > > > > stops the VM and saves state to the migration channels. Old QEMU then\n> > > > > exec's new QEMU, replacing the original process while retaining its PID.\n> > > > > The user specifies the command to exec new QEMU in the migration parameter\n> > > > > cpr-exec-command. The command must pass all old QEMU arguments to new\n> > > > > QEMU, plus the -incoming option. Execution resumes in new QEMU.\n> > > > > \n> > > > > Memory-backend objects must have the share=on attribute, but\n> > > > > memory-backend-epc is not supported. The VM must be started\n> > > > > with the '-machine anon-alloc=memfd' option, which allows anonymous\n> > > > > memory to be transferred in place to the new process.\n> > > > > \n> > > > > Why?\n> > > > > \n> > > > > This mode has less impact on the guest than any other method of updating\n> > > > > in place.\n> > > > \n> > > > So I wonder whether there's comparison between exec() and transfer mode\n> > > > that you recently proposed.\n> > > \n> > > Not yet, but I will measure it.\n> > \n> > Thanks.\n> > \n> > > \n> > > > I'm asking because exec() (besides all the rest of things that I dislike on\n> > > > it in this approach..) should be simply slower, logically, due to the\n> > > > serialized operation to (1) tearing down the old mm, (2) reload the new\n> > > > ELF, then (3) runs through the QEMU init process.\n> > > > \n> > > > If with a generic migration solution, the dest QEMU can start running (2+3)\n> > > > concurrently without even need to run (1).\n> > > > \n> > > > In this whole process, I doubt (2) could be relatively fast, (3) I donno,\n> > > > maybe it could be slow but I never measured; Paolo may have good idea as I\n> > > > know he used to work on qboot.\n> > > \n> > > We'll see, but in any case these take < 100 msec, which is a wonderfully short\n> > \n> > I doubt whether it keeps <100ms when the VM is large. Note that I think we\n> > should cover the case where the user does 4k mapping for a large guest.\n> > \n> > So I agree that 4k mapping over e.g. 1T without hugetlb may not be the\n> > ideal case, but the question is I suspect there're indeed serious users\n> > using QEMU like that, and if we have most exactly a parallel solution that\n> > does cover this case, it is definitely preferrable to consider the other\n> > from this POV, simply because there's nothing to lose there..\n> > \n> > > pause time unless your customer is doing high speed stock trading. If cpr-transfer\n> > > is faster still, that's gravy, but cpr-exec is still great.\n> > > \n> > > > For (1), I also doubt in your test cases it's fast, but it may not always\n> > > > be fast. Consider the guest has a huge TBs of shared mem, even if the\n> > > > memory will be completely shared between src/dst QEMUs, the pgtable won't!\n> > > > It means if the TBs are mapped in PAGE_SIZE tearing down the src QEMU\n> > > > pgtable alone can even take time, and that will be accounted in step (1)\n> > > > and further in exec() request.\n> > > \n> > > Yes, there is an O(n) effect here, but it is a fast O(n) when the memory is\n> > > backed by huge pages. In UEK, we make it faster still by unmapping in parallel\n> > > with multiple threads. I don't have the data handy but can share after running\n> > > some experiments. Regardless, this time is negligible for small and medium\n> > > size guests, which form the majority of instances in a cloud.\n> > \n> > Possible. It's just that it sounds like a good idea to avoid having the\n> > downtime taking any pgtable tearing down into account here for the old mm,\n> > irrelevant of how much time it'll take. It's just that I suspect some use\n> > case can take fair amount of time.\n> \n> Here is the guest pause time, measured as the interval from the start\n> of the migrate command to the new QEMU guest reaching the running state.\n> The average over 10 runs is shown, in msecs.\n> Huge pages are enabled.\n> Guest memory is memfd.\n> The kernel is 6.9.0 (not UEK, so no parallel unmap)\n> The system is old and slow: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz\n> \n> cpr-exec cpr-transfer\n> 256M 180 148\n> 16G 190 150\n> 128G 250 159\n> 1T 300 ? 159 ? // extrapolated\n> \n> At these scales, the difference between exec and transfer is not significant.\n> A provider would choose one vs the other based on ease of implementation in\n> their mgmt agent and container environment.\n\nThanks. Are these maximum huge page the memsize can take (either 2M or 1G),\nor is this only about 2M?\n\n> \n> For small pages and large memory, cpr-exec can take multiple seconds, and\n> the UEK parallel unmap reduces that further. But, that is the exception,\n> not the rule. Providers strive to back huge memories with huge pages. It\n> makes no sense to use such a valuable resource in the crappiest way possible\n> (ie with small pages).\n\nI can't say nobody uses small pages when memory is large. The thing is\nhugetlb loses features comparing to small pages; or say, merely all the\nmemory features Linux provides. So I won't be surprised if someone tells\nme there's TB level VMs that is using small page on purpose for any of\nthose features (swap, ksm, etc.). I had vague memory there was customers\nusing such setup in the past, without remembering the reasons.\n\nI do wish a design that will perform well even there, so it works for all\ncases we can think of so far. Not to mention cpr-transfer seems to\noutperforms everywhere too. Unless the \"management layer benefits\" are so\nstrong.. it seems to me we have a clear choice. I understand it may affect\nyour plan, but let's be fair, or.. is it not the case?\n\n> \n> > So I think this is \"one point less\" for exec() solution, while the issue\n> > can be big or small on its own. What matters is IMHO where exec() is\n> > superior so that we'd like to pay for this. I'll try to stop saying \"let's\n> > try to avoid using exec() as it sounds risky\", but we still need to compare\n> > with solid pros and cons.\n> > \n> > > \n> > > > All these fuss will be avoided if you use a generic live migration model\n> > > > like cpr-transfer you proposed. That's also cleaner.\n> > > > \n> > > > > The pause time is much lower, because devices need not be torn\n> > > > > down and recreated, DMA does not need to be drained and quiesced, and minimal\n> > > > > state is copied to new QEMU. Further, there are no constraints on the guest.\n> > > > > By contrast, cpr-reboot mode requires the guest to support S3 suspend-to-ram,\n> > > > > and suspending plus resuming vfio devices adds multiple seconds to the\n> > > > > guest pause time. Lastly, there is no loss of connectivity to the guest,\n> > > > > because chardev descriptors remain open and connected.\n> > > > \n> > > > Again, I raised the question on why this would matter, as after all mgmt\n> > > > app will need to coop with reconnections due to the fact they'll need to\n> > > > support a generic live migration, in which case reconnection is a must.\n> > > > \n> > > > So far it doesn't sound like a performance critical path, for example, to\n> > > > do the mgmt reconnects on the ports. So this might be an optimization that\n> > > > most mgmt apps may not care much?\n> > > \n> > > Perhaps. I view the chardev preservation as nice to have, but not essential.\n> > > It does not appear in this series, other than in docs. It's easy to implement\n> > > given the CPR foundation. I suggest we continue this discussion when I post\n> > > the chardev series, so we can focus on the core functionality.\n> > \n> > It's just that it can affect our decision on choosing the way to go.\n> > \n> > For example, do we have someone from Libvirt or any mgmt layer can help\n> > justify this point?\n> > \n> > As I said, I thought most facilities for reconnection should be ready, but\n> > I could miss important facts in mgmt layers..\n> \n> I will more deeply study reconnects in the mgmt layer, run some experiments to\n> see if it is seamless for the end user, and get back to you, but it will take\n> some time.\n> > > > > These benefits all derive from the core design principle of this mode,\n> > > > > which is preserving open descriptors. This approach is very general and\n> > > > > can be used to support a wide variety of devices that do not have hardware\n> > > > > support for live migration, including but not limited to: vfio, chardev,\n> > > > > vhost, vdpa, and iommufd. Some devices need new kernel software interfaces\n> > > > > to allow a descriptor to be used in a process that did not originally open it.\n> > > > \n> > > > Yes, I still think this is a great idea. It just can also be built on top\n> > > > of something else than exec().\n> > > > \n> > > > > \n> > > > > In a containerized QEMU environment, cpr-exec reuses an existing QEMU\n> > > > > container and its assigned resources. By contrast, consider a design in\n> > > > > which a new container is created on the same host as the target of the\n> > > > > CPR operation. Resources must be reserved for the new container, while\n> > > > > the old container still reserves resources until the operation completes.\n> > > > \n> > > > Note that if we need to share RAM anyway, the resources consumption should\n> > > > be minimal, as mem should IMHO be the major concern (except CPU, but CPU\n> > > > isn't a concern in this scenario) in container world and here the shared\n> > > > guest mem shouldn't be accounted to the dest container. So IMHO it's about\n> > > > the metadata QEMU/KVM needs to do the hypervisor work, it seems to me, and\n> > > > that should be relatively small.\n> > > > \n> > > > In that case I don't yet see it a huge improvement, if the dest container\n> > > > is cheap to initiate.\n> > > \n> > > It's about reserving memory and CPUs, and transferring those reservations from\n> > > the old instance to the new, and fiddling with the OS mechanisms that enforce\n> > > reservations and limits. The devil is in the details, and with the exec model,\n> > > the management agent can ignore all of that.\n> > > \n> > > You don't see it as a huge improvement because you don't need to write the\n> > > management code. I do!\n> > \n> > Heh, possibly true.\n> > \n> > Could I ask what management code you're working on? Why that management\n> > code doesn't need to already work out these problems with reconnections\n> > (like pre-CPR ways of live upgrade)?\n> \n> OCI - Oracle Cloud Infrastructure.\n> Mgmt needs to manage reconnections for live migration, and perhaps I could\n> leverage that code for live update, but happily I did not need to. Regardless,\n> reconnection is the lesser issue. The bigger issue is resource management and\n> the container environment. But I cannot justify that statement in detail without\n> actually trying to implement cpr-transfer in OCI.\n\nI see. Is OCI open source somewhere?\n\nIf it's close-sourced, maybe it'll be helpful to see how the exec() design\ncould benefit other open source mgmt applications.\n\n> \n> > > Both modes are valid and useful - exec in container, or launch a new container.\n> > > I have volunteered to implement the cpr-transfer mode for the latter, a mode\n> > > I do not use. Please don't reward me by dropping the mode I care about :)\n> > > Both modes can co-exist. The presence of the cpr-exec specific code in qemu\n> > > will not hinder future live migration development.\n> > \n> > I'm trying to remove some of my \"prejudices\" on exec() :). Hopefully that\n> > proved more or less that I simply wanted to be fair on making a design\n> > decision. I don't think I have a strong opinion, but it looks to me not\n> > ideal to merge two solutions if both modes share the use case.\n> > \n> > Or if you think both modes should service different purpose, we might\n> > consider both, but that needs to be justified - IOW, we shouldn't merge\n> > anything that will never be used.\n> \n> The use case is the same for both modes, but they are simply different\n> transport methods for moving descriptors from old QEMU to new. The developer\n> of the mgmt agent should be allowed to choose.\n\nIt's out of my capability to review the mgmt impact on this one. This is\nall based on the idea that I think most mgmt apps supports reconnections\npretty well. If that's the case, I'd definitely go for the transfer mode.\nI'm not sure whether there's anyone from mgmt layer would like to share\nsome opinion; Dan could be the most suitable in the loop already.\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=TIr/ou81;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wk2dl1Tqyz1yYC\n\tfor ; Wed, 14 Aug 2024 06:13:43 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sdxsd-00069d-6u; Tue, 13 Aug 2024 16:12:31 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sdxsb-00068d-HX\n for qemu-devel@nongnu.org; Tue, 13 Aug 2024 16:12:29 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sdxsY-00052N-1N\n for qemu-devel@nongnu.org; Tue, 13 Aug 2024 16:12:28 -0400","from mail-qt1-f199.google.com (mail-qt1-f199.google.com\n [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-324-6TLoYGHBN5GAIsDp470E4Q-1; Tue, 13 Aug 2024 16:12:20 -0400","by mail-qt1-f199.google.com with SMTP id\n d75a77b69052e-44fea369811so8679491cf.0\n for ; Tue, 13 Aug 2024 13:12:20 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n 6a1803df08f44-6bd82e50f99sm36381556d6.104.2024.08.13.13.12.18\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Tue, 13 Aug 2024 13:12:18 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723579942;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=mR9lt0vW9gFFYO/c/SIvIOsr9Zo+gGFWHZcfLefsliQ=;\n b=TIr/ou81w8ReDMIG/0ynxjJtnYw7LGwcw/0kDUTKLchw1mpUr+gSmbKlvLvMAMMbgiV64G\n cQ6eOspaUVo2lcJWGUdQVgef9sZispbKcT/j/XQTU6cRo0T7Lq89rPmoX6rQ2MPb1paXo2\n 168/5CdW5YZlk8Ab7qJNP9sET3S3tOY=","X-MC-Unique":"6TLoYGHBN5GAIsDp470E4Q-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1723579940; x=1724184740;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=mR9lt0vW9gFFYO/c/SIvIOsr9Zo+gGFWHZcfLefsliQ=;\n b=CwyYuhXuJ/28sxsf+YCuplra5h+yE4J88k/8l5i7aX7wzfsdktPrYUjJpLBhTDCkoF\n yKFckG5wOqv3jyPI06C5p+lKCueEPfOJVARpJ4PuM5bZ8LpNzPrJGPtnRKjXf5qjXB3o\n Ei6lgsjJ1D5QzNgyZI+H5Hb5Y+LW+wS4Se7uqiHBApkUMXlr4fSF5Dr6oeSBoUbn+oI6\n L3u4Tj9un5VEwnaKbh1rB4IsuWb5dfJPF2tGhRQZSMxBRCt2C/9iTlk6MRC0353//BzR\n 4Gq8U6u9t84Wh7YlSrQgKUmYEuOT41DbGAJZovg/3/47U+AAmpMI9HKoyxy1vK5xQN3/\n LHoA==","X-Gm-Message-State":"AOJu0YzX+aSNEPFBeSULds7KiuK3ixnqw1j92ZvBDXQtcRXCb1Kuw1bD\n 60bfmilqLX57VBtligA8U1PUl8OrnooOVz1LQnPkI9rsN5FI80olp5EFA3MCl+MMzj8Q+vlR7ef\n RY6peFZ/Y4H/Tycu4luYcTpWhDvMNO/CsT/vuluPSQ4FJjb9cLnlT","X-Received":["by 2002:a05:6214:f2f:b0:6b5:ec9a:41ef with SMTP id\n 6a1803df08f44-6bf5d16148cmr4630896d6.2.1723579940181;\n Tue, 13 Aug 2024 13:12:20 -0700 (PDT)","by 2002:a05:6214:f2f:b0:6b5:ec9a:41ef with SMTP id\n 6a1803df08f44-6bf5d16148cmr4630626d6.2.1723579939519;\n Tue, 13 Aug 2024 13:12:19 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IG/BNfE4XpJWv/9Rmjfy9gmf1Dqo0XLzgVOfmuyB37D43EBSuyaRgBzPpBPS0J/vIgncdZH+g==","Date":"Tue, 13 Aug 2024 16:12:16 -0400","From":"Peter Xu ","To":"Steven Sistare ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini ,\n \"Daniel P. Berrange\" ,\n Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>\n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.125,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363028,"web_url":"http://patchwork.ozlabs.org/comment/3363028/","msgid":"<46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>","list_archive_url":null,"date":"2024-08-15T20:55:20","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/13/2024 3:46 PM, Peter Xu wrote:\n> On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n>>> The flipside, however, is that localhost migration via 2 separate QEMU\n>>> processes has issues where both QEMUs want to be opening the very same\n>>> file, and only 1 of them can ever have them open.\n> \n> I thought we used to have similar issue with block devices, but I assume\n> it's solved for years (and whoever owns it will take proper file lock,\n> IIRC, and QEMU migration should properly serialize the time window on who's\n> going to take the file lock).\n> \n> Maybe this is about something else?\n\nI don't have an example where this fails.\n\nI can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\nthe same block device, but the error is suppressed if you add the -incoming\nargument, due to this code:\n\n blk_attach_dev()\n if (runstate_check(RUN_STATE_INMIGRATE))\n blk->disable_perm = true;\n\n>> Indeed, and \"files\" includes unix domain sockets. \n\nMore on this -- the second qemu to bind a unix domain socket for listening\nwins, and the first qemu loses it (because second qemu unlinks and recreates\nthe socket path before binding on the assumption that it is stale).\n\nOne must use a different name for the socket for second qemu, and clients\nthat wish to connect must be aware of the new port.\n\n>> Network ports also conflict.\n>> cpr-exec avoids such problems, and is one of the advantages of the method that\n>> I forgot to promote.\n> \n> I was thinking that's fine, as the host ports should be the backend of the\n> VM ports only anyway so they don't need to be identical on both sides?\n> \n> IOW, my understanding is it's the guest IP/ports/... which should still be\n> stable across migrations, where the host ports can be different as long as\n> the host ports can forward guest port messages correctly?\n\nYes, one must use a different host port number for the second qemu, and clients\nthat wish to connect must be aware of the new port.\n\nThat is my point -- cpr-transfer requires fiddling with such things.\ncpr-exec does not.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=e3q7wbDX;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=ot2epvWq;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WlHTb3rXvz1yfB\n\tfor ; Fri, 16 Aug 2024 06:56:03 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sehVU-00062n-0X; Thu, 15 Aug 2024 16:55:40 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sehVP-00061s-QB\n for qemu-devel@nongnu.org; Thu, 15 Aug 2024 16:55:36 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sehVN-0005c1-EK\n for qemu-devel@nongnu.org; Thu, 15 Aug 2024 16:55:35 -0400","from pps.filterd (m0246629.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 47FJteOv027220;\n Thu, 15 Aug 2024 20:55:29 GMT","from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta02.appoci.oracle.com [147.154.18.20])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40x039b7bw-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Thu, 15 Aug 2024 20:55:29 +0000 (GMT)","from pps.filterd\n (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 47FJ0wSf020583; Thu, 15 Aug 2024 20:55:27 GMT","from nam04-bn8-obe.outbound.protection.outlook.com\n (mail-bn8nam04lp2040.outbound.protection.outlook.com [104.47.74.40])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id\n 40wxnj4fuk-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Thu, 15 Aug 2024 20:55:27 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by DS0PR10MB6126.namprd10.prod.outlook.com (2603:10b6:8:c6::20) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.16; Thu, 15 Aug\n 2024 20:55:25 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7875.016; Thu, 15 Aug 2024\n 20:55:25 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=ciCon4RrkGi764WBLcIOWxNLb4Fec4wDJzFO+MijzIw=; b=\n e3q7wbDXGMI/bWXOqxd4Gg104luZjHeUPH7Yuw5CK2BTXZDgDu0JkccPDHuAAktF\n S1l6NlNbTZx/oE5aeC4OhxRcB8gapqsTxjT3gitJ6zmYVvHVFR8leqsFJilX4OlP\n RcTPgXU1Grg+IahxLjkARwvPAP3/kQViCrY17R34+0KIZe8wPShXX3AGzcID3XTp\n RugZxKnQHuqoNawkrCAQtEIJydRx6IT25Z7LWNtUuWGzcdUHpBIe3r12/ZS0Tfam\n 1KW+JurKc0f/GVx0XXVX/i3SyHe2sERZ6shG0CJvAZlDulKVYU9mD+x8PERSWjX1\n gDru8GrXczlZiYLQjd+brw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=ciCon4RrkGi764WBLcIOWxNLb4Fec4wDJzFO+MijzIw=;\n b=ot2epvWqrBZC9JTFKvCkSSD2KWXQzWG8ozp9t3ZPycKGIBVfUTf7h8Xpjt9Q/ZwW0EgGcDjzqPcFtKbPhFOSZ9j+cF6cp3XBiu1/OQ22b3pZ3UL98VnTZOU8j77m32GJgOuSRvXQ1DYnKedCsCW2WyowUM9Ad7t3ILh6JPf47Ew="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=abZEYfpUUlVvfBufWj5/ZjG/cjT8FrYmf7PbnXMmUvyTX2gq54H4ZMxt4wxJdk4Y+BX4WfN58K0JC/xGmmjQhy3o4GUeOXdF/dMhx19G0U8gvQQxxH16PP/ubnfeFAOP5vAVfGYLxh1yS4jTzpQQuMtI1P3FHXmiKGA/Wy0uNtlFpU9SfF75KzgUASSdDacbRAPSWEltER3Xd5ihUEGa0k8mQD9eFiUmRRrNG3E9EnhETTKEsqoeX2JcKZASVdQOI4r9e2qE7D/2rcuByaJYystLYzMV541ZrljGSB3+kWX/RFvYSr+E08U8pBWxfjG2V8g6ZJDzPv3JwTySS39nXA==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=ciCon4RrkGi764WBLcIOWxNLb4Fec4wDJzFO+MijzIw=;\n b=joBfLp/cqRau+/hN6VbXWZDdC5nSR1+wHJC/KpuUzOmujOXXxE+DMgRTyR1O1AeYWAqI2hQZCNN3/03b1tlpUukp05CpimtV3v45eH2oUSM5NNsEP7cEByUR3UV9akVmXKbSzsfCWpyQ7sL+IoKKvMoIIqxxCteyKxGvVKgJ/7pHbQ5h1HLxROeEtQA1XS4MH1AmpExqlq2m7FjPveiA3nwkT3pLFxMUuzrmyTuVnLJZVx82jC0hWb8jjQfcelkc8gQz+haqAKylxqtShThRlf7SrknV+C+gml7DJvdyoVIWYoy6qy4l9LILKP5K1JnVI4P16aXYgM9DnI6X/U7jig==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"<46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>","Date":"Thu, 15 Aug 2024 16:55:20 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ","Cc":"=?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= ,\n qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com> ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-ClientProxiedBy":"BYAPR07CA0029.namprd07.prod.outlook.com\n (2603:10b6:a02:bc::42) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|DS0PR10MB6126:EE_","X-MS-Office365-Filtering-Correlation-Id":"648b64cc-4952-4eae-a5ba-08dcbd6c9621","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|366016|7416014|376014|1800799024;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?3THzQQ24+SoW+abm7WmagrGPGEq1Jg5?=\n\t=?utf-8?q?NqjIVhCDKtKyxwiBFqIG7gOwjbfbA9m5hNuToLwSU9sAo0znCPjZ7TqnI0xW7NEj9?=\n\t=?utf-8?q?XihKf+ZM/77Kp3LFDJM9dkg5JkGj9/RyelPzgcLtPWWMq3zYlsotbHz/1jmLfyIns?=\n\t=?utf-8?q?oMm2FAOve9XXPQPGwTkR0iP0S+Zek4XJzcBitKI5ON2vlgImsbZ2lnL7Ob77Y0C6+?=\n\t=?utf-8?q?l2TUSTmAxMj49fB3tgpGr6tZYVn3AFNsRVpSwsakb3npK07/4HVhOChCfxbEX0YjQ?=\n\t=?utf-8?q?wacGZnXtAJV+9bYTwiYGoDhnWLrgJpxNZqFPr+KVV8Vz48jZuuadotuw6ew+z1wHW?=\n\t=?utf-8?q?kh3BnRze0SDlAjphgp2121rKYrYw2IzWGYun+4MdC9U8LWnr46SccChZ2dlCxSghr?=\n\t=?utf-8?q?40m3yJfWwq/elqWzvQrzpaSU552L3PRe7Gu/mTbN9Y3UsQ7h2OlL9EdnJPRk0V+dk?=\n\t=?utf-8?q?T8OyeoR8PWbBDsSMuHENILWYPOdOZ21j7BG9eubownSrPqKzcUjDZq3PCgLZKVMM+?=\n\t=?utf-8?q?mk+G2MsEBG+wqnY2N22+2ZyHJof5OI9wFa8acIhXr7PQ+bovayagoo9dJCdilmCbA?=\n\t=?utf-8?q?cYKRr58iGUgdXIgRKBsTBhql+/kbYJbIy7YRvmjquB9nGMxoXqvGBR6POrZEo5xMz?=\n\t=?utf-8?q?3c0nxFIISJ3UB7tILS4n0WBJE5lpalY1Hhn7GFfGeRlRueAPOM+PRG6a/AuQNSo8X?=\n\t=?utf-8?q?UGbcgN0sgF9mPBMt7BdtnU3fI30u5bqEhtFvFE5qmGmUPmni/TUzGx6iPszTc7mTT?=\n\t=?utf-8?q?6N/ZQiamdOs6bGCHW1rn8pj2i6D0Rw/IDgmawu1k5lsLwZL5ETIwd2IzFRjcmEcOR?=\n\t=?utf-8?q?VB2y5P4I1a0g+qPqG0OoG60anb2G+EWmzrSqE/JJX5CzZZHomMTYA5LzRCVwy8vXf?=\n\t=?utf-8?q?kOqLHaZuyuUMUpY4AhPgZTs9dyMPknJlwxe5cKb0ntyYsNwivp0Qf1KEKTM9UssN/?=\n\t=?utf-8?q?ofOYr7O9CVlildRh0qYMmOF2YTAmshQeGQTUXzGns83aqLPlII1N4jdI98lcfs+GQ?=\n\t=?utf-8?q?7G34qKCkp4r0xrXb60ph1RyrRQBjR7GNkmjP28RyQrrEyKiO+3tIOt4Evq8pnsYKS?=\n\t=?utf-8?q?O+uyNWi21FyOhijCCMUDNh3TCXKjH5EzGhRWnV5o+piwoCHF6s/yqjwd0LaG8XuCz?=\n\t=?utf-8?q?25fAiFu/uiRgaWifapi03lj08CJX32cjHK4Df6CfYpEsvCQoO+31gAOeM1JjJAecO?=\n\t=?utf-8?q?7h2xfVRMp1rekB8OT//uZ21hOz0GF1xyTp+82KPPPWW0F5nHIiAiivTYQcxRkrLnO?=\n\t=?utf-8?q?fG2Zn/NeZvsYDNYwxR1jkK0MNdnZeR6erXQ=3D=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(7416014)(376014)(1800799024); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?eXT6VusTiikk0RJPQRMvq8fOq14v?=\n\t=?utf-8?q?Cdc6etoBCbE3+3rZI/R3TOYrKDSG5CPUN+EHICSpg7XQU+pYpO4v4eI4DSkb8EHp+?=\n\t=?utf-8?q?NOz8Tql6SLqJjH4xcHeoGOKYFm94hwjoRkR8rZ02lgGF5ghgIpGpFFRBeFcDMkU0G?=\n\t=?utf-8?q?qgjIxDzwwVBzUVqsceoTUn5X77oZrryAr9vBbUEweJAQILAW/0gEL0I5XHCVnKGxx?=\n\t=?utf-8?q?yp1pAIYVhwR67vAlYhkDxF4Q57eIyoUr3YK4BQNL1UcZ32fGRd0h+CCuFWM8PDPtp?=\n\t=?utf-8?q?9LErmXatm9A3hgTRXCPsswL6Fi3Ch9GfFO1ct20hpdeqYEa/yjcfDyZ43cdixCO6y?=\n\t=?utf-8?q?rfawr06R/sIeRunvh7ZWMZUXzaCtIHUMKoBGQG79xiV6r+ApTeehh+HvGFVmfhGAB?=\n\t=?utf-8?q?F7PtYOxQ2ru6M2L6TEHoN8myf5RprOTMo5ozmusBszQ7Dx4s8l7mey3UX0LHHUuK1?=\n\t=?utf-8?q?XFG182zJv2QgwnkUR6p2ZcOnJ0xTg/g1qnTKzgtu6h3lEhW9ebnIaiQGwIDnjwTWT?=\n\t=?utf-8?q?EnO/3CJxnO25I6s9CaLqTcUjGQBh9dub5M3wE0lB1P4wPHfRQO/TWo/+diePMCpsg?=\n\t=?utf-8?q?MSxrzxxXc0CcRFWWBIpvS7zelqFARgIeAXl2XBzMK1OZAQkr9IKmLtrnQTfwkB7iA?=\n\t=?utf-8?q?dyg04V3w1DufW6ocjYvZIyk33eOYL0fQs5rebwGEIfmBY3iN2yx5ZR+FzyIhlV5Qa?=\n\t=?utf-8?q?fbdmB/d0+DLkLMKT9Vscsic/YGit6MRAItlUsAlvnN0Z0WGQ/NlXZrXzc4IvRskTZ?=\n\t=?utf-8?q?dGXGdKlhS8VsvPumd3ddbzsRFg0z52cTT8rQBzcAD1FsEY5rKWbywjyfcFlEeY+PS?=\n\t=?utf-8?q?xOvjthePIPJImv6JR7EU7iTEl7uR992EWBp832eXStHsQk/7l35Znd442LyVxsC5q?=\n\t=?utf-8?q?E0nBvUrftKgtgvNOBxcYhRZRvzQ/MC/65ClTt40y0SVH9XVc+2EaYGtWBqnMeDIz4?=\n\t=?utf-8?q?fhsu8WQlyon+nko6ZPO2X5VbvBvRIxCNdRF+B4IawhrDLEajuGJVVcoqc5P9/H9LL?=\n\t=?utf-8?q?t/TxIH22NjdXpDedi6AEbnIbTE0xrZDwzILAjphYxRp3A6+YKnyebc+P5XNxx/S/X?=\n\t=?utf-8?q?jnPU/S0Xrt15vNjF36Qjd8Q+CBIZpGGDPZGHBv8ahd0eGFpuUhO/ciIOrSSnwM1Z6?=\n\t=?utf-8?q?FU83vABGvaG16vURrCcuk/bRUcPvGCitd8aSmk2HEoKbYEp0bOC0ivQ+PcdLI+cBg?=\n\t=?utf-8?q?2iRihlZkPBzY/DK/TmgZpvuxMeuezijA/uArK2l/g7CQHcSkdYs6P3JIh7D5CJ2io?=\n\t=?utf-8?q?OadfeApCKf473kpwi2UzMXxvunbCPVHAoST7RUcZ48xzheBK+dSXBMeC/aYz4k/7O?=\n\t=?utf-8?q?LVuz4tgXZzvxnNiNiBpE6P9wzqX/pWZlHuvjIP/GM17x90UBIIRkPILFmca5UX36W?=\n\t=?utf-8?q?XGO/lZQyI3XAYS30ni1CWNMHqWJXKql/Q0vK8+ILlJEJ0wLqZwzsozwpbNFTS+X3u?=\n\t=?utf-8?q?wYTU0uAkpEgUybJBc6n+0fPsSE228IQngtvkS4ByhwYf2YXAAY5qPQXsmdY9S0WsO?=\n\t=?utf-8?q?FAFmXZt58TyvTheLZKuJaIP+vFgrN+0Bmw=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n rDdkAXokJkxEyOYobHAume7eQUJfq+QPiZ5dy9rKq0h3ZgsInpYbfr4vp40dXJMdzEe/ziow3hIiiLihXSXK4JLNcsqfCa10WYl+0nXn08agRSetoHNjGOgbV5Vb4Wde3W3PXZLcRSKPZDfebzcW385HWfVgYrVpJx2bdEiR6/QhZulF46JcDuECMtYT2uBUGoGzA1PkuYgEirdOnAvjQOApGYZH2NenHxlOkh1Pfymvy0GdhGlAT5aE4K4XLQlsSwxo5IxMgfZn5Hkl6vbMK7w43QyjhSG5wCSxyWHpQqRDyDUykH8dM5461IUwwJikOEUvEN8sgrcwPQ48BMeu4drJmXfESZrhMd4Fz5btd3gVFXHw0gZqHbJoUH0mlmsIpp2wKjgcXka3z2g8o60Gwy7FpmNT1DVsioLs8j1jbm9GC8YAn7TkfRlrcODg9n08YBN+4ShNXG3IKEVqQEmjgT2feh/7ZtO6deyYXijE6VeApZVD+EekDTYjkYUYigSJYbKajfZ07MNSrpls9FOt4mbJaEJUyIdVT8VBqKEeVC1c2lA06FoHEIK0FL4Df27WQICAMhtGo/h8tgBNG8m+M/Gu2xRVu06gtPE1Z1+ntbc=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 648b64cc-4952-4eae-a5ba-08dcbd6c9621","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"15 Aug 2024 20:55:25.4159 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n qSSeFBdDaBV/5sPvV8eNQRIrcYfG9kD3cnX9PW8/rcWP0BjIq63MENt0zh2GkDUi3vF0jlu/cjAhynvm0d7RjUQz8yQoxuXbWDyik1KCGDA=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"DS0PR10MB6126","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-15_12,2024-08-15_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 adultscore=0\n mlxscore=0 bulkscore=0\n phishscore=0 malwarescore=0 suspectscore=0 spamscore=0 mlxlogscore=999\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2408150152","X-Proofpoint-ORIG-GUID":"Q0lBIFUIR6DOadabUiORv8gVzuQWpQ1R","X-Proofpoint-GUID":"Q0lBIFUIR6DOadabUiORv8gVzuQWpQ1R","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363485,"web_url":"http://patchwork.ozlabs.org/comment/3363485/","msgid":"","list_archive_url":null,"date":"2024-08-16T15:06:10","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > processes has issues where both QEMUs want to be opening the very same\n> > > > file, and only 1 of them can ever have them open.\n> > \n> > I thought we used to have similar issue with block devices, but I assume\n> > it's solved for years (and whoever owns it will take proper file lock,\n> > IIRC, and QEMU migration should properly serialize the time window on who's\n> > going to take the file lock).\n> > \n> > Maybe this is about something else?\n> \n> I don't have an example where this fails.\n> \n> I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> the same block device, but the error is suppressed if you add the -incoming\n> argument, due to this code:\n> \n> blk_attach_dev()\n> if (runstate_check(RUN_STATE_INMIGRATE))\n> blk->disable_perm = true;\n\nYep, this one is pretty much expected.\n\n> \n> > > Indeed, and \"files\" includes unix domain sockets.\n> \n> More on this -- the second qemu to bind a unix domain socket for listening\n> wins, and the first qemu loses it (because second qemu unlinks and recreates\n> the socket path before binding on the assumption that it is stale).\n> \n> One must use a different name for the socket for second qemu, and clients\n> that wish to connect must be aware of the new port.\n> \n> > > Network ports also conflict.\n> > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > I forgot to promote.\n> > \n> > I was thinking that's fine, as the host ports should be the backend of the\n> > VM ports only anyway so they don't need to be identical on both sides?\n> > \n> > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > stable across migrations, where the host ports can be different as long as\n> > the host ports can forward guest port messages correctly?\n> \n> Yes, one must use a different host port number for the second qemu, and clients\n> that wish to connect must be aware of the new port.\n> \n> That is my point -- cpr-transfer requires fiddling with such things.\n> cpr-exec does not.\n\nRight, and my understanding is all these facilities are already there, so\nno new code should be needed on reconnect issues if to support cpr-transfer\nin Libvirt or similar management layers that supports migrations.\n\nI suppose that's also why I'm slightly confused on how cpr-exec can provide\nbenefit for mgmt layers yet so far with these open projects. It might\naffect Oracle's mgmt layers, but again I'm curious why Oracle does not\nsupport these, because if that should support normal live migration, I\nthought it should be needed to support changed ports on host etc..","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=NnSXiZU4;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wllhq4LGJz1yYl\n\tfor ; Sat, 17 Aug 2024 01:07:23 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1seyX4-0006ae-Fa; Fri, 16 Aug 2024 11:06:26 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1seyWz-0006Yu-7t\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:06:21 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1seyWu-0006NE-Tj\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:06:19 -0400","from mail-qv1-f70.google.com (mail-qv1-f70.google.com\n [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-433-MvUewqN5O6ubNlmLUQmqow-1; Fri, 16 Aug 2024 11:06:13 -0400","by mail-qv1-f70.google.com with SMTP id\n 6a1803df08f44-6bf6f1ad75cso4441776d6.3\n for ; Fri, 16 Aug 2024 08:06:13 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n d75a77b69052e-4536a072683sm16936571cf.85.2024.08.16.08.06.10\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Fri, 16 Aug 2024 08:06:11 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723820774;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=E3KC6rF1jt182BaYrTc0ZcHki6d26eYQ0/S1Xo0Wfdk=;\n b=NnSXiZU4m1qPM982yNrxBAqCHuKkO3dTG2oxOB+CVz3PkoGZNXtm5TpWXbCNVw0mlp4HrA\n JSTEWn3Q++tP/fq83IGp3LYmj0AyMJnyYMgPGgdAfog5TOCbr0L7QPnT6yvG5S6CZ6FJqR\n TtZN62unTnkIvGOQk3f37c7n1rfIFTw=","X-MC-Unique":"MvUewqN5O6ubNlmLUQmqow-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1723820773; x=1724425573;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=E3KC6rF1jt182BaYrTc0ZcHki6d26eYQ0/S1Xo0Wfdk=;\n b=meNI/nQrB+M0eDICm9MSXOkvahko/T5xSoXCG1ohNgatA9TcjbVoIMKNt6WlPgvDje\n q57L3Z0zp1t50v5EAQaNcwCw9Gb2Nwm52YtsR2gciuPl7Jb6m89dT/RRUyCM51JtiF5d\n qPkPIbkuxfOe9DJC7dBNCvo5fiaO8ueYaTA92Zju3pTABO/QSF4s/dePK7JkcBT/LqnL\n rcDB9T95ZHxN9wSs47I/C8yjjAOefqNYMO8yD59PkOLaQzc8zYZqAiVKnDu+zrzxz+h/\n wmI63OkiNRrUHN1Vazw4sdk2IVZQHl33nwrfjfN+jIEAkVNr18KFn1k6uGNzuZ3Aty+h\n g3QA==","X-Forwarded-Encrypted":"i=1;\n AJvYcCW7raP059l4AKPnxB3l6Sfd5saubgGniNUcfkXMZp69i0iuhLawathsnhjLN75LgLAj+32Y2qcf3cq8@nongnu.org","X-Gm-Message-State":"AOJu0Yx4RX9jrnIwa+cRyxou8RLU06QLBZZma6uTsMGFZjJYHpi87oEj\n OkZRU9sXrygsoLS3Sdp1fcWstg9Hy/l0Z81ZzJY4q+/WpvMna3fp4y/11I2ihxvDrl/7wbIfnf1\n qRz7gFP5iW1iqHLKCImK2UgYBaaxtZ7ZIQLt8o99VUTW8mVyazh9R","X-Received":["by 2002:a05:622a:2c9:b0:44f:d7b5:a665 with SMTP id\n d75a77b69052e-4537419cc08mr19736501cf.1.1723820772791;\n Fri, 16 Aug 2024 08:06:12 -0700 (PDT)","by 2002:a05:622a:2c9:b0:44f:d7b5:a665 with SMTP id\n d75a77b69052e-4537419cc08mr19736241cf.1.1723820772285;\n Fri, 16 Aug 2024 08:06:12 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IF+wycdqYUXMU2ZCYxW6FL4s6cbJpJmuiamKptDzS0VHI2F1iWFmUZvbq99xsPI9oHSlJ0QkQ==","Date":"Fri, 16 Aug 2024 11:06:10 -0400","From":"Peter Xu ","To":"Steven Sistare ","Cc":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ,\n qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363494,"web_url":"http://patchwork.ozlabs.org/comment/3363494/","msgid":"","list_archive_url":null,"date":"2024-08-16T15:16:50","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > file, and only 1 of them can ever have them open.\n> > > \n> > > I thought we used to have similar issue with block devices, but I assume\n> > > it's solved for years (and whoever owns it will take proper file lock,\n> > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > going to take the file lock).\n> > > \n> > > Maybe this is about something else?\n> > \n> > I don't have an example where this fails.\n> > \n> > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > the same block device, but the error is suppressed if you add the -incoming\n> > argument, due to this code:\n> > \n> > blk_attach_dev()\n> > if (runstate_check(RUN_STATE_INMIGRATE))\n> > blk->disable_perm = true;\n> \n> Yep, this one is pretty much expected.\n> \n> > \n> > > > Indeed, and \"files\" includes unix domain sockets.\n> > \n> > More on this -- the second qemu to bind a unix domain socket for listening\n> > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > the socket path before binding on the assumption that it is stale).\n> > \n> > One must use a different name for the socket for second qemu, and clients\n> > that wish to connect must be aware of the new port.\n> > \n> > > > Network ports also conflict.\n> > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > I forgot to promote.\n> > > \n> > > I was thinking that's fine, as the host ports should be the backend of the\n> > > VM ports only anyway so they don't need to be identical on both sides?\n> > > \n> > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > stable across migrations, where the host ports can be different as long as\n> > > the host ports can forward guest port messages correctly?\n> > \n> > Yes, one must use a different host port number for the second qemu, and clients\n> > that wish to connect must be aware of the new port.\n> > \n> > That is my point -- cpr-transfer requires fiddling with such things.\n> > cpr-exec does not.\n> \n> Right, and my understanding is all these facilities are already there, so\n> no new code should be needed on reconnect issues if to support cpr-transfer\n> in Libvirt or similar management layers that supports migrations.\n\nNote Libvirt explicitly blocks localhost migration today because\nsolving all these clashing resource problems is a huge can of worms\nand it can't be made invisible to the user of libvirt in any practical\nway.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=iTrjkBBp;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WllwS1cmgz1yfB\n\tfor ; Sat, 17 Aug 2024 01:17:28 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1seyhP-0007kB-Pk; Fri, 16 Aug 2024 11:17:07 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1seyhO-0007ek-99\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:17:06 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1seyhM-0007yQ-AD\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:17:06 -0400","from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-671-XNZYqBDbMdCelGLaJd8Z8A-1; Fri,\n 16 Aug 2024 11:16:59 -0400","from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id DAF731955BF9; Fri, 16 Aug 2024 15:16:57 +0000 (UTC)","from redhat.com (unknown [10.42.28.143])\n by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id 95C3B19560A3; Fri, 16 Aug 2024 15:16:54 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723821422;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:in-reply-to:in-reply-to: references:references;\n bh=h268iraODaINcGNSXqFo17MOMSjBV5lzLDaKD99mPpI=;\n b=iTrjkBBpixlWcMNhz0zGuCqGqHx7uDfzNBorxDB5V2R6Id/VQkHOrQytgaMY2Y9kswMMpd\n Ofi242N3X8q6T8ejqaYX/QpO354/GtSQGrr+PijoXBlgcANPPYxn3B1bQ9+kU5mhp1gPUD\n z5EXZBxaBbNJdYV6TJZEfbnrnfnmBz8=","X-MC-Unique":"XNZYqBDbMdCelGLaJd8Z8A-1","Date":"Fri, 16 Aug 2024 16:16:50 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Peter Xu ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.40","Received-SPF":"pass client-ip=170.10.129.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363495,"web_url":"http://patchwork.ozlabs.org/comment/3363495/","msgid":"<588cd3f3-629a-451e-bc21-f255b5e0690c@oracle.com>","list_archive_url":null,"date":"2024-08-16T15:19:46","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/16/2024 11:16 AM, Daniel P. Berrangé wrote:\n> On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n>> On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n>>> On 8/13/2024 3:46 PM, Peter Xu wrote:\n>>>> On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n>>>>>> The flipside, however, is that localhost migration via 2 separate QEMU\n>>>>>> processes has issues where both QEMUs want to be opening the very same\n>>>>>> file, and only 1 of them can ever have them open.\n>>>>\n>>>> I thought we used to have similar issue with block devices, but I assume\n>>>> it's solved for years (and whoever owns it will take proper file lock,\n>>>> IIRC, and QEMU migration should properly serialize the time window on who's\n>>>> going to take the file lock).\n>>>>\n>>>> Maybe this is about something else?\n>>>\n>>> I don't have an example where this fails.\n>>>\n>>> I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n>>> the same block device, but the error is suppressed if you add the -incoming\n>>> argument, due to this code:\n>>>\n>>> blk_attach_dev()\n>>> if (runstate_check(RUN_STATE_INMIGRATE))\n>>> blk->disable_perm = true;\n>>\n>> Yep, this one is pretty much expected.\n>>\n>>>\n>>>>> Indeed, and \"files\" includes unix domain sockets.\n>>>\n>>> More on this -- the second qemu to bind a unix domain socket for listening\n>>> wins, and the first qemu loses it (because second qemu unlinks and recreates\n>>> the socket path before binding on the assumption that it is stale).\n>>>\n>>> One must use a different name for the socket for second qemu, and clients\n>>> that wish to connect must be aware of the new port.\n>>>\n>>>>> Network ports also conflict.\n>>>>> cpr-exec avoids such problems, and is one of the advantages of the method that\n>>>>> I forgot to promote.\n>>>>\n>>>> I was thinking that's fine, as the host ports should be the backend of the\n>>>> VM ports only anyway so they don't need to be identical on both sides?\n>>>>\n>>>> IOW, my understanding is it's the guest IP/ports/... which should still be\n>>>> stable across migrations, where the host ports can be different as long as\n>>>> the host ports can forward guest port messages correctly?\n>>>\n>>> Yes, one must use a different host port number for the second qemu, and clients\n>>> that wish to connect must be aware of the new port.\n>>>\n>>> That is my point -- cpr-transfer requires fiddling with such things.\n>>> cpr-exec does not.\n>>\n>> Right, and my understanding is all these facilities are already there, so\n>> no new code should be needed on reconnect issues if to support cpr-transfer\n>> in Libvirt or similar management layers that supports migrations.\n> \n> Note Libvirt explicitly blocks localhost migration today because\n> solving all these clashing resource problems is a huge can of worms\n> and it can't be made invisible to the user of libvirt in any practical\n> way.\n\nThank you! This is what I suspected but could not prove due to my lack of\nexperience with libvirt.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=j+M8NMCd;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=fGiNkxsk;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wllzr4Sr5z1yYl\n\tfor ; Sat, 17 Aug 2024 01:20:24 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1seykC-0000W5-MP; Fri, 16 Aug 2024 11:20:00 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1seykB-0000Vb-2E\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:19:59 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1seyk8-00088j-T4\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:19:58 -0400","from pps.filterd (m0246629.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 47GC8tEN020972;\n Fri, 16 Aug 2024 15:19:52 GMT","from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com\n (phxpaimrmta01.appoci.oracle.com [138.1.114.2])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40x039cj5n-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Fri, 16 Aug 2024 15:19:52 +0000 (GMT)","from pps.filterd\n (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 47GEU5sU000759; Fri, 16 Aug 2024 15:19:51 GMT","from nam02-dm3-obe.outbound.protection.outlook.com\n (mail-dm3nam02lp2049.outbound.protection.outlook.com [104.47.56.49])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id\n 40wxncdend-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Fri, 16 Aug 2024 15:19:51 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by IA1PR10MB7388.namprd10.prod.outlook.com (2603:10b6:208:42c::15)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.10; Fri, 16 Aug\n 2024 15:19:48 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7875.016; Fri, 16 Aug 2024\n 15:19:48 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=v0aIuRIdrIN/n2ZPhfgkcgX2KxYbZ0HPMn7jMTT6ocw=; b=\n j+M8NMCdNWhO168cWuyRXjLoiSiF5mK+DwS/MZU1h4QQKr72SPCXpuOnM9B1d5Ma\n dbHanM+snJv5sy95QlLCQGsJ0VSdsbZKuJT/wqn6wfEYcp7BX6lxFrdIw76UTNJd\n M1/aIAfOJsoXpasMWNgEheyPPUGkpm0bqoFqhXYiVCT9B+7NWx745sHhc9SrEpaU\n Y+K0TdDu20qxBIHQzxTwxoISba951JbXRJkuDF9GSj+KaG5s3yrLIvygDilXmVzy\n HbMfNZH9IeH1w7nDekntS2KLTHny/rWBBtwVjV/3vPiKnl+0KU85ezdsFZTQ8DXN\n 8eTeYuJNsiyYDSn+TJEfNg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=v0aIuRIdrIN/n2ZPhfgkcgX2KxYbZ0HPMn7jMTT6ocw=;\n b=fGiNkxskMFBM0dzj59gMr4Z9bo8dIrXKxWFVoaUEWGrmNfSVNRdF8xEPpddVh3Y90jz3qX0frhaj1+K3AHcXa8du9IIfDn6dmfcKLtDESD9m/e7NPeOkr/okksQL5hNQY7hjXoYkAp55e11TTonjjphK2hanpXCkH4eRk1FKp4s="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=Y3SGPfIGqUo+JcB5WfFQpi3uBEiQspSIOVVBjVU55uB6QrtXkrRzabaQscKSXMMKt8/NDbmtSjUgPBBmofZKEIH7euSxd/BdHKXRCbD9acnG9wA6foLuFBLfh7CNn9Ro5hGoWckR3bOBPPcTQs9Ye1no4J0pOqhY0sq/iimM616U+MxfdjEg0Tv5Ry8fa+0vg7HSDjjLvzTymF/Yxkw10rtEl+wmrpyHFIxeUsjmHKiB17y8tzMryO9K7QdpT+rkaz+u0u+eHbuRX1xVODioTst5YOISxfpdvHyE+irg51WV4Hxi6sKY5OLBIkMeeJjY5om45d75CE73ppr+064MgA==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=v0aIuRIdrIN/n2ZPhfgkcgX2KxYbZ0HPMn7jMTT6ocw=;\n b=l64Shlhbj3mnu6E14DKPcQedPhSDs89fEAPBWHtb+Kq3yN+AZe9AdnTiP1Sg/q5BbuP9Y25sE/9gFF8Jc41cziGlFmHSBrNVqzlFmJDBfYIOXKeDuHvIpe1VQ3yi/xWMo5zRnm/2qs0PiO8ISFOSNX8RelGUleyx3uUr7kOVsV0yfxRJjp8uy55NXeyqbq99C4CGx8pZkg7NSvSbx0a8Wo9LxWKkEPWzMxfoN9D/MSjK+Zqp2AO9pk72v8gXuXblsvs6Kj6Bjz3LPwJIUrr7MyJSd9c8FdjlYbT6xYBTTVTKzRyQrFnPUL0lKQ+4M0NQSNwOC+5qQ4HlMXF+/sOOqg==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"<588cd3f3-629a-451e-bc21-f255b5e0690c@oracle.com>","Date":"Fri, 16 Aug 2024 11:19:46 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"=?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= ,\n Peter Xu ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com> \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com> \n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-ClientProxiedBy":"BN9PR03CA0302.namprd03.prod.outlook.com\n (2603:10b6:408:112::7) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|IA1PR10MB7388:EE_","X-MS-Office365-Filtering-Correlation-Id":"0938f3f1-93e0-49f5-1c53-08dcbe06de1e","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|1800799024|366016|7416014|376014;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?vR+PD5EgOZ9KXNw52FTEuKa611SH+Iz?=\n\t=?utf-8?q?pGJD5eLkz1aN24NXvgVaeDW3SdH4XRlVCZihm6Zg0AZEDHkGTKoxrzeceSWlgol70?=\n\t=?utf-8?q?OiB+UjXrB5FUF2iLfGy14TRAksCUa4X2uX+RAQRu1cuVmax/kb8+fuj9Uy3cn4Zqv?=\n\t=?utf-8?q?V/RsaA2K9lo9fXyVsfEnUVr8pyZ0I5LMzxCEZwxJz7hcLsC2Oc95egRnV0uA0aRnJ?=\n\t=?utf-8?q?L6Qjt4om4zOBKI0WbuGJ+B+6DX8JPUq9skPElaBnOt/QIZgOQkTmHg2ZYXuU6gQ1T?=\n\t=?utf-8?q?N6MWxavtTYBRjJFCBwNwZfys23iCi39xtuUMTXn17VLy148Soez06izNE9zFX7swK?=\n\t=?utf-8?q?qXv9jVBf9zzXdyiViqULgUL198gnvINVGjwzVJySM+OEzMrlL9iyTGVn3teElVguN?=\n\t=?utf-8?q?Ckc7wmpe/j17dVas6q2J+pu7nYRnAUbT2kGfMKPKUDOW4PWLm2P+wcEUTZGYyKjGX?=\n\t=?utf-8?q?aey5or0t7bV1WLIweyR5LMJ2cLOH4UnXC8XtrzzmIdoECXhxEW1PEQgICLswRTVym?=\n\t=?utf-8?q?Y4rScC4mzVx+IOCGNzUwSAQZwPhdKlXCPTT5oejxVK2tB3ywMabIAOQHRv+EGgYWy?=\n\t=?utf-8?q?R3tRXCe+DGGFHgsE9+4LIlErC9k9OnxG+XJVcWrqur6jUdxtQvElPqfUbRzjLbiCj?=\n\t=?utf-8?q?V4BpPz8OqqztrFFUZq9S8s3R8l25uv2PqLlFqROi5b8a81VnD1ogxpEvUhH/SaGiD?=\n\t=?utf-8?q?Dvkt/4Hys+9CeyaPokMhVR7Wc/kHhnQEG7sFORw1UKa+/xKMzSB2/3yM/GdNXKljQ?=\n\t=?utf-8?q?xzhUHgg+5yLQPQN+598KJFqleDiBsR9zIRBcm+ypm6Dp7brO018+qybShVETxRZ40?=\n\t=?utf-8?q?8VSocCiDFEV+S/oVODBKnnlubQ+yE2WHCm+ywuyb2GyNJ+jQOt5WLPf8hsNqOQ0E+?=\n\t=?utf-8?q?kx1Q7heiuT0ZZ678Oz53PPUBrQPND8cXgKBmnrYT9QpWhiHVfA6Su/oDsn8alaK2O?=\n\t=?utf-8?q?JzIl9qdFfLzOJZ5eGKRgBVgbPYKqrnvky+YnxvmCSAUEuSCfJ+iHwvEKFEs+v+PiX?=\n\t=?utf-8?q?bwX7LXNM/cV7RTDapL4pQjJCgV3N/CymCLK+fW2LzGi5a26KFomP3EeeSkuZ9updB?=\n\t=?utf-8?q?PL7++W8jtD4ZinvaiTcQkU9hmGmRkExhZ11dhjO3haFyvnr6/BlN0hd8b844kJncu?=\n\t=?utf-8?q?Uv1a/uKYeVBxK+uELhjsOgz6+4pGlq5OlkZugcZyQxNnaBo69mYCcgbqc/qix0QWh?=\n\t=?utf-8?q?6RMUDCUnRsGTYyXZ8gBqltDfavma18VSCzqZK2csHW1XMu+lGjdm2oh2qyHsYJl/C?=\n\t=?utf-8?q?AxW4hCxqA7GB1Gl29KEoj/aTUNqswa9WILQ=3D=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(1800799024)(366016)(7416014)(376014); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?PKAxqlfXbq2BbhP3TystJa6FWkQD?=\n\t=?utf-8?q?Ygarc9aBtjDAAu+popZu56V9SrMOclFz0nl/CLmT8riMTJCTorTONhonUA94yZNX1?=\n\t=?utf-8?q?JqTUzjtQzyXHJZl4SUD1xCAEU3y4Y5FXUr+yJslCMJkvOfJkegKqh3pogyZYv6lKC?=\n\t=?utf-8?q?YnUqekzx1KtIAWJWB8bOgTEQouJsg+LPkGhvt7McQkIT5+TI1ImD3qTTcIpBu+RGm?=\n\t=?utf-8?q?G9TGQ84l+o8Vbp4dYqaiTGCu9UGPruRVIBPaxiuoDJ4h4kQAN1OvrwtDOq2RFH2ot?=\n\t=?utf-8?q?Str6HmHQLJSJiIRDFaj9j1qbEoJO0q+csdjrSaLRTor/y0LFi5X2b+fdqo5q+xaNG?=\n\t=?utf-8?q?/v1ICxaadL8Yqqg7kaeMv2euM3xndyDIRbvUcxuqn9r/e1TQFWqthSNXYBBwuMHfk?=\n\t=?utf-8?q?bOxWMS9vtRPm+UnHbZINjTi91U3xj4Sj9apsZS7lRfRdqjHmdZUHJNCgMvRX9ZeUh?=\n\t=?utf-8?q?x+P81/hhIDNELkh6oO1wkoFiNCIDSS/LgWzdhyx/uZKsZgMBGMVxWTaD5ycHoqosJ?=\n\t=?utf-8?q?yP3bvzHFcrDVSWZ4aOozMTCSkKPkVj1CjAyLDTgTfnLVtuF3ahiu1QftG1PrlZqai?=\n\t=?utf-8?q?83vjp2dA7lu0wG8lYBQ3qbE1KhMhGAkshKGwdeQw3d9Zl2+yuKzTyLHeVLbyBtDQR?=\n\t=?utf-8?q?YBm8MHIi/yx/pcuNTJ5xCBap8cuvouiZsirB6bNFTUe/p7C70Ddb5zBSw+kry6x+x?=\n\t=?utf-8?q?WhLurqRVvuv31JJ93ozIdGwlV2nHbxT+Z5AFwW6iUh3bdov4z0RZI3oIRSQoV/3kO?=\n\t=?utf-8?q?g+UrMRjAjVaSHt5zIRnNi91YUnTB/knPWZlVSI4I01Y3J+18eumVYeAaHrpyDCt1X?=\n\t=?utf-8?q?/4vgtLD7oSZrvBYiw8uu3ygD4LSsFPN5ZAAncODr58C+NjRXSwTQolMOccl0IDJCP?=\n\t=?utf-8?q?feoolE33J1kdlR5lPsTZAhHHejkoUF2k1KtZuompoQGquKxNyGKIacZLt+VvY7Hnt?=\n\t=?utf-8?q?UQECeLFSFfDYiG/o7CrkUJV2BGsu3pN8npvZiuLvwcEjmfdleNDERgO0BSg//uuBw?=\n\t=?utf-8?q?/SLrkiACnYmkUx+dPFIQODOZ+CYwPJVidgh6MKJE9V6b+LQmcxrKm5odxWTacGXaG?=\n\t=?utf-8?q?YF3M9215B/0Ra/2vo2+MC1KDkLOG9Pxx2ZnHTtWtEFOkBXWhA7YOodb2sVkCv+TW5?=\n\t=?utf-8?q?MLvyN34sBXjQIKwzsDfItYv8tdpPz6B5fGMVdm0SoojOOT3ar+LH4ePRWsy989FuW?=\n\t=?utf-8?q?GciluFkTG/EhChJv/4aUQLb721EV3iE4SBcjXuPancRwiYl1/LLJ0S9rBuHoBxX0P?=\n\t=?utf-8?q?I+rVu3NjuJWzkWH45AzId+mCzvo2jHCLQlMyVoXe/rJDnl7ffGQkPOr55AW0Bc2jy?=\n\t=?utf-8?q?cFgyMr3YWvXBh15LpOFSziDwflqYSZIJUx720gVf6sTBJA8iXhhNbYKgh9wFC3ooc?=\n\t=?utf-8?q?kIvP/UQyDRfYpF+JLrz4OQNpMGR/mDqIKv6JPmZx9DCsa3XPcWZyccdJWzibs+4RX?=\n\t=?utf-8?q?RRoSS/AKrgUMG+Mdxrc0cDrjcXlM5QiO8xHYBqP/aggjZw33AiBfx6pVg9hoJQi2Q?=\n\t=?utf-8?q?qBfcWaiRe3poHDVD3n/IH4/PAuTY3MKb/A=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n T1L2He3V1Trkh0Lx/aplJ5Z3usc1WtiOfc7HuRC12bhEuLGulPnNHELG0yV0VJvoLIleCsf3162gy+zo/1nBSoizGgwKk/LxxsySQJjgw8UACinAhvgVvuIPYyQal4kCXwW/Nq/N8hZUwXJ7fR93ateB93o9sorZH6eveDOAmx8D3GhA50jq1LqVxO/WfhEgT/I9H1F31zhAum0zV0BdZucP+ve1p2anQYF3dY2JOy/Z2Ujz/gH7tsAbmiVLFEG8quXRnkZp7QoFGyIsVROHScMu6SdZQIYecKM1iEohcdOc1i7pC1clC6VcRyIhcnG6LEMRPFK9oaVwT8fpEtf+ls1FThDPdkIK+EcyV9GkzjaFQJ5NmNdqATEms+F5SFbb2+8gTa2ln5uPvEEqZRpS48QcYv3utW1OkSCCPCmo0G+rf8XgtmnbOqTZjFSTQH8PRnUE/xbWdF+2KoHP5wG+la4RCTaGey6AcOiTFlV8nGNxC+5mDg/hHTBy1rRvbkjv1j6JC/V1zYOqETiGwoOc7nVOqes8neSHmguJ4ITikvuqnO6CSkpuaZAANOBWvTb2dme9lPjv/+joC+pULCxtx16Lg/T35SnEbg0UUmJCNJE=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 0938f3f1-93e0-49f5-1c53-08dcbe06de1e","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"16 Aug 2024 15:19:48.7151 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n aBnF7+rejh3FnUjH20/NFs9i4S75cji5d+8NthgLrmlTa8va1nDgrlNf88kDWBvoAziKPnZ0GTxTMCjfP8zr8keEpLj1iNmPoNCSmNUSjLk=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"IA1PR10MB7388","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-16_09,2024-08-16_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 adultscore=0\n malwarescore=0\n mlxlogscore=999 bulkscore=0 phishscore=0 mlxscore=0 spamscore=0\n suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1\n engine=8.12.0-2407110000 definitions=main-2408160110","X-Proofpoint-ORIG-GUID":"eniq6BNMKiA22Fd_fi4TdxmFmZezjoWi","X-Proofpoint-GUID":"eniq6BNMKiA22Fd_fi4TdxmFmZezjoWi","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363498,"web_url":"http://patchwork.ozlabs.org/comment/3363498/","msgid":"","list_archive_url":null,"date":"2024-08-16T15:34:10","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > file, and only 1 of them can ever have them open.\n> > > > \n> > > > I thought we used to have similar issue with block devices, but I assume\n> > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > going to take the file lock).\n> > > > \n> > > > Maybe this is about something else?\n> > > \n> > > I don't have an example where this fails.\n> > > \n> > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > the same block device, but the error is suppressed if you add the -incoming\n> > > argument, due to this code:\n> > > \n> > > blk_attach_dev()\n> > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > blk->disable_perm = true;\n> > \n> > Yep, this one is pretty much expected.\n> > \n> > > \n> > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > \n> > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > the socket path before binding on the assumption that it is stale).\n> > > \n> > > One must use a different name for the socket for second qemu, and clients\n> > > that wish to connect must be aware of the new port.\n> > > \n> > > > > Network ports also conflict.\n> > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > I forgot to promote.\n> > > > \n> > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > \n> > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > stable across migrations, where the host ports can be different as long as\n> > > > the host ports can forward guest port messages correctly?\n> > > \n> > > Yes, one must use a different host port number for the second qemu, and clients\n> > > that wish to connect must be aware of the new port.\n> > > \n> > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > cpr-exec does not.\n> > \n> > Right, and my understanding is all these facilities are already there, so\n> > no new code should be needed on reconnect issues if to support cpr-transfer\n> > in Libvirt or similar management layers that supports migrations.\n> \n> Note Libvirt explicitly blocks localhost migration today because\n> solving all these clashing resource problems is a huge can of worms\n> and it can't be made invisible to the user of libvirt in any practical\n> way.\n\nAhhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\nsupported local migration somehow on top of libvirt.\n\nDoes it mean that cpr-transfer is a no-go in this case at least for Libvirt\nto consume it (as cpr-* is only for local host migrations so far)? Even if\nall the rest issues we're discussing with cpr-exec, is that the only way to\ngo for Libvirt, then?","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=It8MRCr6;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WlmK30bgQz1yYl\n\tfor ; Sat, 17 Aug 2024 01:35:15 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1seyy5-0002J4-SS; Fri, 16 Aug 2024 11:34:21 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1seyy4-0002IY-CM\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:34:20 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1seyy2-0001b5-Jv\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 11:34:20 -0400","from mail-oo1-f71.google.com (mail-oo1-f71.google.com\n [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-394-YZP2IC-RP9-hkY8_M0gyug-1; Fri, 16 Aug 2024 11:34:14 -0400","by mail-oo1-f71.google.com with SMTP id\n 006d021491bc7-5d5bb101d53so318883eaf.1\n for ; Fri, 16 Aug 2024 08:34:14 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n 6a1803df08f44-6bf6fdd9c68sm18637636d6.3.2024.08.16.08.34.11\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Fri, 16 Aug 2024 08:34:12 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723822455;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=B9RtqEswVmPxQ+vIIwqYss5Erzk/MhyDzg6O3ZT/S40=;\n b=It8MRCr68bwTdAXrKfBTxgMMwnX3HFgEqwu/cZiMJ3CcbcLHBiICL2DbkUe2gL/OJl2eEm\n TW11sLHza6KMvwzcbumtbkdHo6HQ0NF2XKS94S5g2nKkl4NBPpszvZ/BQeIOn3wC++jCyH\n UGtGm7lr3GsxTyDUoNSwkb3xK6HB+KU=","X-MC-Unique":"YZP2IC-RP9-hkY8_M0gyug-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1723822453; x=1724427253;\n h=in-reply-to:content-transfer-encoding:content-disposition\n :mime-version:references:message-id:subject:cc:to:from:date\n :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n bh=B9RtqEswVmPxQ+vIIwqYss5Erzk/MhyDzg6O3ZT/S40=;\n b=qt21NgCRL4qNt9PS/5hBd07tmfj02lCLIxeQEFI1rICnC0TyFYscSbI0pmEr9KfHnb\n WAQvcZBjVfaUXzE7kr1hWVtjTUt7Apf066++a1j9wlaunXm44/ZWVC7Fl+DrDsJUP3eI\n oNMPb6Piv7xWqLcEtBBo+mFcHKLywV6C5ME0T5Eak6owspv8nRCCZxp9z2bRYHf/4lmr\n +agKoOmwf1oxQsdpwxTmO234pofyCPQG0ZMngjEqTfVaTXY39ALc6uTX6jFh3RDQsNi1\n zovVJDax2t0Bah4mKnORSbad6VqmboNBAKqCEdnxqba/+AgBtRY+lUSpvAjHiMncyu1z\n vMsg==","X-Forwarded-Encrypted":"i=1;\n AJvYcCU4nx6EhwHQ8rWxjAKm0ohivaTjMC334vbFl9Kkwf42x0tuN6WuZ656MtQ0LzW+chQlCx4TLW/5ow5Q@nongnu.org","X-Gm-Message-State":"AOJu0YwbOG1o6SvYQ02PFmCXCtpoZqDGdi5SA2OiP08rbt7YweZyHe6L\n 8S2WmZoaaE/bbDtQC4+AyaQFRL8fOsPoJAoYVOyPQkSCed/wBt5L3MOb9QUjpfh+CUtHs6XfDbG\n k2hwQ4ugO8sIsS+xJjcJWX+aU6huw8K+qCDV18SOiWaRc8NoQsLusT0K3KZ0m","X-Received":["by 2002:a4a:de16:0:b0:5d8:5a15:146b with SMTP id\n 006d021491bc7-5da97d404d8mr2305287eaf.0.1723822453115;\n Fri, 16 Aug 2024 08:34:13 -0700 (PDT)","by 2002:a4a:de16:0:b0:5d8:5a15:146b with SMTP id\n 006d021491bc7-5da97d404d8mr2305276eaf.0.1723822452703;\n Fri, 16 Aug 2024 08:34:12 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IEZvso1gtwMuoYctZvI9hzfBib7AGHir6ww53Ue+ruIjP1yqSr7E44jW1OXZM39DGnvVrlUYw==","Date":"Fri, 16 Aug 2024 11:34:10 -0400","From":"Peter Xu ","To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363530,"web_url":"http://patchwork.ozlabs.org/comment/3363530/","msgid":"","list_archive_url":null,"date":"2024-08-16T16:00:32","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > file, and only 1 of them can ever have them open.\n> > > > > \n> > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > going to take the file lock).\n> > > > > \n> > > > > Maybe this is about something else?\n> > > > \n> > > > I don't have an example where this fails.\n> > > > \n> > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > argument, due to this code:\n> > > > \n> > > > blk_attach_dev()\n> > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > blk->disable_perm = true;\n> > > \n> > > Yep, this one is pretty much expected.\n> > > \n> > > > \n> > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > \n> > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > the socket path before binding on the assumption that it is stale).\n> > > > \n> > > > One must use a different name for the socket for second qemu, and clients\n> > > > that wish to connect must be aware of the new port.\n> > > > \n> > > > > > Network ports also conflict.\n> > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > I forgot to promote.\n> > > > > \n> > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > \n> > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > stable across migrations, where the host ports can be different as long as\n> > > > > the host ports can forward guest port messages correctly?\n> > > > \n> > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > that wish to connect must be aware of the new port.\n> > > > \n> > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > cpr-exec does not.\n> > > \n> > > Right, and my understanding is all these facilities are already there, so\n> > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > in Libvirt or similar management layers that supports migrations.\n> > \n> > Note Libvirt explicitly blocks localhost migration today because\n> > solving all these clashing resource problems is a huge can of worms\n> > and it can't be made invisible to the user of libvirt in any practical\n> > way.\n> \n> Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> supported local migration somehow on top of libvirt.\n\nSince kubevirt runs inside a container, \"localhost\" migration\nis effectively migrating between 2 completely separate OS installs\n(containers), that happen to be on the same physical host. IOW, it\nis a cross-host migration from Libvirt & QEMU's POV, and there are\nno clashing resources to worry about.\n\n> Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> to consume it (as cpr-* is only for local host migrations so far)? Even if\n> all the rest issues we're discussing with cpr-exec, is that the only way to\n> go for Libvirt, then?\n\ncpr-exec is certainly appealing from the POV of avoiding the clashing\nresources problem in libvirt.\n\nIt has own issues though, because libvirt runs all QEMU processes with\nseccomp filters that block 'execve', as we consider QEMU to be untrustworthy\nand thus don't want to allow it to exec anything !\n\nI don't know which is the lesser evil from libvirt's POV.\n\nPersonally I see security controls as an overriding requirement for\neverything.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=VXLdoHig;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wlmv3288xz1yfB\n\tfor ; Sat, 17 Aug 2024 02:01:19 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sezO2-00023c-5u; Fri, 16 Aug 2024 12:01:10 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sezNi-0000ok-3l\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:01:01 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sezNe-0005qb-13\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:00:49 -0400","from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-568-IrTPgRwTMh-klPgzeHnYmw-1; Fri,\n 16 Aug 2024 12:00:41 -0400","from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id 8D49B1955F3D; Fri, 16 Aug 2024 16:00:39 +0000 (UTC)","from redhat.com (unknown [10.42.28.143])\n by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id D39D219560A3; Fri, 16 Aug 2024 16:00:35 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723824044;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=OtcEckEWkXRpUlT92tRdtVJBlm22Cr3rtbZUHCc8bBc=;\n b=VXLdoHigvBbSOeWuyA29U1nG1PWP0VRFY+plkEcb90cByKxTbBIlir5wJWYgQ9E0WfsOhi\n pcSYyJ3grYkwmeE7xzj3Wdqjlqwdi7+i0tweBe4Xw/yMLqxkb9sAQ1gSVVoHBYohObzx6n\n EBqY0cNIfN0TVt8mQpHCR5Xd89xVA04=","X-MC-Unique":"IrTPgRwTMh-klPgzeHnYmw-1","Date":"Fri, 16 Aug 2024 17:00:32 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Peter Xu ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.40","Received-SPF":"pass client-ip=170.10.133.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363543,"web_url":"http://patchwork.ozlabs.org/comment/3363543/","msgid":"","list_archive_url":null,"date":"2024-08-16T16:17:30","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n> On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> > On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > > file, and only 1 of them can ever have them open.\n> > > > > > \n> > > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > > going to take the file lock).\n> > > > > > \n> > > > > > Maybe this is about something else?\n> > > > > \n> > > > > I don't have an example where this fails.\n> > > > > \n> > > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > > argument, due to this code:\n> > > > > \n> > > > > blk_attach_dev()\n> > > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > > blk->disable_perm = true;\n> > > > \n> > > > Yep, this one is pretty much expected.\n> > > > \n> > > > > \n> > > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > > \n> > > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > > the socket path before binding on the assumption that it is stale).\n> > > > > \n> > > > > One must use a different name for the socket for second qemu, and clients\n> > > > > that wish to connect must be aware of the new port.\n> > > > > \n> > > > > > > Network ports also conflict.\n> > > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > > I forgot to promote.\n> > > > > > \n> > > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > > \n> > > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > > stable across migrations, where the host ports can be different as long as\n> > > > > > the host ports can forward guest port messages correctly?\n> > > > > \n> > > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > > that wish to connect must be aware of the new port.\n> > > > > \n> > > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > > cpr-exec does not.\n> > > > \n> > > > Right, and my understanding is all these facilities are already there, so\n> > > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > > in Libvirt or similar management layers that supports migrations.\n> > > \n> > > Note Libvirt explicitly blocks localhost migration today because\n> > > solving all these clashing resource problems is a huge can of worms\n> > > and it can't be made invisible to the user of libvirt in any practical\n> > > way.\n> > \n> > Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> > supported local migration somehow on top of libvirt.\n> \n> Since kubevirt runs inside a container, \"localhost\" migration\n> is effectively migrating between 2 completely separate OS installs\n> (containers), that happen to be on the same physical host. IOW, it\n> is a cross-host migration from Libvirt & QEMU's POV, and there are\n> no clashing resources to worry about.\n\nOK, makes sense.\n\nThen do you think it's possible to support cpr-transfer in that scenario\nfrom Libvirt POV?\n\n> \n> > Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> > to consume it (as cpr-* is only for local host migrations so far)? Even if\n> > all the rest issues we're discussing with cpr-exec, is that the only way to\n> > go for Libvirt, then?\n> \n> cpr-exec is certainly appealing from the POV of avoiding the clashing\n> resources problem in libvirt.\n> \n> It has own issues though, because libvirt runs all QEMU processes with\n> seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n> and thus don't want to allow it to exec anything !\n> \n> I don't know which is the lesser evil from libvirt's POV.\n> \n> Personally I see security controls as an overriding requirement for\n> everything.\n\nOne thing I am aware of is cpr-exec is not the only one who might start to\nuse exec() in QEMU. TDX fundamentally will need to create another key VM to\ndeliver the keys and the plan seems to be using exec() too. However in\nthat case per my understanding the exec() is optional - the key VM can also\nbe created by Libvirt.\n\nIOW, it looks like we can still stick with execve() being blocked yet so\nfar except cpr-exec().\n\nHmm, this makes the decision harder to make. We need to figure out a way\non knowing how to consume this feature for at least open source virt\nstack.. So far it looks like it's only possible (if we take seccomp high\npriority) we use cpr-transfer but only in a container.\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=YTRa/EH8;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WlnG733rBz1yfB\n\tfor ; Sat, 17 Aug 2024 02:17:51 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1seze4-0007tc-4l; Fri, 16 Aug 2024 12:17:44 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1seze0-0007cq-03\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:17:40 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sezdy-00089V-4h\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:17:39 -0400","from mail-qv1-f70.google.com (mail-qv1-f70.google.com\n [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-495-hi4n2gscM06lDB1sDpXMLg-1; Fri, 16 Aug 2024 12:17:34 -0400","by mail-qv1-f70.google.com with SMTP id\n 6a1803df08f44-6bf7b853f0dso2633056d6.0\n for ; Fri, 16 Aug 2024 09:17:33 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n 6a1803df08f44-6bf6fdd9050sm18920626d6.27.2024.08.16.09.17.31\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Fri, 16 Aug 2024 09:17:32 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723825055;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=NzLoHDJXEyJ2UjOgFn3oYMoUmhbTiFliCp/lLie4WcM=;\n b=YTRa/EH8rsfccs01lFDiwqWG2k9Tenc4q5jsV9H1QHpK0MFnPMEPeyyJ5jvE3RpmeZppjU\n wItQ1K24c0VLqtZLzg3038Px29YwwCFK17RPxDhQRUd1OH7dZprT5sCtdnkZ35gKXWCWh7\n MTqZ4fpV2ufdV83EvclQ3W4mYqJaIJQ=","X-MC-Unique":"hi4n2gscM06lDB1sDpXMLg-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1723825053; x=1724429853;\n h=in-reply-to:content-transfer-encoding:content-disposition\n :mime-version:references:message-id:subject:cc:to:from:date\n :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n bh=NzLoHDJXEyJ2UjOgFn3oYMoUmhbTiFliCp/lLie4WcM=;\n b=H3NKHLg6cLS88rFKd/BD5faOxxzJWu2Vxi8eFlm6sCC+IRlGPw1floEQx19CglVfLv\n BjgfV0rv0jGJzykb6xVypypnyZLhcqTXi7YG7674UwIl/O0BREvU7+OKc9N2J+/ZR+oq\n DT2otS4xqUL0GPOyYzqU3Wy85HxIGUQl63Hya6wirdSbsLBMYr1Gz6al3jXMuCeRU7AE\n ToryiWI4zODZHsTfihCs65pWbp5zwsyDpVnVFF82rhlsSEj5bRM0dpewm0azEGtu69a3\n JW9U5mesWA4kBcY/lp0qOZUTE4Hxq8tInD9vayq2Hr06DPAWESv/QChKkYlXPvbK+Q7T\n KrRg==","X-Forwarded-Encrypted":"i=1;\n AJvYcCVMa3yaJYbtGagEw6ig3Jsw/EVcMei6BH6yqouE0/7++hlhFTw2gaH4dn3OvCyvdjb4SSNDzhJiAS97@nongnu.org","X-Gm-Message-State":"AOJu0YwTIH4zq4JvdS3hbi/HbPjHNuvDpfJClDoUmc7xGbIwWXVLw5D+\n gn71Ir0gVrbmgCXL/JolnZHqn7LkSC5QboqQ9wwegSGAUHTOd3+iEJlLTnIC0dXqqSSAqtRG79p\n uFkEgwpipzejmf8Bo5uCISaN5M9BD0/z+ekCuUSZBKgMUU+YPd2Rh","X-Received":["by 2002:a05:6214:ccd:b0:6b2:af3c:f710 with SMTP id\n 6a1803df08f44-6bf7cd9f4aamr20957976d6.2.1723825053199;\n Fri, 16 Aug 2024 09:17:33 -0700 (PDT)","by 2002:a05:6214:ccd:b0:6b2:af3c:f710 with SMTP id\n 6a1803df08f44-6bf7cd9f4aamr20957756d6.2.1723825052776;\n Fri, 16 Aug 2024 09:17:32 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IGXZ/NARZCg+rV/HGh//wjRcGvQMHaQrFgR/3G8my8ekVwAyN2K2MqTnhebEYKg9XkFMsKVRQ==","Date":"Fri, 16 Aug 2024 12:17:30 -0400","From":"Peter Xu ","To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363549,"web_url":"http://patchwork.ozlabs.org/comment/3363549/","msgid":"","list_archive_url":null,"date":"2024-08-16T16:28:18","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Aug 16, 2024 at 12:17:30PM -0400, Peter Xu wrote:\n> On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n> > On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> > > On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > > > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > > > file, and only 1 of them can ever have them open.\n> > > > > > > \n> > > > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > > > going to take the file lock).\n> > > > > > > \n> > > > > > > Maybe this is about something else?\n> > > > > > \n> > > > > > I don't have an example where this fails.\n> > > > > > \n> > > > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > > > argument, due to this code:\n> > > > > > \n> > > > > > blk_attach_dev()\n> > > > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > > > blk->disable_perm = true;\n> > > > > \n> > > > > Yep, this one is pretty much expected.\n> > > > > \n> > > > > > \n> > > > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > > > \n> > > > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > > > the socket path before binding on the assumption that it is stale).\n> > > > > > \n> > > > > > One must use a different name for the socket for second qemu, and clients\n> > > > > > that wish to connect must be aware of the new port.\n> > > > > > \n> > > > > > > > Network ports also conflict.\n> > > > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > > > I forgot to promote.\n> > > > > > > \n> > > > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > > > \n> > > > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > > > stable across migrations, where the host ports can be different as long as\n> > > > > > > the host ports can forward guest port messages correctly?\n> > > > > > \n> > > > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > > > that wish to connect must be aware of the new port.\n> > > > > > \n> > > > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > > > cpr-exec does not.\n> > > > > \n> > > > > Right, and my understanding is all these facilities are already there, so\n> > > > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > > > in Libvirt or similar management layers that supports migrations.\n> > > > \n> > > > Note Libvirt explicitly blocks localhost migration today because\n> > > > solving all these clashing resource problems is a huge can of worms\n> > > > and it can't be made invisible to the user of libvirt in any practical\n> > > > way.\n> > > \n> > > Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> > > supported local migration somehow on top of libvirt.\n> > \n> > Since kubevirt runs inside a container, \"localhost\" migration\n> > is effectively migrating between 2 completely separate OS installs\n> > (containers), that happen to be on the same physical host. IOW, it\n> > is a cross-host migration from Libvirt & QEMU's POV, and there are\n> > no clashing resources to worry about.\n> \n> OK, makes sense.\n> \n> Then do you think it's possible to support cpr-transfer in that scenario\n> from Libvirt POV?\n> \n> > \n> > > Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> > > to consume it (as cpr-* is only for local host migrations so far)? Even if\n> > > all the rest issues we're discussing with cpr-exec, is that the only way to\n> > > go for Libvirt, then?\n> > \n> > cpr-exec is certainly appealing from the POV of avoiding the clashing\n> > resources problem in libvirt.\n> > \n> > It has own issues though, because libvirt runs all QEMU processes with\n> > seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n> > and thus don't want to allow it to exec anything !\n> > \n> > I don't know which is the lesser evil from libvirt's POV.\n> > \n> > Personally I see security controls as an overriding requirement for\n> > everything.\n> \n> One thing I am aware of is cpr-exec is not the only one who might start to\n> use exec() in QEMU. TDX fundamentally will need to create another key VM to\n> deliver the keys and the plan seems to be using exec() too. However in\n> that case per my understanding the exec() is optional - the key VM can also\n> be created by Libvirt.\n\nSince nothing is merged, I'd consider whatever might have been presented\nwrt TDX migration to all be open for potential re-design. With SNP there\nis the SVSM paravisor which runs inside the guest to provide services,\nand IIUC its intended to address migration service needs.\n\nThere's a push to support SVSM with TDX too, in order to enable vTPM\nsupport. With that it might make sense to explore whether SVSM can\nservice migration for TDX too, instead of having a separate parallel\nVM on the host. IMHO its highly desirable to have a common architecture\nfor CVM migration from QEMU's POV, and from a libvirt POV I'd like to\navoid having extra host VMs too.\n\n\n> IOW, it looks like we can still stick with execve() being blocked yet so\n> far except cpr-exec().\n> \n> Hmm, this makes the decision harder to make. We need to figure out a way\n> on knowing how to consume this feature for at least open source virt\n> stack.. So far it looks like it's only possible (if we take seccomp high\n> priority) we use cpr-transfer but only in a container.\n\nOr we have cpr-transfer, but libvirt has to more games to solve the\nclashing resources problem, by making much more use of FD passing,\nand/or by changnig path conventions, or a mix of both.\n\nWhat might make this viable is that IIUC, CPR only permits a subset\nof backends to be used, so libvirt doesn't have to solve clashing\nresources for /everything/, just parts that are supported by CPR.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=OIo2JRbg;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WlnVq1SkVz20Bh\n\tfor ; Sat, 17 Aug 2024 02:28:51 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sezoa-00048p-Kl; Fri, 16 Aug 2024 12:28:36 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sezoZ-00040T-KU\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:28:35 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sezoW-00015w-Vq\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 12:28:34 -0400","from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-117-6whYl7p_OemslGUPfWb9oA-1; Fri,\n 16 Aug 2024 12:28:28 -0400","from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id 9C6B21956048; Fri, 16 Aug 2024 16:28:25 +0000 (UTC)","from redhat.com (unknown [10.42.28.143])\n by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id 273211954B04; Fri, 16 Aug 2024 16:28:21 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1723825711;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=aNkaJQ/DOIkhSCnNQ5G0CYowaq/sSde4/HKQXjiwMw4=;\n b=OIo2JRbgZyHud4clThfbV5V5XpVxlGi8bqRtUb8BMPQ6SPf5vQZOm2o6oSAe3UwsTZEE7t\n 7rvt9bURD2rkW3xWbQvwHtQrm8mgozxzt9VuiRumJYhk5fMiiXVe169HjpTO4lTwSNt/IT\n ZJK4biONWyIht4hiiQ9DAauvNdQbyOc=","X-MC-Unique":"6whYl7p_OemslGUPfWb9oA-1","Date":"Fri, 16 Aug 2024 17:28:18 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Peter Xu ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":" \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.15","Received-SPF":"pass client-ip=170.10.129.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.131,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3363580,"web_url":"http://patchwork.ozlabs.org/comment/3363580/","msgid":"","list_archive_url":null,"date":"2024-08-16T17:09:23","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/16/2024 12:17 PM, Peter Xu wrote:\n> On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n>> On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n>>> On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n>>>> On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n>>>>> On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n>>>>>> On 8/13/2024 3:46 PM, Peter Xu wrote:\n>>>>>>> On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n>>>>>>>>> The flipside, however, is that localhost migration via 2 separate QEMU\n>>>>>>>>> processes has issues where both QEMUs want to be opening the very same\n>>>>>>>>> file, and only 1 of them can ever have them open.\n>>>>>>>\n>>>>>>> I thought we used to have similar issue with block devices, but I assume\n>>>>>>> it's solved for years (and whoever owns it will take proper file lock,\n>>>>>>> IIRC, and QEMU migration should properly serialize the time window on who's\n>>>>>>> going to take the file lock).\n>>>>>>>\n>>>>>>> Maybe this is about something else?\n>>>>>>\n>>>>>> I don't have an example where this fails.\n>>>>>>\n>>>>>> I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n>>>>>> the same block device, but the error is suppressed if you add the -incoming\n>>>>>> argument, due to this code:\n>>>>>>\n>>>>>> blk_attach_dev()\n>>>>>> if (runstate_check(RUN_STATE_INMIGRATE))\n>>>>>> blk->disable_perm = true;\n>>>>>\n>>>>> Yep, this one is pretty much expected.\n>>>>>\n>>>>>>\n>>>>>>>> Indeed, and \"files\" includes unix domain sockets.\n>>>>>>\n>>>>>> More on this -- the second qemu to bind a unix domain socket for listening\n>>>>>> wins, and the first qemu loses it (because second qemu unlinks and recreates\n>>>>>> the socket path before binding on the assumption that it is stale).\n>>>>>>\n>>>>>> One must use a different name for the socket for second qemu, and clients\n>>>>>> that wish to connect must be aware of the new port.\n>>>>>>\n>>>>>>>> Network ports also conflict.\n>>>>>>>> cpr-exec avoids such problems, and is one of the advantages of the method that\n>>>>>>>> I forgot to promote.\n>>>>>>>\n>>>>>>> I was thinking that's fine, as the host ports should be the backend of the\n>>>>>>> VM ports only anyway so they don't need to be identical on both sides?\n>>>>>>>\n>>>>>>> IOW, my understanding is it's the guest IP/ports/... which should still be\n>>>>>>> stable across migrations, where the host ports can be different as long as\n>>>>>>> the host ports can forward guest port messages correctly?\n>>>>>>\n>>>>>> Yes, one must use a different host port number for the second qemu, and clients\n>>>>>> that wish to connect must be aware of the new port.\n>>>>>>\n>>>>>> That is my point -- cpr-transfer requires fiddling with such things.\n>>>>>> cpr-exec does not.\n>>>>>\n>>>>> Right, and my understanding is all these facilities are already there, so\n>>>>> no new code should be needed on reconnect issues if to support cpr-transfer\n>>>>> in Libvirt or similar management layers that supports migrations.\n>>>>\n>>>> Note Libvirt explicitly blocks localhost migration today because\n>>>> solving all these clashing resource problems is a huge can of worms\n>>>> and it can't be made invisible to the user of libvirt in any practical\n>>>> way.\n>>>\n>>> Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n>>> supported local migration somehow on top of libvirt.\n>>\n>> Since kubevirt runs inside a container, \"localhost\" migration\n>> is effectively migrating between 2 completely separate OS installs\n>> (containers), that happen to be on the same physical host. IOW, it\n>> is a cross-host migration from Libvirt & QEMU's POV, and there are\n>> no clashing resources to worry about.\n> \n> OK, makes sense.\n> \n> Then do you think it's possible to support cpr-transfer in that scenario\n> from Libvirt POV?\n> \n>>\n>>> Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n>>> to consume it (as cpr-* is only for local host migrations so far)? Even if\n>>> all the rest issues we're discussing with cpr-exec, is that the only way to\n>>> go for Libvirt, then?\n>>\n>> cpr-exec is certainly appealing from the POV of avoiding the clashing\n>> resources problem in libvirt.\n>>\n>> It has own issues though, because libvirt runs all QEMU processes with\n>> seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n>> and thus don't want to allow it to exec anything !\n>>\n>> I don't know which is the lesser evil from libvirt's POV.\n>>\n>> Personally I see security controls as an overriding requirement for\n>> everything.\n> \n> One thing I am aware of is cpr-exec is not the only one who might start to\n> use exec() in QEMU. TDX fundamentally will need to create another key VM to\n> deliver the keys and the plan seems to be using exec() too. However in\n> that case per my understanding the exec() is optional - the key VM can also\n> be created by Libvirt.\n> \n> IOW, it looks like we can still stick with execve() being blocked yet so\n> far except cpr-exec().\n> \n> Hmm, this makes the decision harder to make. We need to figure out a way\n> on knowing how to consume this feature for at least open source virt\n> stack.. So far it looks like it's only possible (if we take seccomp high\n> priority) we use cpr-transfer but only in a container.\n\nlibvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\nand change namespace operations. I have a patch in my workspace to be submitted\nlater called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\nlibvirt to block fork and namespace but allow exec.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=SJ93ZdFJ;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=JTDvk10r;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WlpQz2KMwz1yfB\n\tfor ; Sat, 17 Aug 2024 03:10:35 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sf0SK-0003dV-IR; Fri, 16 Aug 2024 13:09:40 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sf0SJ-0003d2-L2\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 13:09:39 -0400","from mx0b-00069f02.pphosted.com ([205.220.177.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sf0SG-00077b-0V\n for qemu-devel@nongnu.org; Fri, 16 Aug 2024 13:09:39 -0400","from pps.filterd (m0246631.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 47GFiSkE014821;\n Fri, 16 Aug 2024 17:09:32 GMT","from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta01.appoci.oracle.com [130.35.100.223])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40wy035584-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Fri, 16 Aug 2024 17:09:32 +0000 (GMT)","from pps.filterd\n (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 47GGlhZX010633; Fri, 16 Aug 2024 17:09:31 GMT","from nam12-dm6-obe.outbound.protection.outlook.com\n (mail-dm6nam12lp2170.outbound.protection.outlook.com [104.47.59.170])\n by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id\n 40wxncmpm3-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Fri, 16 Aug 2024 17:09:31 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by SJ2PR10MB7810.namprd10.prod.outlook.com (2603:10b6:a03:578::10)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.9; Fri, 16 Aug\n 2024 17:09:28 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7875.016; Fri, 16 Aug 2024\n 17:09:28 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=SNfqrwgjQ1+kQblaobuIMVJDUhxwZrc5CY38SBhKnQc=; b=\n SJ93ZdFJKt38ga60ZRaD6VwjIiyD1nM1uZA7y0+fhZ4WqD/NS3LIchEc1TCmHk6a\n dzzGfhZQ89oroVnNyF4hRAY4Vou9jRBmLUEDTLhcwGgTKBO9nITKCdJ8iPkSXRhI\n mW1geH7X8ywCx6Ss4atkcJ9/s7Zdukr7QJK/5jehNxKNFZRasjCgdqRZhFY6/XFt\n BImAIlSXxk5H5aAt+xclAwqiL91o49a6q7krYBb0vp72IhSI743w7f6FKDel7ZOa\n IOoNpWykVmpRgU1hHtkFNCjmhXuXU9puDzjKEkf4h95Jgu1RS0zU7+1HEQiJoBVP\n 3Gka2y7/mvDnGJzH/7zKYg==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=SNfqrwgjQ1+kQblaobuIMVJDUhxwZrc5CY38SBhKnQc=;\n b=JTDvk10rFO7FmsHvjvdqtwRmhP/UHfDfqOlxyfW6v4rJfMHrVPLLpiRVPk/eOAw6tVKh6WBBL7hO9nuIIApIV8iBb52kdtiRu5MktI93QvkdptzZQq/qZ5gW9R+0sqXNOGuwdslup9L8rlvlCkEjRh7kk6wibioQdtFAiBzhoyQ="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=VtUlRSL1RqUSRYJ8Lhv5ouoXlLcmpY1GZr9MM4wLvTFo5mzieVVmc39oi74JfjeaNuYh6yB0//YfXTKMjo5Yd73Ciz1caG8JA+dB2K81aAl09AQbgRz+9nW7O5MFfYTmVrUtwSGIdCbOAdgChPc267JkTkAKhy+ciS1XM8T86pF2cDmTgOgVBf2eLT/+MNf+2kkmJccnO8uJqFMfU/syPfE07kRFo/iek9cs0yDxTU1qs6plBn+mNmN+kIqbNHQRmWF5yc7dUrEbRImZiaf100WEGm7DaqhSfnA9yD27bpgOUVkvq5ywU7597MVFVh3W7iOa1cBET9G4k9rRGuuAlQ==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=SNfqrwgjQ1+kQblaobuIMVJDUhxwZrc5CY38SBhKnQc=;\n b=pZj+c1lLuHJeH9W0UXyv41w6d8SVirEeKUmafK/l4PDzLZ3qd1dtyo7BUB2YrEViBFgiPAkHwRgyIhB1ggHGh0dRBf9Z/jkNYMFNHxC/1dIACTIGy2X1Cdr6rmRFPGkJs3k3aQ+wC10G0sKikMI6iqkP0KieNauZRKhkZfsC90pM31E17D8wT7Mlvo6+qMdwuvROks3K2/mOjsrpGVN2hCAyvCshEB3wgxzDIXki2MFZPwV4oj1UbYhqtxk02735R4GtQSj04bZflEhkcqu7QNpgMu7D7YkJFM0jDvXIOOfzZ+G576wOd3NWKuUDc80gXvy7Q0437JN1tV5wQpitNg==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"","Date":"Fri, 16 Aug 2024 13:09:23 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ,\n =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n \n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com> \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com> \n \n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-ClientProxiedBy":"SJ0PR13CA0099.namprd13.prod.outlook.com\n (2603:10b6:a03:2c5::14) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|SJ2PR10MB7810:EE_","X-MS-Office365-Filtering-Correlation-Id":"4ebc9915-1a3e-4365-2d90-08dcbe162fe3","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|1800799024|7416014|376014|366016;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?Y8v3of3PMZHU8bCUZpOSxhhBBYzaz84?=\n\t=?utf-8?q?sDTxeRnbbn8uXuZOz3wc3yXJrqk4pI468lLKd7bnI0oCVPY/GNj0tC6sm1WiUM7+a?=\n\t=?utf-8?q?6o7MK4d5dYXcpDg4X3UfapcJElU7ga3SJbjdWbFbqEwt2EHYTz95fJgPH/cLqmP1S?=\n\t=?utf-8?q?DLimnuRMITt0gZk9IGXFvpEcMrXL5byKTnq1jBj6Esnx4FmXvR6gqHpHrK/1p08Ji?=\n\t=?utf-8?q?f01nqtbcEz/COwTQKLrtOVe11iRgwpChDURAXcbdbfi2KMp03CG1q6z4lLZnbHXl/?=\n\t=?utf-8?q?NZ1AzLMwk7a0JcNnj2MOaa2drfX9EiPFi+p45/AIAv51xfqPQNIOduNRJsm+Gwskf?=\n\t=?utf-8?q?/rezqehKhoqh4VQoqKCUAgVoDyiw3pu3gsybVYyCdSLELp6swMvZsULg3cDWDqOwX?=\n\t=?utf-8?q?2xhnRoGNMun7NWiI/h2GC+cWQa3/7AKOmiFWq+Xl6tH6Qi2dm0BQfmd7bNZPhvxCb?=\n\t=?utf-8?q?ZJdXB0rsNG4wr3SzOy5D7jmNMgIFC6V5AE0bE9+yMMOiacUEE8BoiaZIvMVCYOqgg?=\n\t=?utf-8?q?IUnourO+WnyvmgCnom4r+rbmERwlqsyq02v8qbz180UicmyfdOtKNJbybiSQ/jGou?=\n\t=?utf-8?q?JNbzZr6RA28wWT/hbBUbJN/W5IVDwsYpaxDYfTPwTh3vx1XCmlFc8pa0fNu4muKOh?=\n\t=?utf-8?q?yKhzJfkdDwLnmWRxM8BT0ya2hasVPK9X94SS6WQZ5xFzYlmMI/Y4e2Ec85r/NlXaW?=\n\t=?utf-8?q?PXZUjUhdlOvCpC/mzDA4iBz5NFgnAM/rJJ3MqHe+RrWyp0byOghH3X1z1uB8pa2cs?=\n\t=?utf-8?q?lPUkuav0TyDlmZixlqMfpJUd68aHIz3GovPZGPxwhRBe4IP1SYbJS95KNLhHNBn2z?=\n\t=?utf-8?q?BOMrLZulyQuh/haeY5Y04ahH++fsb2mfLYsbPhLe14tA4qF8QmmGqOzu762PCk6g2?=\n\t=?utf-8?q?eyYdNhGrIIFCA2m7z8sm9AuDUV1xwdS4tT1lI9ziVcds7t/A+ot3mYhrUckr6jANE?=\n\t=?utf-8?q?tgA82B6qWvRUPzQBBJWkXKfodTpoL+wuPxJ7uX7APyYIKHBPdR/w47b/TG4ahsDUk?=\n\t=?utf-8?q?Tm4QfycNTu0xg8LXkKRxehy7NpVI/wpyOVFa4NbGN204mtrJIJs90WZ4xM99e4ADD?=\n\t=?utf-8?q?ih7C8UbuRD0hSaHCGw4SNHWfePiyfN32DqPj6mGxiJKaO3LPWAal/4flTwdWWNJPL?=\n\t=?utf-8?q?Rb0kvrTfq62xcilSqVbWZpjHSbKcTN59ED2k6Jx1RRuxkKtci3d/jkyXV64LZsYvi?=\n\t=?utf-8?q?aeRAPDQw2xjqisjJ8447e9cWqjLmd+WvwAJOkhETW8nNT97g4LJMdCr4YOh5pCUbz?=\n\t=?utf-8?q?sk2Va6QmBRgHsgUOs/wYggvEL3dWJujgRNQ=3D=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(1800799024)(7416014)(376014)(366016); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?cYbzdbjWplyuCVdivI91C0ohxhf0?=\n\t=?utf-8?q?430mx1jGpV4fY69ntOClH2ZkmXGrvwRrkfyaZjaR7gOvjBrR17/cJ3B6rMUSr4CXp?=\n\t=?utf-8?q?cwrIB9vJGqkVduVNXjMq75rrCMzOVEhI/NYNLkKjgnyHW57ZPLkHnbV+5adhwrSOP?=\n\t=?utf-8?q?JLG2a08U9HY+9Nf6DY6jPbemi49+jIdp1857kbNhgEo1oOPB8doFUIfWfh1CNiIEc?=\n\t=?utf-8?q?mnqSgULY4ZSzrdItmN+E8roSUpYXgJQ8icT31vZ3WhbtP147ptEg+WvfGaM/cosXF?=\n\t=?utf-8?q?+uP6WR8b4y14+x/7IWnDrtpvCXhjTvXqmdLITCUQ9s0rdu7haOUVnSsrZ3+PG4/3Q?=\n\t=?utf-8?q?2IRXf1S3bSNLthsnADhFb3WMFBlBJ5TTnGZpeZVhKV1sOGNZdkau3yym0Dl7xqgU3?=\n\t=?utf-8?q?UdzOgjGPyazKJIGqrdYnoskN4gBaUwkZm1rtZbS+OhcGYAwvWWLxL3Ur4TQYGJ1IF?=\n\t=?utf-8?q?cjuqG6Ch73tlH4oPszX/4dcNqUbWXFXQxZckz241is2bW5kG3pYb4KPjGcq7KhmOJ?=\n\t=?utf-8?q?gXlhi/CzTlM4twMcQ9mYGvxvtxwMPoxKYsclfhQjTVdF2M4fkRw2G+7jONyBgKlG0?=\n\t=?utf-8?q?ZBmNd/BzYBmlpG6GgOWy2fp/7eo8kfEUkMful9ahGY48EGF1dnzmDTluMCX3XS3Bs?=\n\t=?utf-8?q?VUGvVrdMydEbEZtQWfz05K0ebhwKusaOGezjcU9G0dN8JbQqyGmtKWpIcfhZl07GH?=\n\t=?utf-8?q?VfFtE3LaznhaFJiFH4VC0ap5oUG6D3mEP+zN8dP1+zxL8RAs6328804oxF1C6KgYM?=\n\t=?utf-8?q?jJS5DfY+xeyWAoEIZOONg14azWHli6CABE0fZWOFAjzBR71wHD2B9IqcifXWY3Ahi?=\n\t=?utf-8?q?HMA6h3kf2M0MQ4XV0IQueHuTybdrAl/O0+ZZkyGaRK1cw3EvySFXME1OJyoBFFje5?=\n\t=?utf-8?q?AwHo63onOUQQbb/sNEF2g5iRVDGCoSxlsmmIdCnAr9KisP64lPIA+vB5EuFWOk+2z?=\n\t=?utf-8?q?vaU/T5Zg5Pmj3xRcReX65/9Bcd2JHAAUV36TDZuQq8WVk+ZT0nu3aHSwYKmszzpv7?=\n\t=?utf-8?q?TnG9L4RMDRTB9ulMKw1YHRVAqzZtQyQxumJ2sl7NVwo2OV/bB8/O2/SvzSgBewSXb?=\n\t=?utf-8?q?4Zq/WIhI22t0toNdDsB/sKQ/XjypQI1UCdBPBAmF3eFs3ora6J6hOECHonOscNM+x?=\n\t=?utf-8?q?KXlldTcjZQ2NXaFgarRE1xdHhYOvbUrzmt2Z+CiNAd+fLAisL98kawT2kuyHio6bF?=\n\t=?utf-8?q?HLwBiHIaIoJeyXQ6DsrFXiDTo+pXggaCSGoEpkpQeVFnWAMpEqw9GYvi9FH/Y4pLr?=\n\t=?utf-8?q?irjE+4zcMSe81G61vjR6bFxVQqym2NHQTncx0eYCOV6Gw3oB4s1W1Oq7DT6IO+IBr?=\n\t=?utf-8?q?qVEm4+9C6PpPOPK+kXO9BRtJdgcW9C+VwPD8qiDwVBN+ZD2eyTi3r0WnnE3wjjhHl?=\n\t=?utf-8?q?r/z119dtKJ2c9WERRXZjAUbfOhHLdE9HEnwFkCjgaoCD+hxAApF1bVliowPIRpJ27?=\n\t=?utf-8?q?EG3RXqIXL92zI92ZdJXwNgYWmyZ+P/BR49i+iA0imElUbmYTPkbQpjhpVKiceRZeq?=\n\t=?utf-8?q?9NpT9ydgvEFoIZyqoVO6kVw67PFGrZfRew=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n 0+5mSOKuCqLZLrnKMw0jwVN1Qhq0YTYjzx7GFVaEEd6hGESvcOV8ETXAdKIFMnflB/aZ+mc0QAg5AhY1kQ8mwqWO7bStM+hXSq9JL325SLTGt+cTpBhg7xynRxBED0ckjGHrnzh5ZMhJn8oAAqMea660S28zaMA/QOHk4ZlLJ9Kqj1uTWG+EkZWraD+ogdgsvqrjDOH2wZ6fjrizf+MkxB0dKepnMyRUnTTXmr+P2MhDhdpjQZkoQ9/oaaqLsu2++a2nxRr59+S7BEd183ZEEhPARIc5EHQkAuthra5hHThzSpZPLQAXRlL0THDYJ3rLtjXAEsCTh4bst5NTbDNXhkVRYoBW+mo94+/5JZSFUhbYrF7020PMeaqqZUfzoBqdOhtmPsv+IV7cZB+yCQs3viVlHIoE6++Uy2RNXno6hvZ/tP1qheWkJDrNJNKVofylOPFOzW0L7mCSe2QncBzEjjdD8xB4prN8ydo9DwaAsE2r1D74ZZui9M+T9Mpc1M+Wwg9gzgVAx9JSKPmlaZMLy5ZF4LXxC7MpBhoI8Z0ilipF2A+tcVRClAhpObeADHH2aXC+Kx6fcaNGW+iO/l6XIbdEVOlBCA05nQoGqw9Z27M=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 4ebc9915-1a3e-4365-2d90-08dcbe162fe3","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"16 Aug 2024 17:09:28.3896 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n GoCI0HdYc00E2SMaKs5uA53xRZM8LFEmJSdw6IYy82b5u+UEXz1/goQoCRlkNQmZDg7xPmb61AuJNMqK49IxzeyLMhsWQQsnFBTB8Z8c3dI=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"SJ2PR10MB7810","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-16_11,2024-08-16_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 spamscore=0\n malwarescore=0 bulkscore=0\n adultscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2408160121","X-Proofpoint-GUID":"org5z07wTK1YTLSCgzSdBeA4lZquqZkt","X-Proofpoint-ORIG-GUID":"org5z07wTK1YTLSCgzSdBeA4lZquqZkt","Received-SPF":"pass client-ip=205.220.177.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com","X-Spam_score_int":"-27","X-Spam_score":"-2.8","X-Spam_bar":"--","X-Spam_report":"(-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3365212,"web_url":"http://patchwork.ozlabs.org/comment/3365212/","msgid":"<28d2f43d-4354-4e12-87ba-93e33806b5ab@oracle.com>","list_archive_url":null,"date":"2024-08-20T16:28:39","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec (reconnections)","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/13/2024 4:12 PM, Peter Xu wrote:\n> On Wed, Aug 07, 2024 at 03:47:47PM -0400, Steven Sistare wrote:\n>> On 8/4/2024 12:10 PM, Peter Xu wrote:\n>>> On Sat, Jul 20, 2024 at 05:26:07PM -0400, Steven Sistare wrote:\n>>>> On 7/18/2024 11:56 AM, Peter Xu wrote:\n[...]\n>>>>>> Lastly, there is no loss of connectivity to the guest,\n>>>>>> because chardev descriptors remain open and connected.\n>>>>>\n>>>>> Again, I raised the question on why this would matter, as after all mgmt\n>>>>> app will need to coop with reconnections due to the fact they'll need to\n>>>>> support a generic live migration, in which case reconnection is a must.\n>>>>>\n>>>>> So far it doesn't sound like a performance critical path, for example, to\n>>>>> do the mgmt reconnects on the ports. So this might be an optimization that\n>>>>> most mgmt apps may not care much?\n>>>>\n>>>> Perhaps. I view the chardev preservation as nice to have, but not essential.\n>>>> It does not appear in this series, other than in docs. It's easy to implement\n>>>> given the CPR foundation. I suggest we continue this discussion when I post\n>>>> the chardev series, so we can focus on the core functionality.\n>>>\n>>> It's just that it can affect our decision on choosing the way to go.\n>>>\n>>> For example, do we have someone from Libvirt or any mgmt layer can help\n>>> justify this point?\n>>>\n>>> As I said, I thought most facilities for reconnection should be ready, but\n>>> I could miss important facts in mgmt layers..\n>>\n>> I will more deeply study reconnects in the mgmt layer, run some experiments to\n>> see if it is seamless for the end user, and get back to you, but it will take\n>> some time.\n\nSee below.\n\n[...]\n>>> Could I ask what management code you're working on? Why that management\n>>> code doesn't need to already work out these problems with reconnections\n>>> (like pre-CPR ways of live upgrade)?\n>>\n>> OCI - Oracle Cloud Infrastructure.\n>> Mgmt needs to manage reconnections for live migration, and perhaps I could\n>> leverage that code for live update, but happily I did not need to. Regardless,\n>> reconnection is the lesser issue. The bigger issue is resource management and\n>> the container environment. But I cannot justify that statement in detail without\n>> actually trying to implement cpr-transfer in OCI.\n\n[...]\n\n>> The use case is the same for both modes, but they are simply different\n>> transport methods for moving descriptors from old QEMU to new. The developer\n>> of the mgmt agent should be allowed to choose.\n> \n> It's out of my capability to review the mgmt impact on this one. This is\n> all based on the idea that I think most mgmt apps supports reconnections\n> pretty well. If that's the case, I'd definitely go for the transfer mode.\n\nClosing the loop here on reconnections --\n\nThe managers I studied do not reconnect QEMU chardevs such as the guest console\nafter live migration. In all cases, the old console goes dark and the user must\nmanually reconnect to the console on the target.\n\nOCI does not auto reconnect. libvirt does not, one must reconnect through libvirtd\non the target. kubevirt does not AFAICT; one must reconnect on the target using\nvirtctl console.\n\nThus chardev preservation does offer an improved user experience in this regard.\nchardevs can be preserved using either cpr-exec or cpr-transfer. But, if QEMU\nruns in a containerized environment that has agents that proxy connections between\nQEMU chardevs and the outside world, then only cpr-exec (which preserves the existing\ncontainer) preserves connections end-to-end. OCI has such agents. I believe kubevirt\ndoes also.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=MF+X7vbh;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=ZsMWscZ6;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WpFKz4cWlz1yYZ\n\tfor ; Wed, 21 Aug 2024 02:29:41 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sgRjB-0006ul-I1; Tue, 20 Aug 2024 12:29:02 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sgRjA-0006uH-I6\n for qemu-devel@nongnu.org; Tue, 20 Aug 2024 12:29:00 -0400","from mx0b-00069f02.pphosted.com ([205.220.177.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sgRj8-0002SP-Ep\n for qemu-devel@nongnu.org; Tue, 20 Aug 2024 12:29:00 -0400","from pps.filterd (m0333520.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 47KF4MhD028061;\n Tue, 20 Aug 2024 16:28:53 GMT","from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com\n (phxpaimrmta01.appoci.oracle.com [138.1.114.2])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 412m3dnpbc-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Tue, 20 Aug 2024 16:28:53 +0000 (GMT)","from pps.filterd\n (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2)\n with ESMTP id 47KGJOXP025990; Tue, 20 Aug 2024 16:28:52 GMT","from nam12-mw2-obe.outbound.protection.outlook.com\n (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id\n 414xjg0d7h-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Tue, 20 Aug 2024 16:28:52 +0000","from CH3PR10MB7436.namprd10.prod.outlook.com (2603:10b6:610:158::7)\n by SJ0PR10MB4733.namprd10.prod.outlook.com (2603:10b6:a03:2ae::12)\n with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7897.13; Tue, 20 Aug\n 2024 16:28:47 +0000","from CH3PR10MB7436.namprd10.prod.outlook.com\n ([fe80::dfba:d89:d13d:cd54]) by CH3PR10MB7436.namprd10.prod.outlook.com\n ([fe80::dfba:d89:d13d:cd54%5]) with mapi id 15.20.7897.010; Tue, 20 Aug 2024\n 16:28:46 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=LtC0iQZ0Fu19RZDz36UZsOSbnbI25pzT4lTClA87y84=; b=\n MF+X7vbhUKwn2yIomftoh1121fHhbgxu0Q7FXDntcxuj+rA4ntbcdY83eqmilucu\n 91Bla0O2G5rWgsGqWwV/sEckM77rzqe+Jn6we1K09/5s2D3PFTI0SIM/pQ3+3/Ko\n KUsTYudx/5NlDbJsSUrGih6tmb6Lv5yTKjhJxPi988UjGY1ZtLQ2Bm6EIrt1+xrh\n MBGgYrbXc1qFy502h/akFAc7XWHo52Lr3m94HO+Ee0lEAuynUqVykROtfecWWbVf\n 6NsxfJQq9DnXwO3mf6z9wfNcJGTinHq2FuqyKthcNT/Fr0BW8Kue8wid3ww/PEX6\n xbT8UE1P3HjCTvAp97wbuw==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=LtC0iQZ0Fu19RZDz36UZsOSbnbI25pzT4lTClA87y84=;\n b=ZsMWscZ6V9ENehF18CdNx8iONCkEtp3s8jXnF9r7o4WGONOf8FtKYP4m191LitMOkbZ5aOdkmNlSA0gmb1lzXbzWnoLeq8wUnn8potOIwyxPWE9Joho6/FPVaK7j2vNV4wpB+LZjp5894TWPKMnOVfhYcqmFirOBJXbK6cDjGwo="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=qxVls+BMImyT0qx+2bnGobbmRBX9iqhCnNXoUD1x4cK2vFFwxVe7dlRCHbUe8VRrXEkOb9W4P4McPa193Pg+RaodjgQzpvPVtCp6RCEMJVQmFExXde85/Z7YouvKiP9TZO8HAVEjuZfudngn/faLUzk3+3r0IsjsPhZYIWUdFgGETSYZPyGRiMs//YqB/froP4Up+L/PMcR4uEHr2gXrhrFtJwkuAQXIjY6H0mIWH6JU8PRhIBHdiP0uioLG72+utNnRUILUo43WvjmYpAtkqj5YOxo1SFTUW5BrQGYZWGgD1mLaLnOQEHhy3RN5bt7WM9L4dBjMJWpFP+WWxriI3w==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=LtC0iQZ0Fu19RZDz36UZsOSbnbI25pzT4lTClA87y84=;\n b=cJnKrvRQoCxI0UmY4WGdIZ3HZvlAErFM7nRXTsMPCeKYGMiFUVK9N+X07zZAKzG9wbUtbJVTVOoceVUjYH2/nC5EjCp5Bcu7Ic51NRA8cIAWCFZkOAW0TWT8nBowPVeF+Bn3Rt/hj21IEvmylqRCD+TT3yHrHREYgZ1Q7rYmSaUQjew/onxNlHD6iKftoURaoK+Zt+WPYJho/oqjFNQo7PQqQ21Mha7tCtKQJWHNXKEApwuQBu/eOy5VjdhfGTqnnJLVvk2QME5NkuJFV1qgHSEj4Ni89y6+lILbNyBMpRKuP6jMIxRfqGW+YNTlNWfJxQUssZoAvKFbwiEYJP47iw==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"<28d2f43d-4354-4e12-87ba-93e33806b5ab@oracle.com>","Date":"Tue, 20 Aug 2024 12:28:39 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec (reconnections)","To":"Peter Xu , \"Daniel P. Berrange\" ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <4f6200fa-c052-4295-b71a-c6deca11db9f@oracle.com>\n \n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-ClientProxiedBy":"LO4P123CA0509.GBRP123.PROD.OUTLOOK.COM\n (2603:10a6:600:272::13) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"CH3PR10MB7436:EE_|SJ0PR10MB4733:EE_","X-MS-Office365-Filtering-Correlation-Id":"f70a627a-d5d1-4691-2cb6-08dcc135294a","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|366016|1800799024|376014|7416014;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?vza5bZJNCJ0IYTRe2hlq1s9kQTxIf3x?=\n\t=?utf-8?q?9QnRjfvIakKAMuBcqsnKj+X4enHRGAU/8F3EhwnGY/P/+F88mQCszrHfJtkdJicY6?=\n\t=?utf-8?q?jczN3VCCNzLo5HlDth1g/l4jZYZw/NCtZTrG4Zr1WiRyBdCrAIeKb/lzaDdghFKd5?=\n\t=?utf-8?q?Crd1yQyyz2NJYTpla6xFcXX5peMxm9N7fPwQqcaWE9Go8XZSwNAKwEXRYFb3ch/m5?=\n\t=?utf-8?q?ANhx4MiLsUp8vPEjtE2mWgTFXB1YXK3pS532NiJZcYMhzATmUOnDaANZixmhNQkoc?=\n\t=?utf-8?q?zdwhmbOhxcVrwyLqK/QI0rnKd6b7bfaKHYZjqUWADQdfrmvzlbFgelz6OQzjw2EWG?=\n\t=?utf-8?q?OI2ejxy0P7XDBc9IWNyloCFa70ZEb/WN/Ek5YV+DEob34NoXaPhgaewsFUZGwE3QU?=\n\t=?utf-8?q?OSeGZ2n/VUbCTiwrqoWpH5AO2Im2q+sLt5GGgh47Dpg6XbTIl1eXvtRiyxuG/v0K3?=\n\t=?utf-8?q?5jVqzPPDobh5/pXPuh5pgSDnw/n0i30f/a72TKLxeoh6UYO92rDT83squNSheLwwW?=\n\t=?utf-8?q?RK7lAYw05nVqHDx6S0pRLl5HaWWbFFtC9acxLThiThlI5jogumh/WxNLCVwhdrwoA?=\n\t=?utf-8?q?q5aDdOlYpSr/lFhNTtfSkjBBuzSaB15LSS7mWw4a0Q1EKn97dGGXck3HE23EkzsCI?=\n\t=?utf-8?q?Q/RnTgJc4hRTUZXvuh8kNCTnTdU+BX1F7jIUSnpAiHTSMShcTznP3rnyfMWGTELoC?=\n\t=?utf-8?q?AUgxV+GUe4GcjcJT9nAPpYe5fjYWZbX/PSmSeEHUEK8Vvx4FYsKq5UkWKNudALj7K?=\n\t=?utf-8?q?igeLSmOSuMz1gI16OPSyrL3hnNhAnNgje6Pa6EtWRyyD31umomAfTU2LJFjqb0YuM?=\n\t=?utf-8?q?XTj1Ui8s+ogwDtEbd8yH8Y/9sVKI8d8mgGOvRCNUa5uqHENLesglxiajfiWdtdaep?=\n\t=?utf-8?q?K7BgPj9Yt0TPpnwK73uWiEA9C65aycLlBQTFeit+yBWH6uyH4Fvu3JtHvCSdx14yP?=\n\t=?utf-8?q?rF6wEWDopzEe7/09mAdRzIuMrDUYF359lXqdwa6SYlef8N72cftobKRsMgYfvm3zk?=\n\t=?utf-8?q?z4SBBBC9Oz4tuqpE2DEHN80L59V8lc0ChO/eRAPhIL5q+NSdk66cEJUhJvoeQq67L?=\n\t=?utf-8?q?WBxKZT+dhXQX1gIhp6ehcrAIl84WDa8/m0ZXc+oszUIuURVFCe2e9y0xa9Z13FkkT?=\n\t=?utf-8?q?NyWBij9D+8fAD1RV+p0Vnvt2YTg0RERhPT/0DwqTbga+nVi3ALvaatlfYRXImu9kb?=\n\t=?utf-8?q?TiZV4BAzBeUNFPy6zqrLFyeBEIgXKXpw97vthzCyY4fFNh0rr/w5TaXJjt2NmEgjM?=\n\t=?utf-8?q?f7AQ7p4VSGyxpM4o9qUXlJW7MmVu/WMndVA=3D=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:CH3PR10MB7436.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(366016)(1800799024)(376014)(7416014); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?5nAJPMvGI8p97VpV413uNRV7cXA8?=\n\t=?utf-8?q?zR1UCZmajPF0XyHERHqSobawnk34AgSVl/q+Q5vgIULVHt/b22tvEB1L8dp9DYYh3?=\n\t=?utf-8?q?+jHHPjJVAH2aRVriSGa7H3bBtirxH4sfn83VlV2jQHDDGDgn8DmAeYW/Z//9H0y/y?=\n\t=?utf-8?q?l+p7NL1ooRh2XMj4B/239T7UfcghflvBET8u/Kj+kmpYzyWeS6WSlUq/IbFQ+uk5O?=\n\t=?utf-8?q?3KuRocPvkyifU3FR4M/VTaMyt9zG0MP1eF22SgShp1rKz5VCje6DteFZveYqGWokF?=\n\t=?utf-8?q?/VOWXsntqu7SRzVpjGSVBePBnkeH3tSzgKzf+OR3b41+sn9g4Edeo/g462Yp1QByG?=\n\t=?utf-8?q?4YPE5rL7s2ILevSRjDvIQGmsYdzJbCUJls9X8aPo2jC4I4kJWzwlKphvGuyWXh0p/?=\n\t=?utf-8?q?9eKU54xuASBA4XWJQI/NF/YWXt3fWaHhvHP33yNUHKzm/0AgWLgakIXzpTtOJajDM?=\n\t=?utf-8?q?mqHnsnQ4Vv4xGDjKyku+WCilm74jeO1N4g8nT3OpLTrJ7bTT5bk6Y3ZBPCGEqo1yU?=\n\t=?utf-8?q?107TsVkVWtEgNFv61g0g+ARHE0Q2gP4+rLVd58/GzOcJ3bRvLDNJLcZaWD+RYb1LP?=\n\t=?utf-8?q?Izo0GSBks+yXZRXP09LmaE1zGsPT26PyK8WUBG0BdKwGteHgrFiMynTHJy95vY7+s?=\n\t=?utf-8?q?fTVUV5GgQSECpfvWY+GhMstVaTlvYVehKZ7JfHX9r5gfd4P/nxZ+IHCkOVBoOOZzh?=\n\t=?utf-8?q?+ZAo9oNq20KKCIkTF6Ah/q/cCbdsgY0ThLzTl+8ukRRNK/ZDSfrR10BMxNCXQN9pD?=\n\t=?utf-8?q?ZzXi4JllYxJ0KckvobyjJFVG2kBT/44sKhiwYPjQ+RbnoY6DFdcaK0OZ0RAHPQqLa?=\n\t=?utf-8?q?EH2pYB3yZ3LZz2ua2wA7AS1/ndP9vlGeq2q97BTDbNwvThayjFMu/tsy76qQV2iMt?=\n\t=?utf-8?q?5jEErKEzZ3+gtQdVOdLkvEDn5TPt2BiKzPv/xOlv5pP/KRUv3Z2CkSQzEgfYwN2lz?=\n\t=?utf-8?q?l5jdhxrg0Z0aD76kfANHq8xs8Vo0Ewgn6bmtQiYRJo/cnpiJaOB0/cbwndNjfkd+S?=\n\t=?utf-8?q?OGl/0BehJF+oT9f+krS+ziOkWNcUm8GrBc1hm8eM/FK5QuKg62BHmh/OAqYkliu/+?=\n\t=?utf-8?q?TWwUdVEY6R4oN8/v9et0gU10njMPEpq4j5FL3Ii+8kis7tFuZR670cSgicncR/0z8?=\n\t=?utf-8?q?TkNMYrYOs8RtdwdcU5s3vAxxo/nd0wqTPyS5uvR3VtxbCgmxaT055mrq7lvLSpWF0?=\n\t=?utf-8?q?Pd0NMGtfAha4aKBhgRtiUDotc2FCLVdjv+f8mLyCNQ8OMb6VENGhKZvb071K53/JO?=\n\t=?utf-8?q?h6GyO18LPTUKOw33SfhRbEXEZ1leqA2g9/Axwwi7V95jUAUIxoYvJSMXtPyP7IdFy?=\n\t=?utf-8?q?6aUZhVZSoOPi/wr9sjAyNdDFrzKI8zPwgpbSbVcvKUiGZDxPe6u6ydJ/v3JOEqbK2?=\n\t=?utf-8?q?BQSlDshpuw3KIaCG1MihGqK+Ihp3D4fha/5AP8m/vMlfCoC6hklSQn3hNbWI0LdzI?=\n\t=?utf-8?q?10smZGVrPTGlqnXpGnbx1u7rqIXarNFRWJGDZM6vwEMDMxo1RscjRtP98c5g9VzNB?=\n\t=?utf-8?q?99k7KD565skgJ5KkADsjo+cD7zf86ZTlKw=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n kkctdMYnXVA2yzBc9thW91UVrf01E0OMERnRT2ly9/KD4LHnQf0HRssfK1FEnAeMjY7AdgcpmJcQ7Bh9gL7F2uxeADnj+ZYn/Gl/CB+thQwSzbBOZSJfbAhdapJlmUn5VOGCQTza72dHAyTAQiKE+6LqURjMoSgpdaLmIWrah3YNnOBAH5BoCQ/6KrqvTq086ym8f8BQVkI8J49cBp/Ufwncve63AlQiIINfr+R7ONRJuJLyoQL5xTJwvkHS1+hSX4MZwO5FzHF30UbPIWjM3sNKT0xIhXgF0RMRSDASffvhcPacz0L2OD2FQFIwsnfu/eHAt3U7h9zUAcGbs0UFYYRFzrMvUDAzMjjufBvPY5odPxvOM9Qp78BICKnvrL66CjpFBPXme8oz4gsQTV6wDnYuszc8fpEQ8Ix740UGVSDgZLdIus9qXqWY988xU+pfzUwhpaIEdRN1qtl1zTAG5JjkLxmWxgKzOfeEcsrLfSnLoeRCD6CR9SpcbhrWd/vFIxcwYH+QHEWGWt7ht0lmE0YXX/w2BeZWiZ1AlrpbSY+f3FQNtvNHkqRQmoMaefhm+yR93LbZzJ0LeyFtxysAf527rmIA75OL52buzcopHGU=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n f70a627a-d5d1-4691-2cb6-08dcc135294a","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"20 Aug 2024 16:28:46.2292 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n 89A1A6bVBt+1ERhKBtH+MPSsvjMmwr8BmVyS5xPYucY7n/yKbFaFfQ77G7gQvR/TGWhj82fYc9CeH7TqPoCWWsSkcn3A8pehWlsw3WMpipE=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"SJ0PR10MB4733","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-08-20_12,2024-08-19_03,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 phishscore=0\n adultscore=0 bulkscore=0\n malwarescore=0 mlxlogscore=999 spamscore=0 suspectscore=0 mlxscore=0\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2408200122","X-Proofpoint-ORIG-GUID":"KHy0QAj5JFyYL_cxCR2XE5gyUGci7xmX","X-Proofpoint-GUID":"KHy0QAj5JFyYL_cxCR2XE5gyUGci7xmX","Received-SPF":"pass client-ip=205.220.177.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com","X-Spam_score_int":"-27","X-Spam_score":"-2.8","X-Spam_bar":"--","X-Spam_report":"(-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3365936,"web_url":"http://patchwork.ozlabs.org/comment/3365936/","msgid":"","list_archive_url":null,"date":"2024-08-21T18:34:08","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n> On 8/16/2024 12:17 PM, Peter Xu wrote:\n> > On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n> > > On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> > > > On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > > > > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > > > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > > > > file, and only 1 of them can ever have them open.\n> > > > > > > > \n> > > > > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > > > > going to take the file lock).\n> > > > > > > > \n> > > > > > > > Maybe this is about something else?\n> > > > > > > \n> > > > > > > I don't have an example where this fails.\n> > > > > > > \n> > > > > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > > > > argument, due to this code:\n> > > > > > > \n> > > > > > > blk_attach_dev()\n> > > > > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > > > > blk->disable_perm = true;\n> > > > > > \n> > > > > > Yep, this one is pretty much expected.\n> > > > > > \n> > > > > > > \n> > > > > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > > > > \n> > > > > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > > > > the socket path before binding on the assumption that it is stale).\n> > > > > > > \n> > > > > > > One must use a different name for the socket for second qemu, and clients\n> > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > \n> > > > > > > > > Network ports also conflict.\n> > > > > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > > > > I forgot to promote.\n> > > > > > > > \n> > > > > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > > > > \n> > > > > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > > > > stable across migrations, where the host ports can be different as long as\n> > > > > > > > the host ports can forward guest port messages correctly?\n> > > > > > > \n> > > > > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > \n> > > > > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > > > > cpr-exec does not.\n> > > > > > \n> > > > > > Right, and my understanding is all these facilities are already there, so\n> > > > > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > > > > in Libvirt or similar management layers that supports migrations.\n> > > > > \n> > > > > Note Libvirt explicitly blocks localhost migration today because\n> > > > > solving all these clashing resource problems is a huge can of worms\n> > > > > and it can't be made invisible to the user of libvirt in any practical\n> > > > > way.\n> > > > \n> > > > Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> > > > supported local migration somehow on top of libvirt.\n> > > \n> > > Since kubevirt runs inside a container, \"localhost\" migration\n> > > is effectively migrating between 2 completely separate OS installs\n> > > (containers), that happen to be on the same physical host. IOW, it\n> > > is a cross-host migration from Libvirt & QEMU's POV, and there are\n> > > no clashing resources to worry about.\n> > \n> > OK, makes sense.\n> > \n> > Then do you think it's possible to support cpr-transfer in that scenario\n> > from Libvirt POV?\n> > \n> > > \n> > > > Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> > > > to consume it (as cpr-* is only for local host migrations so far)? Even if\n> > > > all the rest issues we're discussing with cpr-exec, is that the only way to\n> > > > go for Libvirt, then?\n> > > \n> > > cpr-exec is certainly appealing from the POV of avoiding the clashing\n> > > resources problem in libvirt.\n> > > \n> > > It has own issues though, because libvirt runs all QEMU processes with\n> > > seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n> > > and thus don't want to allow it to exec anything !\n> > > \n> > > I don't know which is the lesser evil from libvirt's POV.\n> > > \n> > > Personally I see security controls as an overriding requirement for\n> > > everything.\n> > \n> > One thing I am aware of is cpr-exec is not the only one who might start to\n> > use exec() in QEMU. TDX fundamentally will need to create another key VM to\n> > deliver the keys and the plan seems to be using exec() too. However in\n> > that case per my understanding the exec() is optional - the key VM can also\n> > be created by Libvirt.\n> > \n> > IOW, it looks like we can still stick with execve() being blocked yet so\n> > far except cpr-exec().\n> > \n> > Hmm, this makes the decision harder to make. We need to figure out a way\n> > on knowing how to consume this feature for at least open source virt\n> > stack.. So far it looks like it's only possible (if we take seccomp high\n> > priority) we use cpr-transfer but only in a container.\n> \n> libvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\n> and change namespace operations. I have a patch in my workspace to be submitted\n> later called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\n> libvirt to block fork and namespace but allow exec.\n\nThe question is whether that would be accepted, and it also gives me the\nfeeling that even if it's accepted, it might limit the use cases that cpr\ncan apply to.\n\nWhat I read so far from Dan is that cpr-transfer seems to be also preferred\nfrom Libvirt POV:\n\n https://lore.kernel.org/r/Zr9-IvoRkGjre4CI@redhat.com\n\nDid I read it right?\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=GnUNthju;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4Wpw4G2tRYz1yYZ\n\tfor ; Thu, 22 Aug 2024 04:35:10 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sgqA6-0006c7-BC; Wed, 21 Aug 2024 14:34:26 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sgqA4-0006bc-F8\n for qemu-devel@nongnu.org; Wed, 21 Aug 2024 14:34:24 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sgqA0-0005xW-Cy\n for qemu-devel@nongnu.org; Wed, 21 Aug 2024 14:34:22 -0400","from mail-oo1-f69.google.com (mail-oo1-f69.google.com\n [209.85.161.69]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-335-882NSHjtOQa7u49iLtj7BA-1; Wed, 21 Aug 2024 14:34:14 -0400","by mail-oo1-f69.google.com with SMTP id\n 006d021491bc7-5d601a9374cso45079eaf.1\n for ; Wed, 21 Aug 2024 11:34:13 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n 6a1803df08f44-6bf6fdd8c22sm64142496d6.14.2024.08.21.11.34.11\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Wed, 21 Aug 2024 11:34:12 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1724265257;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=nBlvvD96RzO0MQrzOLGV6+BJW4QxBoCf12C91mRN7Ec=;\n b=GnUNthju4bxTpXdGGNsggev8Ht9lPFKJF9tvaMDqqRVzChARMb1lpDSMQq6L5oeN+xaGlr\n viwyuu29yL6axKodmopdsOwVf6vbe9P6D6snUYmNBvwbT82ohyPYnj/J1aWO+JVIRiUsWN\n 8AoDEY2yDyUuqx/RW4UWSoXFdGYW3rw=","X-MC-Unique":"882NSHjtOQa7u49iLtj7BA-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1724265253; x=1724870053;\n h=in-reply-to:content-transfer-encoding:content-disposition\n :mime-version:references:message-id:subject:cc:to:from:date\n :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n bh=nBlvvD96RzO0MQrzOLGV6+BJW4QxBoCf12C91mRN7Ec=;\n b=t4eSYuCmlNp9+nOITMDmXbDpSYUp64cwDpK2eXUhp9geiB5Xbx0kjH8Ag1obmpiejU\n 1kpQk3+Zp/G3vrq0WMHXkA1zHOStZhWtq0d56FfUYQTZmPo4MfUQNPh0eDRwMgKlZC28\n vBmPmOWqlQgkud5y9YEjHIHen1DHR7Spi/MSb9u/QZQaTXT01ZAFzVF+bBhDkYAIgqRy\n iqewxTFwoeG2UPJDbQN3nLzLA5/2hJs30REDNtXeJmJaTepnzFTaZytOEzH4ujtC2tdi\n H+SqMlf6XrxCSzyp657gk/lPx2j2GxV8/D4opIvZqZ9XnZAlQrMQti2zxxMXylqqp3Xh\n +cPQ==","X-Forwarded-Encrypted":"i=1;\n AJvYcCXxRwK1zu8FmHgw1pp11CZHylcgQKL9YOtPD3whZgDDUbhTiTbD/l27hfLquJWDhj1xu5y8KIm9aE4l@nongnu.org","X-Gm-Message-State":"AOJu0YyQj15ktdSu4sxLjL5EgCB0c/C8Xlin5MP0l+ukQilJBGyP/z0G\n P0mhuyfFpTohi6g9KdzrjtxaY6EHoLqGJYnu2iuMnmSHC1pUnTxyMk4RRaZDdWkEbaIZy0i+CZI\n vOKALbPL/avCemv1/K1QXqScDQVTP0MDU+Uuayb3ehFcYJIo6kRcY","X-Received":["by 2002:a05:6358:419d:b0:1af:15b5:7caa with SMTP id\n e5c5f4694b2df-1b59fbf00eemr407596455d.21.1724265253235;\n Wed, 21 Aug 2024 11:34:13 -0700 (PDT)","by 2002:a05:6358:419d:b0:1af:15b5:7caa with SMTP id\n e5c5f4694b2df-1b59fbf00eemr407593455d.21.1724265252791;\n Wed, 21 Aug 2024 11:34:12 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IGm5eFF9pKBZTJDGErbyaokV7D0e7Dp/YTyBK0RH7e/7I/s8Uu3RQB5GG5Yg+hrY/upNGoiyw==","Date":"Wed, 21 Aug 2024 14:34:08 -0400","From":"Peter Xu ","To":"Steven Sistare , Daniel =?utf-8?b?UC4gQmVy?=\n\t=?utf-8?b?cmFuZ8Op?= ","Cc":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ,\n qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"\n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","Received-SPF":"pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.138,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3374329,"web_url":"http://patchwork.ozlabs.org/comment/3374329/","msgid":"","list_archive_url":null,"date":"2024-09-04T20:58:14","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 8/21/2024 2:34 PM, Peter Xu wrote:\n> On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n>> On 8/16/2024 12:17 PM, Peter Xu wrote:\n>>> On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n>>>> On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n>>>>> On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n>>>>>> On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n>>>>>>> On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n>>>>>>>> On 8/13/2024 3:46 PM, Peter Xu wrote:\n>>>>>>>>> On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n>>>>>>>>>>> The flipside, however, is that localhost migration via 2 separate QEMU\n>>>>>>>>>>> processes has issues where both QEMUs want to be opening the very same\n>>>>>>>>>>> file, and only 1 of them can ever have them open.\n>>>>>>>>>\n>>>>>>>>> I thought we used to have similar issue with block devices, but I assume\n>>>>>>>>> it's solved for years (and whoever owns it will take proper file lock,\n>>>>>>>>> IIRC, and QEMU migration should properly serialize the time window on who's\n>>>>>>>>> going to take the file lock).\n>>>>>>>>>\n>>>>>>>>> Maybe this is about something else?\n>>>>>>>>\n>>>>>>>> I don't have an example where this fails.\n>>>>>>>>\n>>>>>>>> I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n>>>>>>>> the same block device, but the error is suppressed if you add the -incoming\n>>>>>>>> argument, due to this code:\n>>>>>>>>\n>>>>>>>> blk_attach_dev()\n>>>>>>>> if (runstate_check(RUN_STATE_INMIGRATE))\n>>>>>>>> blk->disable_perm = true;\n>>>>>>>\n>>>>>>> Yep, this one is pretty much expected.\n>>>>>>>\n>>>>>>>>\n>>>>>>>>>> Indeed, and \"files\" includes unix domain sockets.\n>>>>>>>>\n>>>>>>>> More on this -- the second qemu to bind a unix domain socket for listening\n>>>>>>>> wins, and the first qemu loses it (because second qemu unlinks and recreates\n>>>>>>>> the socket path before binding on the assumption that it is stale).\n>>>>>>>>\n>>>>>>>> One must use a different name for the socket for second qemu, and clients\n>>>>>>>> that wish to connect must be aware of the new port.\n>>>>>>>>\n>>>>>>>>>> Network ports also conflict.\n>>>>>>>>>> cpr-exec avoids such problems, and is one of the advantages of the method that\n>>>>>>>>>> I forgot to promote.\n>>>>>>>>>\n>>>>>>>>> I was thinking that's fine, as the host ports should be the backend of the\n>>>>>>>>> VM ports only anyway so they don't need to be identical on both sides?\n>>>>>>>>>\n>>>>>>>>> IOW, my understanding is it's the guest IP/ports/... which should still be\n>>>>>>>>> stable across migrations, where the host ports can be different as long as\n>>>>>>>>> the host ports can forward guest port messages correctly?\n>>>>>>>>\n>>>>>>>> Yes, one must use a different host port number for the second qemu, and clients\n>>>>>>>> that wish to connect must be aware of the new port.\n>>>>>>>>\n>>>>>>>> That is my point -- cpr-transfer requires fiddling with such things.\n>>>>>>>> cpr-exec does not.\n>>>>>>>\n>>>>>>> Right, and my understanding is all these facilities are already there, so\n>>>>>>> no new code should be needed on reconnect issues if to support cpr-transfer\n>>>>>>> in Libvirt or similar management layers that supports migrations.\n>>>>>>\n>>>>>> Note Libvirt explicitly blocks localhost migration today because\n>>>>>> solving all these clashing resource problems is a huge can of worms\n>>>>>> and it can't be made invisible to the user of libvirt in any practical\n>>>>>> way.\n>>>>>\n>>>>> Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n>>>>> supported local migration somehow on top of libvirt.\n>>>>\n>>>> Since kubevirt runs inside a container, \"localhost\" migration\n>>>> is effectively migrating between 2 completely separate OS installs\n>>>> (containers), that happen to be on the same physical host. IOW, it\n>>>> is a cross-host migration from Libvirt & QEMU's POV, and there are\n>>>> no clashing resources to worry about.\n>>>\n>>> OK, makes sense.\n>>>\n>>> Then do you think it's possible to support cpr-transfer in that scenario\n>>> from Libvirt POV?\n>>>\n>>>>\n>>>>> Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n>>>>> to consume it (as cpr-* is only for local host migrations so far)? Even if\n>>>>> all the rest issues we're discussing with cpr-exec, is that the only way to\n>>>>> go for Libvirt, then?\n>>>>\n>>>> cpr-exec is certainly appealing from the POV of avoiding the clashing\n>>>> resources problem in libvirt.\n>>>>\n>>>> It has own issues though, because libvirt runs all QEMU processes with\n>>>> seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n>>>> and thus don't want to allow it to exec anything !\n>>>>\n>>>> I don't know which is the lesser evil from libvirt's POV.\n>>>>\n>>>> Personally I see security controls as an overriding requirement for\n>>>> everything.\n>>>\n>>> One thing I am aware of is cpr-exec is not the only one who might start to\n>>> use exec() in QEMU. TDX fundamentally will need to create another key VM to\n>>> deliver the keys and the plan seems to be using exec() too. However in\n>>> that case per my understanding the exec() is optional - the key VM can also\n>>> be created by Libvirt.\n>>>\n>>> IOW, it looks like we can still stick with execve() being blocked yet so\n>>> far except cpr-exec().\n>>>\n>>> Hmm, this makes the decision harder to make. We need to figure out a way\n>>> on knowing how to consume this feature for at least open source virt\n>>> stack.. So far it looks like it's only possible (if we take seccomp high\n>>> priority) we use cpr-transfer but only in a container.\n>>\n>> libvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\n>> and change namespace operations. I have a patch in my workspace to be submitted\n>> later called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\n>> libvirt to block fork and namespace but allow exec.\n> \n> The question is whether that would be accepted, and it also gives me the\n> feeling that even if it's accepted, it might limit the use cases that cpr\n> can apply to.\n\nThis is more acceptable for libvirt running in a container (such as under kubevirt)\nwith a limited set of binaries in /bin that could be exec'd. In that case allowing\nexec is more reasonable.\n\n> What I read so far from Dan is that cpr-transfer seems to be also preferred\n> from Libvirt POV:\n> \n> https://lore.kernel.org/r/Zr9-IvoRkGjre4CI@redhat.com\n> \n> Did I read it right?\n\nI read that as: cpr-transfer is a viable option for libvirt. I don't hear him\nexcluding the possibility of cpr-exec.\n\nI agree that \"Dan the libvirt expert prefers cpr-transfer\" is a good reason to\nprovide cpr-transfer. Which I will do.\n\nSo does \"Steve the OCI expert prefers cpr-exec\" carry equal weight, for also\nproviding cpr-exec?\n\nWe are at an impasse on this series. To make forward progress, I am willing to\nreorder the patches, and re-submit cpr-transfer as the first mode, so we can\nreview and pull that. I will submit cpr-exec as a follow on and we can resume\nour arguments then.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=a+M+LS55;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=u3/5d49u;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WzZbh4JZmz1yg7\n\tfor ; Thu, 5 Sep 2024 06:58:56 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1slx5I-0005NS-F3; Wed, 04 Sep 2024 16:58:36 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1slx5G-0005Ms-5R\n for qemu-devel@nongnu.org; Wed, 04 Sep 2024 16:58:34 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1slx5D-0000s2-FK\n for qemu-devel@nongnu.org; Wed, 04 Sep 2024 16:58:33 -0400","from pps.filterd (m0246629.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 484Kek7l028538;\n Wed, 4 Sep 2024 20:58:25 GMT","from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com\n (phxpaimrmta01.appoci.oracle.com [138.1.114.2])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 41duw7vm5d-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Wed, 04 Sep 2024 20:58:24 +0000 (GMT)","from pps.filterd\n (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.18.1.2/8.18.1.2)\n with ESMTP id 484KetxJ036694; Wed, 4 Sep 2024 20:58:24 GMT","from nam12-dm6-obe.outbound.protection.outlook.com\n (mail-dm6nam12lp2174.outbound.protection.outlook.com [104.47.59.174])\n by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id\n 41bsmakxx9-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Wed, 04 Sep 2024 20:58:23 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by CY5PR10MB6264.namprd10.prod.outlook.com (2603:10b6:930:40::7) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.24; Wed, 4 Sep\n 2024 20:58:19 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%7]) with mapi id 15.20.7939.010; Wed, 4 Sep 2024\n 20:58:19 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=JScAbAjqiU6Mp0fssYeqlKNgirqEy981aekxhs35HBw=; b=\n a+M+LS55n7G8DkAwpe9TfKbuGkusPaQqgE+EpsISdJ8Ij7yjkrHtncwn6zXwm1Vm\n xWI2uhWawBsEEH6PfyS8ZbPapfnlY6hBKzmvjCP8za8nCOtj5HHYMIXkqJOJq1SB\n 53JFgsGNRnqt8Tol24pMI4nULJLi6H4M2nHl/nQspfFse2NVndlyk8u+gWkarcks\n EIXgECQo3jx8Nt31J6Ns+ljRlLUUCfvUIp1lnnUOgp0RpLYJ8XiEIYnr8/HFIjpz\n vLEpPw3D57WicLWpDum4A3wr+NewDDzWGXPcKEbgSMfbblEGNAhSYaPxPtOMMXSP\n AQSTgz56E4mFH1rFt1oUGQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=JScAbAjqiU6Mp0fssYeqlKNgirqEy981aekxhs35HBw=;\n b=u3/5d49u2vLLocA1WXJ4lIQogObdzgYsOzjkMqq7E+he1/r0gKBTd2SmpkRfRqtvJ481HCqq1gKAzRHCjaCu6K4NKEnbfTLg2ac/6qjr7maRbVeSREqwwiiPdy9qWfcq1yG+tfeHxj6Tl3PIFO7Cmu1qq8KSw7RlxClWKbj9p2k="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=Cy2DduGOSSpsh7Hcj8jG6BGgPlNn29vSsFnM7OsNLYqfXR5georzYZCcT0oYk14rAbwNLZjIZJRyXSCq2ZVcqb0YkrfcjGuY0qTFhIwkjSjyWJpLt7RnQw6fUeijaJYcYS9nbYTFeJoM01pdKWVcUa5kguHXpZLGWoNmkIytzbWe0gSy2XhlMj15f3dgsJj5c7AzJxuhgKa4IhkotI+4W9LUtrbab0UKtTWLfECTx/rW9xqp/d+CDVkR2m327RPW+ggBdHL8LGpuUIRHsxF6cTSbPZLpIRTlcg73FCU1Ag7+bLZDH893rWRDfakEoUoNHxAW9HEHl9kSvm/+Jlx/lg==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=JScAbAjqiU6Mp0fssYeqlKNgirqEy981aekxhs35HBw=;\n b=WH1W4riZNVBAf3Mev8IzqPvdT8EXDhQq9WjPc6CntDX+loIgZSHV9c1HXJ7vf1bUyZT75atZM+igBh+yeepmvxqe37NRrs89dJ5/p15qVnLNapp0fvwFWSHVPYa2q+KckggrII9TSK4Ua/OZPhPVytqAyVu/uAcBKcS0PrTFKvJKztBquWi9v8ARDVCoH2rl5xQZgVxSUyEsX5JLiGSxfyJ59XnBwznlF7J/wrHDhqlQLlQxlt1DR/Nshmc2larAnub6/xz999IjbHpfNaRhG+sae1xmqFE1HeLx/4QyuPqKxeKayZ8xRplqW6MSIU6c3NVQmsG/UxEjjM3tMKAsUw==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"","Date":"Wed, 4 Sep 2024 16:58:14 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","To":"Peter Xu ,\n =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","References":"\n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com> \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com> \n \n \n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"8bit","X-ClientProxiedBy":"BL0PR02CA0035.namprd02.prod.outlook.com\n (2603:10b6:207:3c::48) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|CY5PR10MB6264:EE_","X-MS-Office365-Filtering-Correlation-Id":"a06e11e7-d3e9-495b-52fe-08dccd244e62","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|376014|366016|7416014|1800799024;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?SrTByhoVBnf/U+77O5cL0xFyMtZDiAv?=\n\t=?utf-8?q?8T+cWiLKjk+neA/l1MhZVMl/ldVrfOFGWfHWLQkqAsofrIosTYbraWr3jQnK65Vv5?=\n\t=?utf-8?q?SXgEXkXna3j3mxHq46UnIln9kb7hSVcyfSPdhWpyXYAr9OTQ6J5s3gCkqkrn7WkK5?=\n\t=?utf-8?q?E4cTsFTlcTB3BikDQM8FN4kS7Vqgq+sEzQF6cmLfh+ahk/8uMjjh5bxwy6yxALhNV?=\n\t=?utf-8?q?5jduNiGOrT71cKE9ugJtTfJqbYGLDHSkYmaFgIsiToDApRuNmXQZnVYUW3Wxqulkj?=\n\t=?utf-8?q?X480Yl7pvIYxHXm9BnlbIegd3e2cm4T/ng18oHpR92pEJtwhlWYyF2r9iJFaB4/Q7?=\n\t=?utf-8?q?Oc4H4Svk2FKDOYdL1JhKQ7SwpcfvrdzKfQwteQv2ywx6lxCyOHn/xul8f+etLMSZI?=\n\t=?utf-8?q?apeckVD/nCUFQGEv6B8lqo7BbLnxjGk4EteIPy/sD4C97KKbeSKM2ZLVD7g6XaTlU?=\n\t=?utf-8?q?Ss/uNa4DuWwLzYp4eMoexlFzf0QM2ONyBwBDqTSS/k2ijCPDre53KSC5QHUAfm/7Q?=\n\t=?utf-8?q?jKYFN3NWf/J8Zr+p8vnDaZ4VOyUAN8zI3xCY1hcGBsauRchcdDedIpi4iqFxNndT4?=\n\t=?utf-8?q?cm88grX1FC2486Fog3OgBjPoWasRgWg4UWBgRSQ789VWhBjpH57SJMApSgi2U+zSy?=\n\t=?utf-8?q?//VaWkZGsOEurvKpp3Yl/1R/SMRorwDW/Njwodf8TygXKZmsaEp39VG/V4UJtov9g?=\n\t=?utf-8?q?LFMJiwVhmG87eNilRLgr/iAG2IXg3KIdWokObp4oVk/9KnV4nnNu88Fly1PUt0yXo?=\n\t=?utf-8?q?bkqa/ftAYzarGVQsgqHvRpWA9SFcm4DmoDlggE1RMk319bWSnOhBPo6Rpt5DNMIGy?=\n\t=?utf-8?q?6EEl+4LlCjpDu7V4m+mBsLHe3aJGil49SykDlsBCRNBdNXZYgwDZrNqOMPJZsQ7gg?=\n\t=?utf-8?q?OGrnzBV9kHYggIJhDPGG44eg/1l9XMVSJe4chLf9bJZyTOcIeY/CSZ05y9baKA4J5?=\n\t=?utf-8?q?+caqblfA6kDE6Y6TJcO0eSSwgZn7/wAELofzQqAVmOtz+KIj/xnVJCDvx5cAHH2jD?=\n\t=?utf-8?q?AMZ/nnU/qErcpufPe7oIl133Vcf0zALFVZh9QG1Snjhzn5WmrOOSEFk74MLHajIKc?=\n\t=?utf-8?q?25hvpUr0CKtaRy73A+5FAvuaeFC9aHrpMIrKuP2oDLqBnJfrRLGrKkixzOxy21xjk?=\n\t=?utf-8?q?OnL8vxy/zO8P4uz0faScJMIhY1Hw1Pno7fSriF3qp1LrXbVq49grMzb7yICdLg5R2?=\n\t=?utf-8?q?tZwnRDVVNfvzr9D0+t35Oy0Dp2+MqD9CHiCL9CDQC/zdfvX+2cVH3zzyrwi2ZScM5?=\n\t=?utf-8?q?Hv8fC6gkDSi/sg4Qe/z1h53zqsbIh49JP67omcSvIsPY4UN6ZwfL4x0M=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(376014)(366016)(7416014)(1800799024); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?sq/rIXZlN50VosrKjDlEzeVel7lI?=\n\t=?utf-8?q?tYW8wmeLkBaiEwUNG0Bc98Qcm9XUvYTvcN68d9HRk9lU9vbBfwKBsZNHIHp3x2QFH?=\n\t=?utf-8?q?TK66Y/4VyQXr2ipcZoejUjyjRT8fM1nekZ/TdzV4qfmHEJgt9BvvBf2ejP90ooWrY?=\n\t=?utf-8?q?BCswERFUhGF+YOtMWG6p89CMARqil8t7GDFrVVHM7kEdQ80BYfc8n89pXq4VbvMsy?=\n\t=?utf-8?q?t8ZezuQyi/0vX/wSsP82Fm86GjjtZNWiEG7HxnsfDsCBhNMJvwpgnjHAfMpgLtacl?=\n\t=?utf-8?q?de/SzczWAt01hk3SPUtfc531MlbA61bVA21LezwVasJ3tcvORaSjaHJDFNYtFc3jV?=\n\t=?utf-8?q?MR0+deoUBa2iAXgFK8wyg9fcKo2V3787joGRhf4Izy1om/KgtUBOJa/PAPIYz+bwr?=\n\t=?utf-8?q?Qib2mllXUnxzBOdS1tQHUwdUfgk5iJFsrmWIcY6Zsgk876czANc5VJAcet6L8OMaN?=\n\t=?utf-8?q?ei5gXd0uYim2PALdn4bmGVfIMEcfX3OvmpLl9npxbqrFxZYggUcG2actQzGdzbPvv?=\n\t=?utf-8?q?87ydRj4Y8EiYcsGbhcMvf22zFhQoNc3K3U6F6A9wUKnXVpx7WiPNxmUb+O/Dw3vhY?=\n\t=?utf-8?q?x2jFm616KGbjCKcip3wuMrWGCV+aR/HJNJB64H0qIO7AJjA9dGXa/Ci+t7JRDJZrB?=\n\t=?utf-8?q?d1790ShasLdrOc2OeBT4IHfTAnU1xhVKqnnnYj/NpyYV7uScukYX1SgfKVqBG3fsj?=\n\t=?utf-8?q?QmMwHVAaXPVrKOjmMFCb01HX2q6LDW6qw7Wk2N23klLBzustdFEKSOTsO2GoG5X1d?=\n\t=?utf-8?q?60BJZ2U5FKfNVxIC8G3go1MkfNTIFQMtq2M6NSZPTj9M07OWigNk+/a2fakWn/AYD?=\n\t=?utf-8?q?ywoyNFuyNHvqhUWxyMgBE+7e0L5rYkkm9dqW93U97sA+HFVOaeVTw2pSXVIa6TRDu?=\n\t=?utf-8?q?N6FfIGyzthoW0wny2FKgVvuKm1CUo7b5HmJ1Cvf1GVk+keLaTMLrftYXC4H19anbQ?=\n\t=?utf-8?q?EtsbkyD4JDxqn49z3C9R1Own1U0mW/naFktWudstJPcz0mls8dUctOeXZSaqdQtDi?=\n\t=?utf-8?q?ijKXGWQyQdKRyzNO8G/miQdnIeHpFlFPD2rOLMLqzc0B8Si28Ns15GxiBQiWlJegl?=\n\t=?utf-8?q?LMEfhvaeCqDmsCZw20n+XxwZQUx8EnfT43eLTWnmQwH2C8Ic/8JjRkBB+S+FZ2tSO?=\n\t=?utf-8?q?+6sMPSFEy9/8rtBBF1ncjD3BM3OWju1ueoe4sCP+wm525sHL4RFBp8sL0ZyejRnTA?=\n\t=?utf-8?q?IAM/oPXnqrMHiZSLm+SMoBBlyNcGfqgcLf7lMMGhwF2gk1//4w+mDMUZyZO+osbrW?=\n\t=?utf-8?q?A+ORiU3znoMGrrY+HHbpem4wTo/75i/+2qjIn9bBWimzt44cq9HxLY4QPnrGfBGk+?=\n\t=?utf-8?q?/aImhtYGj6wjWb35e/erZzEKIUDjONJFaVFTssX7yM6iEuLse/N+vEt4F1K8L+em0?=\n\t=?utf-8?q?wCq+U0RMBrbnar6iR7MHaGUYjgbXrc9yZTxwluE2iKFJXuXp+McjHG7D3C0L5UBYm?=\n\t=?utf-8?q?3/1LI7HSHLLHtHnzFArLdFuIsd5XsAsVPEjLZ+Xqx/0oqO+tRlSpjYtuvezUCg8ni?=\n\t=?utf-8?q?uu8LaYJJZZbIfyKTk67R8JuHdI/4SwIEWw=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n 0yZ1i9XngNTAAEgxXwT0nEj6tQSN6Lv5Mg2PlmGJMPtA4UwU8qtey3HBWIDs8pky/cp8CsCEp1Rzy/9Jjs6HGNRW/7W8VQfe7BQlGEMf58bkVeOcUWZqgHQsyU+fC+wOWUXXb+p0whqgoidtSOxtDEiam22jfqoxdBUclI2uNIGLnoqn2e/JU0uQ8D9bEPXAGhYufaCeDszqjZtgq+RiECUYefEWvTak+MiJJoYrpbRaVRos8Q7qtcgedL8QPhYJFIaie/gnOC9SQUZPYgn1jNN71/zu5aU6h0/Pm02hsQ8jiz0+MuWwrAK8fF65A5qHYrobMSKzM22nFLa+ig1IfpcZRhzsVvA5pUBUPvvmfCca+/vIUTqJWt+0O1t9qjsM4L74GTkvA9xDJgTkolG+am7hG39i7zatbKeNPC+rZWpKVC3Lwao2cB2kniizkHM25fiQBcLXzM4G7/WN674G4SZw5jkVQSHJnnaQWD0eItOOFGOuYLbjRGu1UvtgTk1WvTw3JNT1D0nRaE3JxhPFSfwXQKgi3lv7eRxJVd35M1BkmIswgub5OF47usrUNBZwfGNvx6EUGZWuZifSUtR5HKxOykoarAwSsJDgc47HWkI=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n a06e11e7-d3e9-495b-52fe-08dccd244e62","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"04 Sep 2024 20:58:19.8441 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n 9Y6hDI+x4jTQ4ff3T1suaBKf17GTDXUmzYso00L+5ape3Ih/nOrRNJ2pvV87YiZ/4wWG8kobxw5atAzyQ28RrCuBlJlY+6ow3lacUvGBW4A=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CY5PR10MB6264","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.60.29\n definitions=2024-09-04_18,2024-09-04_01,2024-09-02_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 adultscore=0\n phishscore=0\n suspectscore=0 bulkscore=0 mlxscore=0 spamscore=0 mlxlogscore=999\n malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1\n engine=8.12.0-2407110000 definitions=main-2409040159","X-Proofpoint-GUID":"4IwByhCeAGzQD-O_S917ijDzZz-jgqbk","X-Proofpoint-ORIG-GUID":"4IwByhCeAGzQD-O_S917ijDzZz-jgqbk","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3374367,"web_url":"http://patchwork.ozlabs.org/comment/3374367/","msgid":"","list_archive_url":null,"date":"2024-09-04T22:23:50","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Wed, Sep 04, 2024 at 04:58:14PM -0400, Steven Sistare wrote:\n> On 8/21/2024 2:34 PM, Peter Xu wrote:\n> > On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n> > > On 8/16/2024 12:17 PM, Peter Xu wrote:\n> > > > On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n> > > > > On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> > > > > > On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > > > > > > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > > > > > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > > > > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > > > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > > > > > > file, and only 1 of them can ever have them open.\n> > > > > > > > > > \n> > > > > > > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > > > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > > > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > > > > > > going to take the file lock).\n> > > > > > > > > > \n> > > > > > > > > > Maybe this is about something else?\n> > > > > > > > > \n> > > > > > > > > I don't have an example where this fails.\n> > > > > > > > > \n> > > > > > > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > > > > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > > > > > > argument, due to this code:\n> > > > > > > > > \n> > > > > > > > > blk_attach_dev()\n> > > > > > > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > > > > > > blk->disable_perm = true;\n> > > > > > > > \n> > > > > > > > Yep, this one is pretty much expected.\n> > > > > > > > \n> > > > > > > > > \n> > > > > > > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > > > > > > \n> > > > > > > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > > > > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > > > > > > the socket path before binding on the assumption that it is stale).\n> > > > > > > > > \n> > > > > > > > > One must use a different name for the socket for second qemu, and clients\n> > > > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > > > \n> > > > > > > > > > > Network ports also conflict.\n> > > > > > > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > > > > > > I forgot to promote.\n> > > > > > > > > > \n> > > > > > > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > > > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > > > > > > \n> > > > > > > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > > > > > > stable across migrations, where the host ports can be different as long as\n> > > > > > > > > > the host ports can forward guest port messages correctly?\n> > > > > > > > > \n> > > > > > > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > > > \n> > > > > > > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > > > > > > cpr-exec does not.\n> > > > > > > > \n> > > > > > > > Right, and my understanding is all these facilities are already there, so\n> > > > > > > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > > > > > > in Libvirt or similar management layers that supports migrations.\n> > > > > > > \n> > > > > > > Note Libvirt explicitly blocks localhost migration today because\n> > > > > > > solving all these clashing resource problems is a huge can of worms\n> > > > > > > and it can't be made invisible to the user of libvirt in any practical\n> > > > > > > way.\n> > > > > > \n> > > > > > Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> > > > > > supported local migration somehow on top of libvirt.\n> > > > > \n> > > > > Since kubevirt runs inside a container, \"localhost\" migration\n> > > > > is effectively migrating between 2 completely separate OS installs\n> > > > > (containers), that happen to be on the same physical host. IOW, it\n> > > > > is a cross-host migration from Libvirt & QEMU's POV, and there are\n> > > > > no clashing resources to worry about.\n> > > > \n> > > > OK, makes sense.\n> > > > \n> > > > Then do you think it's possible to support cpr-transfer in that scenario\n> > > > from Libvirt POV?\n> > > > \n> > > > > \n> > > > > > Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> > > > > > to consume it (as cpr-* is only for local host migrations so far)? Even if\n> > > > > > all the rest issues we're discussing with cpr-exec, is that the only way to\n> > > > > > go for Libvirt, then?\n> > > > > \n> > > > > cpr-exec is certainly appealing from the POV of avoiding the clashing\n> > > > > resources problem in libvirt.\n> > > > > \n> > > > > It has own issues though, because libvirt runs all QEMU processes with\n> > > > > seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n> > > > > and thus don't want to allow it to exec anything !\n> > > > > \n> > > > > I don't know which is the lesser evil from libvirt's POV.\n> > > > > \n> > > > > Personally I see security controls as an overriding requirement for\n> > > > > everything.\n> > > > \n> > > > One thing I am aware of is cpr-exec is not the only one who might start to\n> > > > use exec() in QEMU. TDX fundamentally will need to create another key VM to\n> > > > deliver the keys and the plan seems to be using exec() too. However in\n> > > > that case per my understanding the exec() is optional - the key VM can also\n> > > > be created by Libvirt.\n> > > > \n> > > > IOW, it looks like we can still stick with execve() being blocked yet so\n> > > > far except cpr-exec().\n> > > > \n> > > > Hmm, this makes the decision harder to make. We need to figure out a way\n> > > > on knowing how to consume this feature for at least open source virt\n> > > > stack.. So far it looks like it's only possible (if we take seccomp high\n> > > > priority) we use cpr-transfer but only in a container.\n> > > \n> > > libvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\n> > > and change namespace operations. I have a patch in my workspace to be submitted\n> > > later called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\n> > > libvirt to block fork and namespace but allow exec.\n> > \n> > The question is whether that would be accepted, and it also gives me the\n> > feeling that even if it's accepted, it might limit the use cases that cpr\n> > can apply to.\n> \n> This is more acceptable for libvirt running in a container (such as under kubevirt)\n> with a limited set of binaries in /bin that could be exec'd. In that case allowing\n> exec is more reasonable.\n> \n> > What I read so far from Dan is that cpr-transfer seems to be also preferred\n> > from Libvirt POV:\n> > \n> > https://lore.kernel.org/r/Zr9-IvoRkGjre4CI@redhat.com\n> > \n> > Did I read it right?\n> \n> I read that as: cpr-transfer is a viable option for libvirt. I don't hear him\n> excluding the possibility of cpr-exec.\n\nI preferred not having two solution because if they work the same problem\nout, then it potentially means one of them might be leftover at some point,\nunless they suite different needs. But I don't feel strongly, especially\nif cpr-exec is light if cpr-transfer is there.\n\n> \n> I agree that \"Dan the libvirt expert prefers cpr-transfer\" is a good reason to\n> provide cpr-transfer. Which I will do.\n> \n> So does \"Steve the OCI expert prefers cpr-exec\" carry equal weight, for also\n> providing cpr-exec?\n\nAs an open source project, Libvirt using it means the feature can be\nactively used and tested. When e.g. there's a new feature replacing CPR we\nknow when we can obsolete the old CPR, no matter -exec or -transfer.\n\nClose sourced projects can also be great itself but naturally are less\nimportant in open source communities IMHO due to not accessible to anyone\nin the community. E.g., we never know when an close sourced project\nabandoned a feature, then QEMU can carry over that feature forever without\nknowing who's using it.\n\nIt's the same as when Linux doesn't maintain kabi on out-of-tree drivers to\nme. It's just that here the open source virt stack is a huge project and\nQEMU plays its role within.\n\n> \n> We are at an impasse on this series. To make forward progress, I am willing to\n> reorder the patches, and re-submit cpr-transfer as the first mode, so we can\n> review and pull that. I will submit cpr-exec as a follow on and we can resume\n> our arguments then.\n\nYes this could be better to justify how small change cpr-exec would need on\ntop of cpr-transfer, but I'd still wait for some comments from Dan or\nothers in case they'll chime in, just to avoid sinking your time with\nrebases.\n\nThanks,","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=fnfMeJ+j;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WzcVx3G51z1yg3\n\tfor ; Thu, 5 Sep 2024 08:24:57 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1slyPy-0003la-9A; Wed, 04 Sep 2024 18:24:02 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1slyPw-0003kf-Fn\n for qemu-devel@nongnu.org; Wed, 04 Sep 2024 18:24:00 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1slyPt-0006a4-PS\n for qemu-devel@nongnu.org; Wed, 04 Sep 2024 18:24:00 -0400","from mail-oi1-f197.google.com (mail-oi1-f197.google.com\n [209.85.167.197]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-220-LR8f8rdtPV-6POXT-lod1A-1; Wed, 04 Sep 2024 18:23:54 -0400","by mail-oi1-f197.google.com with SMTP id\n 5614622812f47-3df57a518ccso71589b6e.0\n for ; Wed, 04 Sep 2024 15:23:54 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n d75a77b69052e-45801db2c2bsm2317471cf.70.2024.09.04.15.23.52\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Wed, 04 Sep 2024 15:23:52 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1725488635;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=sRRfQKr0NVxtEzt5AWKPXiDVVrEgSuHgq20POCABfLU=;\n b=fnfMeJ+jDgvIACQSLVk5D75u4EUFYnAngLRgtlnfXky1yzeSKSSGr3Aa3Ss695KPvWBPM7\n g1CzIl41+kG5gH1HBe+TyMSF/uuGNVLA5EMepIcPj8EJi3emsMaJG5+KlmRRsn77meFSJq\n 6GaTSsl7+ihVBp3Nn26ZuEqHOfqN+yQ=","X-MC-Unique":"LR8f8rdtPV-6POXT-lod1A-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1725488634; x=1726093434;\n h=in-reply-to:content-transfer-encoding:content-disposition\n :mime-version:references:message-id:subject:cc:to:from:date\n :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;\n bh=sRRfQKr0NVxtEzt5AWKPXiDVVrEgSuHgq20POCABfLU=;\n b=YMlqhK7khJpXuDZLVFpR9jFF2F2g1LyBeAZxxpzpDVeoZZ9uqyQJltYK3ebzaJpMgF\n x21Qio8UnNYEXTUZX54bDrINrtAKVoV+cc2TGaptKm/Ttf/DRi7SpPnbPHWKJufgxOBl\n ywpUCz8sefpf3wZRNq+eLjGYEjmi5yvnE3k7jn4RnMkZyaDrOtPCotK/XXExZdNz+NBg\n 82+tSjkDtfV30vE2yQcNj1NBqmHbCQeYlpLqNXC+8fwKl/FBMPvf59SQQCXk8vsxBjQA\n LWrAI+3HWW69ooJj5eQi75CubNsvLGiKohV52YflQipMUmzd1JjrfBqcuFboFj51HhBE\n u2Ow==","X-Forwarded-Encrypted":"i=1;\n AJvYcCVxoDWcqMCnQN9d7aX1ccpv5Y3zN2Gqc6XKG83dggq51rKmtNWmiC8rGbjEKKoVylP9P3axtjFx9sYc@nongnu.org","X-Gm-Message-State":"AOJu0YzJi3hNxUO9s46xOHvW7uxsMsbLwzOQWQ5CbhARuwN0GbOrg541\n rNfIR2aRiukfykIAWUThaClPYVThWuqdoj3Jn9mBg1/XkrCguEGyoRRVFh1B8njGmjfYJhPpGMe\n apkQ/vgXL7XL0BGs5doxi17BPSZt59yfKvGannywFUfM6k45Uql1X","X-Received":["by 2002:a05:6808:11c8:b0:3db:5260:a994 with SMTP id\n 5614622812f47-3df1b84aa43mr16550224b6e.42.1725488633949;\n Wed, 04 Sep 2024 15:23:53 -0700 (PDT)","by 2002:a05:6808:11c8:b0:3db:5260:a994 with SMTP id\n 5614622812f47-3df1b84aa43mr16550193b6e.42.1725488633475;\n Wed, 04 Sep 2024 15:23:53 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IEoTpmOkBX19zd64RK5Ggs/y8Y3uv6vyZvkfIe78HD6IqPqLzbf2GPRoJgfw2ya1vhKrBpn7w==","Date":"Wed, 4 Sep 2024 18:23:50 -0400","From":"Peter Xu ","To":"Steven Sistare ","Cc":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ,\n qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"\n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n \n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-21","X-Spam_score":"-2.2","X-Spam_bar":"--","X-Spam_report":"(-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.142,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3374622,"web_url":"http://patchwork.ozlabs.org/comment/3374622/","msgid":"","list_archive_url":null,"date":"2024-09-05T09:30:11","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n> On 8/16/2024 12:17 PM, Peter Xu wrote:\n> > On Fri, Aug 16, 2024 at 05:00:32PM +0100, Daniel P. Berrangé wrote:\n> > > On Fri, Aug 16, 2024 at 11:34:10AM -0400, Peter Xu wrote:\n> > > > On Fri, Aug 16, 2024 at 04:16:50PM +0100, Daniel P. Berrangé wrote:\n> > > > > On Fri, Aug 16, 2024 at 11:06:10AM -0400, Peter Xu wrote:\n> > > > > > On Thu, Aug 15, 2024 at 04:55:20PM -0400, Steven Sistare wrote:\n> > > > > > > On 8/13/2024 3:46 PM, Peter Xu wrote:\n> > > > > > > > On Tue, Aug 06, 2024 at 04:56:18PM -0400, Steven Sistare wrote:\n> > > > > > > > > > The flipside, however, is that localhost migration via 2 separate QEMU\n> > > > > > > > > > processes has issues where both QEMUs want to be opening the very same\n> > > > > > > > > > file, and only 1 of them can ever have them open.\n> > > > > > > > \n> > > > > > > > I thought we used to have similar issue with block devices, but I assume\n> > > > > > > > it's solved for years (and whoever owns it will take proper file lock,\n> > > > > > > > IIRC, and QEMU migration should properly serialize the time window on who's\n> > > > > > > > going to take the file lock).\n> > > > > > > > \n> > > > > > > > Maybe this is about something else?\n> > > > > > > \n> > > > > > > I don't have an example where this fails.\n> > > > > > > \n> > > > > > > I can cause \"Failed to get \"write\" lock\" errors if two qemu instances open\n> > > > > > > the same block device, but the error is suppressed if you add the -incoming\n> > > > > > > argument, due to this code:\n> > > > > > > \n> > > > > > > blk_attach_dev()\n> > > > > > > if (runstate_check(RUN_STATE_INMIGRATE))\n> > > > > > > blk->disable_perm = true;\n> > > > > > \n> > > > > > Yep, this one is pretty much expected.\n> > > > > > \n> > > > > > > \n> > > > > > > > > Indeed, and \"files\" includes unix domain sockets.\n> > > > > > > \n> > > > > > > More on this -- the second qemu to bind a unix domain socket for listening\n> > > > > > > wins, and the first qemu loses it (because second qemu unlinks and recreates\n> > > > > > > the socket path before binding on the assumption that it is stale).\n> > > > > > > \n> > > > > > > One must use a different name for the socket for second qemu, and clients\n> > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > \n> > > > > > > > > Network ports also conflict.\n> > > > > > > > > cpr-exec avoids such problems, and is one of the advantages of the method that\n> > > > > > > > > I forgot to promote.\n> > > > > > > > \n> > > > > > > > I was thinking that's fine, as the host ports should be the backend of the\n> > > > > > > > VM ports only anyway so they don't need to be identical on both sides?\n> > > > > > > > \n> > > > > > > > IOW, my understanding is it's the guest IP/ports/... which should still be\n> > > > > > > > stable across migrations, where the host ports can be different as long as\n> > > > > > > > the host ports can forward guest port messages correctly?\n> > > > > > > \n> > > > > > > Yes, one must use a different host port number for the second qemu, and clients\n> > > > > > > that wish to connect must be aware of the new port.\n> > > > > > > \n> > > > > > > That is my point -- cpr-transfer requires fiddling with such things.\n> > > > > > > cpr-exec does not.\n> > > > > > \n> > > > > > Right, and my understanding is all these facilities are already there, so\n> > > > > > no new code should be needed on reconnect issues if to support cpr-transfer\n> > > > > > in Libvirt or similar management layers that supports migrations.\n> > > > > \n> > > > > Note Libvirt explicitly blocks localhost migration today because\n> > > > > solving all these clashing resource problems is a huge can of worms\n> > > > > and it can't be made invisible to the user of libvirt in any practical\n> > > > > way.\n> > > > \n> > > > Ahhh, OK. I'm pretty surprised by this, as I thought at least kubevirt\n> > > > supported local migration somehow on top of libvirt.\n> > > \n> > > Since kubevirt runs inside a container, \"localhost\" migration\n> > > is effectively migrating between 2 completely separate OS installs\n> > > (containers), that happen to be on the same physical host. IOW, it\n> > > is a cross-host migration from Libvirt & QEMU's POV, and there are\n> > > no clashing resources to worry about.\n> > \n> > OK, makes sense.\n> > \n> > Then do you think it's possible to support cpr-transfer in that scenario\n> > from Libvirt POV?\n> > \n> > > \n> > > > Does it mean that cpr-transfer is a no-go in this case at least for Libvirt\n> > > > to consume it (as cpr-* is only for local host migrations so far)? Even if\n> > > > all the rest issues we're discussing with cpr-exec, is that the only way to\n> > > > go for Libvirt, then?\n> > > \n> > > cpr-exec is certainly appealing from the POV of avoiding the clashing\n> > > resources problem in libvirt.\n> > > \n> > > It has own issues though, because libvirt runs all QEMU processes with\n> > > seccomp filters that block 'execve', as we consider QEMU to be untrustworthy\n> > > and thus don't want to allow it to exec anything !\n> > > \n> > > I don't know which is the lesser evil from libvirt's POV.\n> > > \n> > > Personally I see security controls as an overriding requirement for\n> > > everything.\n> > \n> > One thing I am aware of is cpr-exec is not the only one who might start to\n> > use exec() in QEMU. TDX fundamentally will need to create another key VM to\n> > deliver the keys and the plan seems to be using exec() too. However in\n> > that case per my understanding the exec() is optional - the key VM can also\n> > be created by Libvirt.\n> > \n> > IOW, it looks like we can still stick with execve() being blocked yet so\n> > far except cpr-exec().\n> > \n> > Hmm, this makes the decision harder to make. We need to figure out a way\n> > on knowing how to consume this feature for at least open source virt\n> > stack.. So far it looks like it's only possible (if we take seccomp high\n> > priority) we use cpr-transfer but only in a container.\n> \n> libvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\n> and change namespace operations. I have a patch in my workspace to be submitted\n> later called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\n> libvirt to block fork and namespace but allow exec.\n\nIMHO this significantly undermines the protection offered. fork(), without\nexecve() is relatively benign from a security POV, mostly a slightly greater\nresource consumption issue, compared to spawning threads, which is always\nallowed. Blocking execve() is the key security benefit, as that is a way to\npick up new privileges (through setuid), or bring new binary code into\nmemory (via the new ELF images loaded), or pick up new MAC policy through\ntransition rules, etc.\n\nIOW, if you're going to allow 'exec', there's little point in blocking fork\nIMHO, and as such this doesn't sound very appealing as something to add to\nlibvirt.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=RVYUv9xA;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WzvHv1dNPz1yXY\n\tfor ; Thu, 5 Sep 2024 19:31:21 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sm8oy-0002P6-MN; Thu, 05 Sep 2024 05:30:32 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm8ow-0002OV-Qf\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:30:30 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.129.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm8ot-0001kc-V6\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:30:30 -0400","from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-647-tjk9Al5eMIqVPSQZJT7hpA-1; Thu,\n 05 Sep 2024 05:30:19 -0400","from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id CB28919560B5; Thu, 5 Sep 2024 09:30:17 +0000 (UTC)","from redhat.com (unknown [10.42.28.53])\n by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id 7F9EE19560AF; Thu, 5 Sep 2024 09:30:14 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1725528622;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:\n content-transfer-encoding:content-transfer-encoding:\n in-reply-to:in-reply-to:references:references;\n bh=LOwW9xtTW6AwRZ29DaXoueMCz9t68Jc+u3CHKi5NG4o=;\n b=RVYUv9xArPyBFvmT1VzkuZH/SgPw2Ejo1HmX3OcZD9ML9thuEAFQY83Mc9f+tjvhxxwee5\n d907oA8j4Z9yu2y4MU9IszJ8yPPTl2d77v2JAtvRrlC32fmYhUzdnzc4zy64JKxP+Kmo14\n zebokqTr8uu1+GI89wxJaIaqKYJPIjg=","X-MC-Unique":"tjk9Al5eMIqVPSQZJT7hpA-1","Date":"Thu, 5 Sep 2024 10:30:11 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Steven Sistare ","Cc":"Peter Xu , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"\n <5776f6b4-c55f-4e56-aac5-bee7e441ad15@oracle.com>\n \n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.12","Received-SPF":"pass client-ip=170.10.129.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"11","X-Spam_score":"1.1","X-Spam_bar":"+","X-Spam_report":"(1.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.142,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3374630,"web_url":"http://patchwork.ozlabs.org/comment/3374630/","msgid":"","list_archive_url":null,"date":"2024-09-05T09:43:17","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Wed, Sep 04, 2024 at 04:58:14PM -0400, Steven Sistare wrote:\n> On 8/21/2024 2:34 PM, Peter Xu wrote:\n> > On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n> > > \n> > > libvirt starts qemu with the -sandbox spawn=deny option which blocks fork, exec,\n> > > and change namespace operations. I have a patch in my workspace to be submitted\n> > > later called \"seccomp: fine-grained control of fork, exec, and namespace\" that allows\n> > > libvirt to block fork and namespace but allow exec.\n> > \n> > The question is whether that would be accepted, and it also gives me the\n> > feeling that even if it's accepted, it might limit the use cases that cpr\n> > can apply to.\n> \n> This is more acceptable for libvirt running in a container (such as under kubevirt)\n> with a limited set of binaries in /bin that could be exec'd. In that case allowing\n> exec is more reasonable.\n\nRunning inside a container does protect the host to a significant\ndegree. I'd say it is still important, however, to protect the\ncontrol plane (libvirt's daemons & kubevirt's agent) from the QEMU\nprocess being managed, and in that case it still looks pretty\ncompelling to deny exec.\n\n> > What I read so far from Dan is that cpr-transfer seems to be also preferred\n> > from Libvirt POV:\n> > \n> > https://lore.kernel.org/r/Zr9-IvoRkGjre4CI@redhat.com\n> > \n> > Did I read it right?\n> \n> I read that as: cpr-transfer is a viable option for libvirt. I don't hear him\n> excluding the possibility of cpr-exec.\n> \n> I agree that \"Dan the libvirt expert prefers cpr-transfer\" is a good reason to\n> provide cpr-transfer. Which I will do.\n\nBoth approaches have significant challenges for integration, but my general\npreference is towards a solution that doesn't require undermining our security\nprotections.\n\nWhen starting a VM we have no knowledge of whether a user may want to use\nCPR at a later date. We're not going to disable the seccomp sandbox by\ndefault, so that means cpr-exec would not be viable in a default VM\ndeployment.\n\nAdmins could choose to modify /etc/libvirt/qemu.conf to turn off seccomp,\nbut I'm very much not in favour of introducing a feature that requires\nthem todo this. It would be a first in libvirt, as everything else we\nsupport is possible to use with seccomp enabled. The seccomp opt-out is\nessentially just there as an emergency escape hatch, not as something we\nwant used in production.\n\n> We are at an impasse on this series. To make forward progress, I am willing to\n> reorder the patches, and re-submit cpr-transfer as the first mode, so we can\n> review and pull that. I will submit cpr-exec as a follow on and we can resume\n> our arguments then.\n\nConsidering the end result, are there CPR usage scenarios that are possible\nwith cpr-exec, that can't be achieved with cpr-transfer ?\n\nSupporting two ways to doing the same thing is increasing the maint burden\nfor QEMU maintainers, as well as downstream testing engineers who have to\nvalidate this functionality. So unless there's compelling need to support\nboth cpr-transfer and cpr-exec, it'd be nice to standardize on just one of\nthem.\n\ncpr-transfer does look like its probably more viable, even with its own\nchallenges wrt resources being opened twice.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=cFjAsYJF;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WzvZt2mLzz1yfv\n\tfor ; Thu, 5 Sep 2024 19:44:22 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sm91a-0008Ol-Si; Thu, 05 Sep 2024 05:43:34 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm91Y-0008Ni-Ui\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:43:32 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm91W-0004iN-Ah\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:43:32 -0400","from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-322-c5TQrZLRM3KO3YOWdgrJUg-1; Thu,\n 05 Sep 2024 05:43:25 -0400","from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id E7B7E1956069; Thu, 5 Sep 2024 09:43:23 +0000 (UTC)","from redhat.com (unknown [10.42.28.53])\n by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id C936B1956052; Thu, 5 Sep 2024 09:43:20 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1725529408;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:in-reply-to:in-reply-to: references:references;\n bh=XoruUiZtnqwjix1nRp+rt3Ip66D/yTVzkjSRh9uAUrs=;\n b=cFjAsYJFL6UpVcH24MUI+lBDoocFnABKWUwSCBdt3+RhByTzznrp0VIUWcVe9+UGePzhsC\n aMzMhhjlo5zEbd7Kau2mEJ8Z9FX+WOarma1geC+axdquxer18JKwIo4eCGc6cSpghyLTZ9\n vT/kGaXS8t1NuW1WyAquEEsYqVWjOeE=","X-MC-Unique":"c5TQrZLRM3KO3YOWdgrJUg-1","Date":"Thu, 5 Sep 2024 10:43:17 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Steven Sistare ","Cc":"Peter Xu , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"\n <46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n \n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.17","Received-SPF":"pass client-ip=170.10.133.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"11","X-Spam_score":"1.1","X-Spam_bar":"+","X-Spam_report":"(1.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.142,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3374634,"web_url":"http://patchwork.ozlabs.org/comment/3374634/","msgid":"","list_archive_url":null,"date":"2024-09-05T09:49:04","subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","submitter":{"id":2694,"url":"http://patchwork.ozlabs.org/api/people/2694/","name":"Daniel P. Berrangé","email":"berrange@redhat.com"},"content":"On Wed, Sep 04, 2024 at 06:23:50PM -0400, Peter Xu wrote:\n> On Wed, Sep 04, 2024 at 04:58:14PM -0400, Steven Sistare wrote:\n> > On 8/21/2024 2:34 PM, Peter Xu wrote:\n> > > On Fri, Aug 16, 2024 at 01:09:23PM -0400, Steven Sistare wrote:\n> > > > On 8/16/2024 12:17 PM, Peter Xu wrote:\n> > > What I read so far from Dan is that cpr-transfer seems to be also preferred\n> > > from Libvirt POV:\n> > > \n> > > https://lore.kernel.org/r/Zr9-IvoRkGjre4CI@redhat.com\n> > > \n> > > Did I read it right?\n> > \n> > I read that as: cpr-transfer is a viable option for libvirt. I don't hear him\n> > excluding the possibility of cpr-exec.\n> \n> I preferred not having two solution because if they work the same problem\n> out, then it potentially means one of them might be leftover at some point,\n> unless they suite different needs. But I don't feel strongly, especially\n> if cpr-exec is light if cpr-transfer is there.\n> \n> > \n> > I agree that \"Dan the libvirt expert prefers cpr-transfer\" is a good reason to\n> > provide cpr-transfer. Which I will do.\n> > \n> > So does \"Steve the OCI expert prefers cpr-exec\" carry equal weight, for also\n> > providing cpr-exec?\n> \n> As an open source project, Libvirt using it means the feature can be\n> actively used and tested. When e.g. there's a new feature replacing CPR we\n> know when we can obsolete the old CPR, no matter -exec or -transfer.\n> \n> Close sourced projects can also be great itself but naturally are less\n> important in open source communities IMHO due to not accessible to anyone\n> in the community. E.g., we never know when an close sourced project\n> abandoned a feature, then QEMU can carry over that feature forever without\n> knowing who's using it.\n\nIn terms of closed source projects, effectively they don't exist from a\nQEMU maintainer's POV. Our deprecation & removal policy is designed so\nthat we don't need to think about who is using stuff.\n\nWhen QEMU deprecates something, any users (whether open source or closed\nsource) have 2 releases in which to notice this, and make a request that\nwe cancel the deprecation, or change their code.\n\nLibvirt is special in the sense that we'll CC libvirt mailing list on\nchanges to the deprecated.rst file, and we'll often not propose\ndeprecations in the first place if we know libvirt is using it, since\nwe can ask libvirt quite easily & libvirt people pay attention to QEMU.\n\nWith regards,\nDaniel","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=SumyY2RQ;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WzvjG2v0cz1yXY\n\tfor ; Thu, 5 Sep 2024 19:49:54 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sm97A-0004q7-DU; Thu, 05 Sep 2024 05:49:20 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm978-0004pa-BF\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:49:19 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sm976-0006KO-RD\n for qemu-devel@nongnu.org; Thu, 05 Sep 2024 05:49:18 -0400","from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com\n (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by\n relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,\n cipher=TLS_AES_256_GCM_SHA384) id us-mta-528-iXqbsuG6PFG7yBpn8-XTwg-1; Thu,\n 05 Sep 2024 05:49:13 -0400","from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com\n (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest\n SHA256)\n (No client certificate requested)\n by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS\n id 962BA19560A2; Thu, 5 Sep 2024 09:49:11 +0000 (UTC)","from redhat.com (unknown [10.42.28.53])\n by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with\n ESMTPS\n id 87136195608A; Thu, 5 Sep 2024 09:49:08 +0000 (UTC)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1725529755;\n h=from:from:reply-to:reply-to:subject:subject:date:date:\n message-id:message-id:to:to:cc:cc:mime-version:mime-version:\n content-type:content-type:in-reply-to:in-reply-to: references:references;\n bh=OivPxGIHHhykoH10i0ADJzQwQgX6yrBwT3TD5PLf6pY=;\n b=SumyY2RQtPi3lrp++gHTqTE4hIF1hjzvhl1Xzn9l6TlF2wxSJIZ6zq/MzCGYmCLktWoZ7j\n w9/fApPxzNbuB62Jbog4T0+7vh37lfB0AsxDCTsEsfzrnpyYj4Nf1/rDtwfCyOpofRkkSh\n kF6qIj/A9cC+qs5tEn27eeR9vR9J0II=","X-MC-Unique":"iXqbsuG6PFG7yBpn8-XTwg-1","Date":"Thu, 5 Sep 2024 10:49:04 +0100","From":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","To":"Peter Xu ","Cc":"Steven Sistare , qemu-devel@nongnu.org,\n Fabiano Rosas , David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , Markus Armbruster ","Subject":"Re: [PATCH V2 00/11] Live update: cpr-exec","Message-ID":"","References":"<46b63356-9602-4fa2-9d31-186f5f85151f@oracle.com>\n \n \n \n \n \n \n ","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"Mutt/2.2.12 (2023-09-09)","X-Scanned-By":"MIMEDefang 3.0 on 10.30.177.15","Received-SPF":"pass client-ip=170.10.133.124;\n envelope-from=berrange@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"11","X-Spam_score":"1.1","X-Spam_bar":"+","X-Spam_report":"(1.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.142,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001,\n T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Reply-To":"Daniel =?utf-8?b?UC4gQmVycmFuZ8Op?= ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}}]