[{"id":3347852,"web_url":"http://patchwork.ozlabs.org/comment/3347852/","msgid":"<87wmlj278k.fsf@suse.de>","list_archive_url":null,"date":"2024-07-17T18:39:23","subject":"Re: [PATCH V2 02/11] migration: cpr-state","submitter":{"id":85343,"url":"http://patchwork.ozlabs.org/api/people/85343/","name":"Fabiano Rosas","email":"farosas@suse.de"},"content":"Steve Sistare writes:\n\n> CPR must save state that is needed after QEMU is restarted, when devices\n> are realized. Thus the extra state cannot be saved in the migration stream,\n> as objects must already exist before that stream can be loaded. Instead,\n> define auxilliary state structures and vmstate descriptions, not associated\n> with any registered object, and serialize the aux state to a cpr-specific\n> stream in cpr_state_save. Deserialize in cpr_state_load after QEMU\n> restarts, before devices are realized.\n>\n> Provide accessors for clients to register file descriptors for saving.\n> The mechanism for passing the fd's to the new process will be specific\n> to each migration mode, and added in subsequent patches.\n>\n> Signed-off-by: Steve Sistare \n\nReviewed-by: Fabiano Rosas ","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256\n header.s=susede2_rsa header.b=lp0SvOrl;\n\tdkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256\n header.s=susede2_ed25519 header.b=NOOSFG3j;\n\tdkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de\n header.a=rsa-sha256 header.s=susede2_rsa header.b=lp0SvOrl;\n\tdkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256\n header.s=susede2_ed25519 header.b=NOOSFG3j;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)","smtp-out2.suse.de;\n\tnone"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WPPrW3QCdz1xrQ\n\tfor ; Thu, 18 Jul 2024 04:40:27 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sU9Yp-0002vf-K3; Wed, 17 Jul 2024 14:39:31 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sU9Yn-0002uy-Co\n for qemu-devel@nongnu.org; Wed, 17 Jul 2024 14:39:29 -0400","from smtp-out2.suse.de ([195.135.223.131])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)\n (Exim 4.90_1) (envelope-from ) id 1sU9Yl-0003AU-PC\n for qemu-devel@nongnu.org; Wed, 17 Jul 2024 14:39:29 -0400","from imap1.dmz-prg2.suse.org (unknown [10.150.64.97])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest\n SHA256)\n (No client certificate requested)\n by smtp-out2.suse.de (Postfix) with ESMTPS id 32F081FB91;\n Wed, 17 Jul 2024 18:39:26 +0000 (UTC)","from imap1.dmz-prg2.suse.org (localhost [127.0.0.1])\n (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)\n key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest\n SHA256)\n (No client certificate requested)\n by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AD6531368F;\n Wed, 17 Jul 2024 18:39:25 +0000 (UTC)","from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167])\n by imap1.dmz-prg2.suse.org with ESMTPSA id bby9HN0PmGbJTgAAD6G6ig\n (envelope-from ); Wed, 17 Jul 2024 18:39:25 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de;\n s=susede2_rsa;\n t=1721241566;\n h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc:\n mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=bB6QxC3PkHXulcyjM6E2zRQqRnUHICTpemh2xIsKLtk=;\n b=lp0SvOrlbX1kIkZJeCOmmJIB/m33KWhoHYMqopAoBnFOwpLp01tYAcN+wT1L9OQQOLFgqs\n P9XKnXWURAdM//GVOiSbyy7wGR0kYKSNahsmS3UzCRW4BqA3u+ow8Ui6pFiHD2RquCPFxh\n uumPvrHW4NdvJsxo/W3pwQ5OIjQARKE=","v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de;\n s=susede2_ed25519; t=1721241566;\n h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc:\n mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=bB6QxC3PkHXulcyjM6E2zRQqRnUHICTpemh2xIsKLtk=;\n b=NOOSFG3jLOGKM+PKuxDLIRoFPypUVbvkjdkgklu+etESOix8d5j/sLspRkYd7PA6ktInSw\n t100MiVKXMW+LZCw==","v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de;\n s=susede2_rsa;\n t=1721241566;\n h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc:\n mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=bB6QxC3PkHXulcyjM6E2zRQqRnUHICTpemh2xIsKLtk=;\n b=lp0SvOrlbX1kIkZJeCOmmJIB/m33KWhoHYMqopAoBnFOwpLp01tYAcN+wT1L9OQQOLFgqs\n P9XKnXWURAdM//GVOiSbyy7wGR0kYKSNahsmS3UzCRW4BqA3u+ow8Ui6pFiHD2RquCPFxh\n uumPvrHW4NdvJsxo/W3pwQ5OIjQARKE=","v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de;\n s=susede2_ed25519; t=1721241566;\n h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc:\n mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=bB6QxC3PkHXulcyjM6E2zRQqRnUHICTpemh2xIsKLtk=;\n b=NOOSFG3jLOGKM+PKuxDLIRoFPypUVbvkjdkgklu+etESOix8d5j/sLspRkYd7PA6ktInSw\n t100MiVKXMW+LZCw=="],"From":"Fabiano Rosas ","To":"Steve Sistare , qemu-devel@nongnu.org","Cc":"Peter Xu , David Hildenbrand ,\n Marcel Apfelbaum , Eduardo Habkost\n , Philippe Mathieu-Daude , Paolo\n Bonzini , \"Daniel P. Berrange\" ,\n Markus Armbruster , Steve Sistare\n ","Subject":"Re: [PATCH V2 02/11] migration: cpr-state","In-Reply-To":"<1719776434-435013-3-git-send-email-steven.sistare@oracle.com>","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <1719776434-435013-3-git-send-email-steven.sistare@oracle.com>","Date":"Wed, 17 Jul 2024 15:39:23 -0300","Message-ID":"<87wmlj278k.fsf@suse.de>","MIME-Version":"1.0","Content-Type":"text/plain","X-Spamd-Result":"default: False [-1.98 / 50.00]; BAYES_HAM(-2.18)[96.10%];\n SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000];\n NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain];\n FREEMAIL_CC(0.00)[redhat.com,gmail.com,habkost.net,linaro.org,oracle.com];\n MIME_TRACE(0.00)[0:+];\n DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519];\n ARC_NA(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com];\n TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[];\n FREEMAIL_ENVRCPT(0.00)[gmail.com]; MISSING_XM_UA(0.00)[];\n RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[];\n FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[];\n MID_RHS_MATCH_FROM(0.00)[]; TAGGED_RCPT(0.00)[];\n RCPT_COUNT_SEVEN(0.00)[11]; RCVD_VIA_SMTP_AUTH(0.00)[];\n DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,suse.de:email]","X-Spam-Score":"-1.98","Received-SPF":"pass client-ip=195.135.223.131; envelope-from=farosas@suse.de;\n helo=smtp-out2.suse.de","X-Spam_score_int":"-43","X-Spam_score":"-4.4","X-Spam_bar":"----","X-Spam_report":"(-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3348797,"web_url":"http://patchwork.ozlabs.org/comment/3348797/","msgid":"","list_archive_url":null,"date":"2024-07-19T15:03:31","subject":"Re: [PATCH V2 02/11] migration: cpr-state","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/people/67717/","name":"Peter Xu","email":"peterx@redhat.com"},"content":"On Sun, Jun 30, 2024 at 12:40:25PM -0700, Steve Sistare wrote:\n> CPR must save state that is needed after QEMU is restarted, when devices\n> are realized. Thus the extra state cannot be saved in the migration stream,\n> as objects must already exist before that stream can be loaded. Instead,\n> define auxilliary state structures and vmstate descriptions, not associated\n> with any registered object, and serialize the aux state to a cpr-specific\n> stream in cpr_state_save. Deserialize in cpr_state_load after QEMU\n> restarts, before devices are realized.\n> \n> Provide accessors for clients to register file descriptors for saving.\n> The mechanism for passing the fd's to the new process will be specific\n> to each migration mode, and added in subsequent patches.\n> \n> Signed-off-by: Steve Sistare \n> ---\n> include/migration/cpr.h | 21 ++++++\n> migration/cpr.c | 188 ++++++++++++++++++++++++++++++++++++++++++++++++\n> migration/meson.build | 1 +\n> migration/migration.c | 6 ++\n> migration/trace-events | 5 ++\n> system/vl.c | 3 +\n> 6 files changed, 224 insertions(+)\n> create mode 100644 include/migration/cpr.h\n> create mode 100644 migration/cpr.c\n> \n> diff --git a/include/migration/cpr.h b/include/migration/cpr.h\n> new file mode 100644\n> index 0000000..8e7e705\n> --- /dev/null\n> +++ b/include/migration/cpr.h\n> @@ -0,0 +1,21 @@\n> +/*\n> + * Copyright (c) 2021, 2024 Oracle and/or its affiliates.\n> + *\n> + * This work is licensed under the terms of the GNU GPL, version 2 or later.\n> + * See the COPYING file in the top-level directory.\n> + */\n> +\n> +#ifndef MIGRATION_CPR_H\n> +#define MIGRATION_CPR_H\n> +\n> +typedef int (*cpr_walk_fd_cb)(int fd);\n> +void cpr_save_fd(const char *name, int id, int fd);\n> +void cpr_delete_fd(const char *name, int id);\n> +int cpr_find_fd(const char *name, int id);\n> +int cpr_walk_fd(cpr_walk_fd_cb cb);\n> +void cpr_resave_fd(const char *name, int id, int fd);\n> +\n> +int cpr_state_save(Error **errp);\n> +int cpr_state_load(Error **errp);\n> +\n> +#endif\n> diff --git a/migration/cpr.c b/migration/cpr.c\n> new file mode 100644\n> index 0000000..313e74e\n> --- /dev/null\n> +++ b/migration/cpr.c\n> @@ -0,0 +1,188 @@\n> +/*\n> + * Copyright (c) 2021-2024 Oracle and/or its affiliates.\n> + *\n> + * This work is licensed under the terms of the GNU GPL, version 2 or later.\n> + * See the COPYING file in the top-level directory.\n> + */\n> +\n> +#include \"qemu/osdep.h\"\n> +#include \"qapi/error.h\"\n> +#include \"migration/cpr.h\"\n> +#include \"migration/misc.h\"\n> +#include \"migration/qemu-file.h\"\n> +#include \"migration/savevm.h\"\n> +#include \"migration/vmstate.h\"\n> +#include \"sysemu/runstate.h\"\n> +#include \"trace.h\"\n> +\n> +/*************************************************************************/\n> +/* cpr state container for all information to be saved. */\n> +\n> +typedef QLIST_HEAD(CprFdList, CprFd) CprFdList;\n> +\n> +typedef struct CprState {\n> + CprFdList fds;\n> +} CprState;\n> +\n> +static CprState cpr_state;\n> +\n> +/****************************************************************************/\n> +\n> +typedef struct CprFd {\n> + char *name;\n> + unsigned int namelen;\n> + int id;\n> + int fd;\n\n[1]\n\n> + QLIST_ENTRY(CprFd) next;\n> +} CprFd;\n> +\n> +static const VMStateDescription vmstate_cpr_fd = {\n> + .name = \"cpr fd\",\n> + .version_id = 1,\n> + .minimum_version_id = 1,\n> + .fields = (VMStateField[]) {\n> + VMSTATE_UINT32(namelen, CprFd),\n> + VMSTATE_VBUFFER_ALLOC_UINT32(name, CprFd, 0, NULL, namelen),\n> + VMSTATE_INT32(id, CprFd),\n> + VMSTATE_INT32(fd, CprFd),\n> + VMSTATE_END_OF_LIST()\n> + }\n> +};\n> +\n> +void cpr_save_fd(const char *name, int id, int fd)\n> +{\n> + CprFd *elem = g_new0(CprFd, 1);\n> +\n> + trace_cpr_save_fd(name, id, fd);\n> + elem->name = g_strdup(name);\n> + elem->namelen = strlen(name) + 1;\n> + elem->id = id;\n> + elem->fd = fd;\n> + QLIST_INSERT_HEAD(&cpr_state.fds, elem, next);\n> +}\n> +\n> +static CprFd *find_fd(CprFdList *head, const char *name, int id)\n> +{\n> + CprFd *elem;\n> +\n> + QLIST_FOREACH(elem, head, next) {\n> + if (!strcmp(elem->name, name) && elem->id == id) {\n> + return elem;\n> + }\n> + }\n> + return NULL;\n> +}\n> +\n> +void cpr_delete_fd(const char *name, int id)\n> +{\n> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n> +\n> + if (elem) {\n> + QLIST_REMOVE(elem, next);\n> + g_free(elem->name);\n> + g_free(elem);\n> + }\n> +\n> + trace_cpr_delete_fd(name, id);\n> +}\n> +\n> +int cpr_find_fd(const char *name, int id)\n> +{\n> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n> + int fd = elem ? elem->fd : -1;\n> +\n> + trace_cpr_find_fd(name, id, fd);\n> + return fd;\n> +}\n> +\n> +int cpr_walk_fd(cpr_walk_fd_cb cb)\n> +{\n> + CprFd *elem;\n> +\n> + QLIST_FOREACH(elem, &cpr_state.fds, next) {\n> + if (elem->fd >= 0 && cb(elem->fd)) {\n> + return 1;\n> + }\n> + }\n> + return 0;\n> +}\n> +\n> +void cpr_resave_fd(const char *name, int id, int fd)\n> +{\n> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n> + int old_fd = elem ? elem->fd : -1;\n> +\n> + if (old_fd < 0) {\n> + cpr_save_fd(name, id, fd);\n\nI don't think I know well on when old_fd<0 would happen yet, as this series\ndoesn't look like to use this function at all. From that POV, maybe nice\nto add a comment above [1] for \"fd\" field.\n\nMeanwhile, do we need to remove the old_fd<0 element here, or is it\nintended to keep that and the new CprFD?\n\n> + } else if (old_fd != fd) {\n> + error_setg(&error_fatal,\n> + \"internal error: cpr fd '%s' id %d value %d \"\n> + \"already saved with a different value %d\",\n> + name, id, fd, old_fd);\n> + }\n> +}\n> +/*************************************************************************/\n> +#define CPR_STATE \"CprState\"\n> +\n> +static const VMStateDescription vmstate_cpr_state = {\n> + .name = CPR_STATE,\n> + .version_id = 1,\n> + .minimum_version_id = 1,\n> + .fields = (VMStateField[]) {\n> + VMSTATE_QLIST_V(fds, CprState, 1, vmstate_cpr_fd, CprFd, next),\n> + VMSTATE_END_OF_LIST()\n> + }\n> +};\n> +/*************************************************************************/\n> +\n> +int cpr_state_save(Error **errp)\n> +{\n> + int ret;\n> + QEMUFile *f;\n> +\n> + /* set f based on mode in a later patch in this series */\n> + return 0;\n> +\n> + qemu_put_be32(f, QEMU_VM_FILE_MAGIC);\n> + qemu_put_be32(f, QEMU_VM_FILE_VERSION);\n\nHaving magic/version makes sense to me, though I'd suggest we use CPR new\nmagic/versions, so that if we see an binary dump we know what it is, and we\ndon't mixup a CPR image against a migration stream image.\n\n> +\n> + ret = vmstate_save_state(f, &vmstate_cpr_state, &cpr_state, 0);\n\ns/0/NULL/\n\n> + if (ret) {\n> + error_setg(errp, \"vmstate_save_state error %d\", ret);\n> + }\n\nCan consider using vmstate_save_state_with_err().\n\n> +\n> + qemu_fclose(f);\n> + return ret;\n> +}\n> +\n> +int cpr_state_load(Error **errp)\n> +{\n> + int ret;\n> + uint32_t v;\n> + QEMUFile *f;\n> +\n> + /* set f based on mode in a later patch in this series */\n> + return 0;\n> +\n> + v = qemu_get_be32(f);\n> + if (v != QEMU_VM_FILE_MAGIC) {\n> + error_setg(errp, \"Not a migration stream (bad magic %x)\", v);\n> + qemu_fclose(f);\n> + return -EINVAL;\n> + }\n> + v = qemu_get_be32(f);\n> + if (v != QEMU_VM_FILE_VERSION) {\n> + error_setg(errp, \"Unsupported migration stream version %d\", v);\n> + qemu_fclose(f);\n> + return -ENOTSUP;\n> + }\n> +\n> + ret = vmstate_load_state(f, &vmstate_cpr_state, &cpr_state, 1);\n> + if (ret) {\n> + error_setg(errp, \"vmstate_load_state error %d\", ret);\n> + }\n\nSimiarly, can use vmstate_save_state_with_err().\n\n> +\n> + qemu_fclose(f);\n> + return ret;\n> +}\n> +\n> diff --git a/migration/meson.build b/migration/meson.build\n> index 5ce2acb4..87feb4c 100644\n> --- a/migration/meson.build\n> +++ b/migration/meson.build\n> @@ -13,6 +13,7 @@ system_ss.add(files(\n> 'block-dirty-bitmap.c',\n> 'channel.c',\n> 'channel-block.c',\n> + 'cpr.c',\n> 'dirtyrate.c',\n> 'exec.c',\n> 'fd.c',\n> diff --git a/migration/migration.c b/migration/migration.c\n> index 3dea06d..e394ad7 100644\n> --- a/migration/migration.c\n> +++ b/migration/migration.c\n> @@ -27,6 +27,7 @@\n> #include \"sysemu/cpu-throttle.h\"\n> #include \"rdma.h\"\n> #include \"ram.h\"\n> +#include \"migration/cpr.h\"\n> #include \"migration/global_state.h\"\n> #include \"migration/misc.h\"\n> #include \"migration.h\"\n> @@ -2118,6 +2119,10 @@ void qmp_migrate(const char *uri, bool has_channels,\n> }\n> }\n> \n> + if (cpr_state_save(&local_err)) {\n> + goto out;\n> + }\n> +\n> if (addr->transport == MIGRATION_ADDRESS_TYPE_SOCKET) {\n> SocketAddress *saddr = &addr->u.socket;\n> if (saddr->type == SOCKET_ADDRESS_TYPE_INET ||\n> @@ -2142,6 +2147,7 @@ void qmp_migrate(const char *uri, bool has_channels,\n> MIGRATION_STATUS_FAILED);\n> }\n> \n> +out:\n> if (local_err) {\n> if (!resume_requested) {\n> yank_unregister_instance(MIGRATION_YANK_INSTANCE);\n> diff --git a/migration/trace-events b/migration/trace-events\n> index 0b7c332..173f2c0 100644\n> --- a/migration/trace-events\n> +++ b/migration/trace-events\n> @@ -340,6 +340,11 @@ colo_receive_message(const char *msg) \"Receive '%s' message\"\n> # colo-failover.c\n> colo_failover_set_state(const char *new_state) \"new state %s\"\n> \n> +# cpr.c\n> +cpr_save_fd(const char *name, int id, int fd) \"%s, id %d, fd %d\"\n> +cpr_delete_fd(const char *name, int id) \"%s, id %d\"\n> +cpr_find_fd(const char *name, int id, int fd) \"%s, id %d returns %d\"\n> +\n> # block-dirty-bitmap.c\n> send_bitmap_header_enter(void) \"\"\n> send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) \"flags: 0x%x, start_sector: %\" PRIu64 \", nr_sectors: %\" PRIu32 \", data_size: %\" PRIu64\n> diff --git a/system/vl.c b/system/vl.c\n> index 03951be..6521ee3 100644\n> --- a/system/vl.c\n> +++ b/system/vl.c\n> @@ -77,6 +77,7 @@\n> #include \"hw/block/block.h\"\n> #include \"hw/i386/x86.h\"\n> #include \"hw/i386/pc.h\"\n> +#include \"migration/cpr.h\"\n> #include \"migration/misc.h\"\n> #include \"migration/snapshot.h\"\n> #include \"sysemu/tpm.h\"\n> @@ -3713,6 +3714,8 @@ void qemu_init(int argc, char **argv)\n> \n> qemu_create_machine(machine_opts_dict);\n> \n> + cpr_state_load(&error_fatal);\n\nMight be good to add a rich comment here explaining the decision on why\nloading here; I think most of the tricks lie here. E.g., it needs to be\nbefore XXX and it needs to be after YYY.\n\nThanks,\n\n> +\n> suspend_mux_open();\n> \n> qemu_disable_default_devices();\n> -- \n> 1.8.3.1\n>","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256\n header.s=mimecast20190719 header.b=E0VTSw92;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WQXyr3hKSz1yXp\n\tfor ; Sat, 20 Jul 2024 01:04:50 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sUp97-0008EG-V4; Fri, 19 Jul 2024 11:03:45 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sUp96-0008Bb-Sd\n for qemu-devel@nongnu.org; Fri, 19 Jul 2024 11:03:44 -0400","from us-smtp-delivery-124.mimecast.com ([170.10.133.124])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from ) id 1sUp93-0002gC-Fp\n for qemu-devel@nongnu.org; Fri, 19 Jul 2024 11:03:44 -0400","from mail-io1-f69.google.com (mail-io1-f69.google.com\n [209.85.166.69]) by relay.mimecast.com with ESMTP with STARTTLS\n (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id\n us-mta-572-b_nNQcg0MgOH3Vj2FORu5A-1; Fri, 19 Jul 2024 11:03:34 -0400","by mail-io1-f69.google.com with SMTP id\n ca18e2360f4ac-814fead45f2so55176239f.0\n for ; Fri, 19 Jul 2024 08:03:34 -0700 (PDT)","from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com.\n [99.254.121.117]) by smtp.gmail.com with ESMTPSA id\n e9e14a558f8ab-397f7a283c8sm4791495ab.67.2024.07.19.08.03.31\n (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);\n Fri, 19 Jul 2024 08:03:32 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;\n s=mimecast20190719; t=1721401417;\n h=from:from:reply-to:subject:subject:date:date:message-id:message-id:\n to:to:cc:cc:mime-version:mime-version:content-type:content-type:\n in-reply-to:in-reply-to:references:references;\n bh=mpDZ98Hp3N7v83Rqg3uNWUZ5Xb13/ZkZk0nie4mxpII=;\n b=E0VTSw92v7Va/L/ofM0mi7sChA60QczR6EBzMG/JzxodO1Ei/i0st8clhU9Ul/iWXFdDpM\n pRCg0uRFKBpEjapMNnl1uXhgVfUyEndHdZVthIMPIgPony4kAbGPMi083wjyhwTvCqQRFI\n RL7RDq/Qn3wTBfh95iFsp1JFeCFatkQ=","X-MC-Unique":"b_nNQcg0MgOH3Vj2FORu5A-1","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=1e100.net; s=20230601; t=1721401414; x=1722006214;\n h=in-reply-to:content-disposition:mime-version:references:message-id\n :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date\n :message-id:reply-to;\n bh=mpDZ98Hp3N7v83Rqg3uNWUZ5Xb13/ZkZk0nie4mxpII=;\n b=SHpBAkodRvlhe/uolU7mZSk1HRdHr24Bi2ho5tFLMwTde9Rk/GLblZimKMsbWERPM/\n nHqifMnl9oqFoK71FiCKKUZl33yyJq+6RfFVTm92dEFaKKL2vubS5SXmTV6jjhNqqpjl\n wgAwzWZ6Zi9gpUZVduJicI7Fa7DpKn9m8/YIfBkYXY+DiWRw6hnDBWULN+iB99wPhL10\n P9820FGZOVVUJVPTfwBmOd0mHBybmr5Za+EIsFF/SBFmEEZaQToRUON6ES/eQEVYKY6z\n e+bjeVV8j3np8LfgTtMhN5aLOhuUFIWCBu+AAr64NqOXUzcrPQHozZzhbel/kOnJVyrx\n R97w==","X-Gm-Message-State":"AOJu0YxZ+igzujbjnFNIQQBWCIgGLmOOdt3O9IWKYfTiNk4PSJYX7I4G\n z5QAMDuynt4Aw4TxxaR9de+Ni6N0gXYQ0ciwHRdZJCJkyVyd/szf0LPFeCCWWSAo7EYGMWo9nOt\n U0GyZ8jv5frC/+jAF/kzBuvCGKQBLNubovGwGekbukVNjUKJFhRC+","X-Received":["by 2002:a05:6e02:154b:b0:383:297a:bdfb with SMTP id\n e9e14a558f8ab-398e5d23d65mr206045ab.2.1721401413715;\n Fri, 19 Jul 2024 08:03:33 -0700 (PDT)","by 2002:a05:6e02:154b:b0:383:297a:bdfb with SMTP id\n e9e14a558f8ab-398e5d23d65mr205555ab.2.1721401413001;\n Fri, 19 Jul 2024 08:03:33 -0700 (PDT)"],"X-Google-Smtp-Source":"\n AGHT+IGmh4xaoA1F/L0XwSemwfkQhVOLYC+LK5hxlflfQPmQRaYvlEiy0kGWXC98l2QJc4gMn37uKw==","Date":"Fri, 19 Jul 2024 11:03:31 -0400","From":"Peter Xu ","To":"Steve Sistare ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini ,\n \"Daniel P. Berrange\" ,\n Markus Armbruster ","Subject":"Re: [PATCH V2 02/11] migration: cpr-state","Message-ID":"","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <1719776434-435013-3-git-send-email-steven.sistare@oracle.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","In-Reply-To":"<1719776434-435013-3-git-send-email-steven.sistare@oracle.com>","Received-SPF":"pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com;\n helo=us-smtp-delivery-124.mimecast.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001,\n SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}},{"id":3349167,"web_url":"http://patchwork.ozlabs.org/comment/3349167/","msgid":"","list_archive_url":null,"date":"2024-07-20T19:53:58","subject":"Re: [PATCH V2 02/11] migration: cpr-state","submitter":{"id":71906,"url":"http://patchwork.ozlabs.org/api/people/71906/","name":"Steve Sistare","email":"steven.sistare@oracle.com"},"content":"On 7/19/2024 11:03 AM, Peter Xu wrote:\n> On Sun, Jun 30, 2024 at 12:40:25PM -0700, Steve Sistare wrote:\n>> CPR must save state that is needed after QEMU is restarted, when devices\n>> are realized. Thus the extra state cannot be saved in the migration stream,\n>> as objects must already exist before that stream can be loaded. Instead,\n>> define auxilliary state structures and vmstate descriptions, not associated\n>> with any registered object, and serialize the aux state to a cpr-specific\n>> stream in cpr_state_save. Deserialize in cpr_state_load after QEMU\n>> restarts, before devices are realized.\n>>\n>> Provide accessors for clients to register file descriptors for saving.\n>> The mechanism for passing the fd's to the new process will be specific\n>> to each migration mode, and added in subsequent patches.\n>>\n>> Signed-off-by: Steve Sistare \n>> ---\n>> include/migration/cpr.h | 21 ++++++\n>> migration/cpr.c | 188 ++++++++++++++++++++++++++++++++++++++++++++++++\n>> migration/meson.build | 1 +\n>> migration/migration.c | 6 ++\n>> migration/trace-events | 5 ++\n>> system/vl.c | 3 +\n>> 6 files changed, 224 insertions(+)\n>> create mode 100644 include/migration/cpr.h\n>> create mode 100644 migration/cpr.c\n>>\n>> diff --git a/include/migration/cpr.h b/include/migration/cpr.h\n>> new file mode 100644\n>> index 0000000..8e7e705\n>> --- /dev/null\n>> +++ b/include/migration/cpr.h\n>> @@ -0,0 +1,21 @@\n>> +/*\n>> + * Copyright (c) 2021, 2024 Oracle and/or its affiliates.\n>> + *\n>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.\n>> + * See the COPYING file in the top-level directory.\n>> + */\n>> +\n>> +#ifndef MIGRATION_CPR_H\n>> +#define MIGRATION_CPR_H\n>> +\n>> +typedef int (*cpr_walk_fd_cb)(int fd);\n>> +void cpr_save_fd(const char *name, int id, int fd);\n>> +void cpr_delete_fd(const char *name, int id);\n>> +int cpr_find_fd(const char *name, int id);\n>> +int cpr_walk_fd(cpr_walk_fd_cb cb);\n>> +void cpr_resave_fd(const char *name, int id, int fd);\n>> +\n>> +int cpr_state_save(Error **errp);\n>> +int cpr_state_load(Error **errp);\n>> +\n>> +#endif\n>> diff --git a/migration/cpr.c b/migration/cpr.c\n>> new file mode 100644\n>> index 0000000..313e74e\n>> --- /dev/null\n>> +++ b/migration/cpr.c\n>> @@ -0,0 +1,188 @@\n>> +/*\n>> + * Copyright (c) 2021-2024 Oracle and/or its affiliates.\n>> + *\n>> + * This work is licensed under the terms of the GNU GPL, version 2 or later.\n>> + * See the COPYING file in the top-level directory.\n>> + */\n>> +\n>> +#include \"qemu/osdep.h\"\n>> +#include \"qapi/error.h\"\n>> +#include \"migration/cpr.h\"\n>> +#include \"migration/misc.h\"\n>> +#include \"migration/qemu-file.h\"\n>> +#include \"migration/savevm.h\"\n>> +#include \"migration/vmstate.h\"\n>> +#include \"sysemu/runstate.h\"\n>> +#include \"trace.h\"\n>> +\n>> +/*************************************************************************/\n>> +/* cpr state container for all information to be saved. */\n>> +\n>> +typedef QLIST_HEAD(CprFdList, CprFd) CprFdList;\n>> +\n>> +typedef struct CprState {\n>> + CprFdList fds;\n>> +} CprState;\n>> +\n>> +static CprState cpr_state;\n>> +\n>> +/****************************************************************************/\n>> +\n>> +typedef struct CprFd {\n>> + char *name;\n>> + unsigned int namelen;\n>> + int id;\n>> + int fd;\n> \n> [1]\n> \n>> + QLIST_ENTRY(CprFd) next;\n>> +} CprFd;\n>> +\n>> +static const VMStateDescription vmstate_cpr_fd = {\n>> + .name = \"cpr fd\",\n>> + .version_id = 1,\n>> + .minimum_version_id = 1,\n>> + .fields = (VMStateField[]) {\n>> + VMSTATE_UINT32(namelen, CprFd),\n>> + VMSTATE_VBUFFER_ALLOC_UINT32(name, CprFd, 0, NULL, namelen),\n>> + VMSTATE_INT32(id, CprFd),\n>> + VMSTATE_INT32(fd, CprFd),\n>> + VMSTATE_END_OF_LIST()\n>> + }\n>> +};\n>> +\n>> +void cpr_save_fd(const char *name, int id, int fd)\n>> +{\n>> + CprFd *elem = g_new0(CprFd, 1);\n>> +\n>> + trace_cpr_save_fd(name, id, fd);\n>> + elem->name = g_strdup(name);\n>> + elem->namelen = strlen(name) + 1;\n>> + elem->id = id;\n>> + elem->fd = fd;\n>> + QLIST_INSERT_HEAD(&cpr_state.fds, elem, next);\n>> +}\n>> +\n>> +static CprFd *find_fd(CprFdList *head, const char *name, int id)\n>> +{\n>> + CprFd *elem;\n>> +\n>> + QLIST_FOREACH(elem, head, next) {\n>> + if (!strcmp(elem->name, name) && elem->id == id) {\n>> + return elem;\n>> + }\n>> + }\n>> + return NULL;\n>> +}\n>> +\n>> +void cpr_delete_fd(const char *name, int id)\n>> +{\n>> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n>> +\n>> + if (elem) {\n>> + QLIST_REMOVE(elem, next);\n>> + g_free(elem->name);\n>> + g_free(elem);\n>> + }\n>> +\n>> + trace_cpr_delete_fd(name, id);\n>> +}\n>> +\n>> +int cpr_find_fd(const char *name, int id)\n>> +{\n>> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n>> + int fd = elem ? elem->fd : -1;\n>> +\n>> + trace_cpr_find_fd(name, id, fd);\n>> + return fd;\n>> +}\n>> +\n>> +int cpr_walk_fd(cpr_walk_fd_cb cb)\n>> +{\n>> + CprFd *elem;\n>> +\n>> + QLIST_FOREACH(elem, &cpr_state.fds, next) {\n>> + if (elem->fd >= 0 && cb(elem->fd)) {\n>> + return 1;\n>> + }\n>> + }\n>> + return 0;\n>> +}\n>> +\n>> +void cpr_resave_fd(const char *name, int id, int fd)\n>> +{\n>> + CprFd *elem = find_fd(&cpr_state.fds, name, id);\n>> + int old_fd = elem ? elem->fd : -1;\n>> +\n>> + if (old_fd < 0) {\n>> + cpr_save_fd(name, id, fd);\n> \n> I don't think I know well on when old_fd<0 would happen yet, as this series\n> doesn't look like to use this function at all. From that POV, maybe nice\n> to add a comment above [1] for \"fd\" field.\ncpr_resave_fd can simplify client logic. It allows the same name,fd,id triplet to\nbe (re) saved without creating a duplicate entry. The vfio series uses it. Yes,\nI should add some brief API docs in the header file.\n\n> Meanwhile, do we need to remove the old_fd<0 element here, or is it\n> intended to keep that and the new CprFD?\n\nold_fd < 0 is not an entry, it means no entry was found.\n\n>> + } else if (old_fd != fd) {\n>> + error_setg(&error_fatal,\n>> + \"internal error: cpr fd '%s' id %d value %d \"\n>> + \"already saved with a different value %d\",\n>> + name, id, fd, old_fd);\n>> + }\n>> +}\n>> +/*************************************************************************/\n>> +#define CPR_STATE \"CprState\"\n>> +\n>> +static const VMStateDescription vmstate_cpr_state = {\n>> + .name = CPR_STATE,\n>> + .version_id = 1,\n>> + .minimum_version_id = 1,\n>> + .fields = (VMStateField[]) {\n>> + VMSTATE_QLIST_V(fds, CprState, 1, vmstate_cpr_fd, CprFd, next),\n>> + VMSTATE_END_OF_LIST()\n>> + }\n>> +};\n>> +/*************************************************************************/\n>> +\n>> +int cpr_state_save(Error **errp)\n>> +{\n>> + int ret;\n>> + QEMUFile *f;\n>> +\n>> + /* set f based on mode in a later patch in this series */\n>> + return 0;\n>> +\n>> + qemu_put_be32(f, QEMU_VM_FILE_MAGIC);\n>> + qemu_put_be32(f, QEMU_VM_FILE_VERSION);\n> \n> Having magic/version makes sense to me, though I'd suggest we use CPR new\n> magic/versions, so that if we see an binary dump we know what it is, and we\n> don't mixup a CPR image against a migration stream image.\n\nWill do.\n\n>> +\n>> + ret = vmstate_save_state(f, &vmstate_cpr_state, &cpr_state, 0);\n> \n> s/0/NULL/\n\nWill do.\n\n>> + if (ret) {\n>> + error_setg(errp, \"vmstate_save_state error %d\", ret);\n>> + }\n> \n> Can consider using vmstate_save_state_with_err().\n\nCool, will do.\n\n>> +\n>> + qemu_fclose(f);\n>> + return ret;\n>> +}\n>> +\n>> +int cpr_state_load(Error **errp)\n>> +{\n>> + int ret;\n>> + uint32_t v;\n>> + QEMUFile *f;\n>> +\n>> + /* set f based on mode in a later patch in this series */\n>> + return 0;\n>> +\n>> + v = qemu_get_be32(f);\n>> + if (v != QEMU_VM_FILE_MAGIC) {\n>> + error_setg(errp, \"Not a migration stream (bad magic %x)\", v);\n>> + qemu_fclose(f);\n>> + return -EINVAL;\n>> + }\n>> + v = qemu_get_be32(f);\n>> + if (v != QEMU_VM_FILE_VERSION) {\n>> + error_setg(errp, \"Unsupported migration stream version %d\", v);\n>> + qemu_fclose(f);\n>> + return -ENOTSUP;\n>> + }\n>> +\n>> + ret = vmstate_load_state(f, &vmstate_cpr_state, &cpr_state, 1);\n>> + if (ret) {\n>> + error_setg(errp, \"vmstate_load_state error %d\", ret);\n>> + }\n> \n> Simiarly, can use vmstate_save_state_with_err().\n\nHmm, vmstate_load_state_with_err does not exist.\n\n>> +\n>> + qemu_fclose(f);\n>> + return ret;\n>> +}\n>> +\n>> diff --git a/migration/meson.build b/migration/meson.build\n>> index 5ce2acb4..87feb4c 100644\n>> --- a/migration/meson.build\n>> +++ b/migration/meson.build\n>> @@ -13,6 +13,7 @@ system_ss.add(files(\n>> 'block-dirty-bitmap.c',\n>> 'channel.c',\n>> 'channel-block.c',\n>> + 'cpr.c',\n>> 'dirtyrate.c',\n>> 'exec.c',\n>> 'fd.c',\n>> diff --git a/migration/migration.c b/migration/migration.c\n>> index 3dea06d..e394ad7 100644\n>> --- a/migration/migration.c\n>> +++ b/migration/migration.c\n>> @@ -27,6 +27,7 @@\n>> #include \"sysemu/cpu-throttle.h\"\n>> #include \"rdma.h\"\n>> #include \"ram.h\"\n>> +#include \"migration/cpr.h\"\n>> #include \"migration/global_state.h\"\n>> #include \"migration/misc.h\"\n>> #include \"migration.h\"\n>> @@ -2118,6 +2119,10 @@ void qmp_migrate(const char *uri, bool has_channels,\n>> }\n>> }\n>> \n>> + if (cpr_state_save(&local_err)) {\n>> + goto out;\n>> + }\n>> +\n>> if (addr->transport == MIGRATION_ADDRESS_TYPE_SOCKET) {\n>> SocketAddress *saddr = &addr->u.socket;\n>> if (saddr->type == SOCKET_ADDRESS_TYPE_INET ||\n>> @@ -2142,6 +2147,7 @@ void qmp_migrate(const char *uri, bool has_channels,\n>> MIGRATION_STATUS_FAILED);\n>> }\n>> \n>> +out:\n>> if (local_err) {\n>> if (!resume_requested) {\n>> yank_unregister_instance(MIGRATION_YANK_INSTANCE);\n>> diff --git a/migration/trace-events b/migration/trace-events\n>> index 0b7c332..173f2c0 100644\n>> --- a/migration/trace-events\n>> +++ b/migration/trace-events\n>> @@ -340,6 +340,11 @@ colo_receive_message(const char *msg) \"Receive '%s' message\"\n>> # colo-failover.c\n>> colo_failover_set_state(const char *new_state) \"new state %s\"\n>> \n>> +# cpr.c\n>> +cpr_save_fd(const char *name, int id, int fd) \"%s, id %d, fd %d\"\n>> +cpr_delete_fd(const char *name, int id) \"%s, id %d\"\n>> +cpr_find_fd(const char *name, int id, int fd) \"%s, id %d returns %d\"\n>> +\n>> # block-dirty-bitmap.c\n>> send_bitmap_header_enter(void) \"\"\n>> send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) \"flags: 0x%x, start_sector: %\" PRIu64 \", nr_sectors: %\" PRIu32 \", data_size: %\" PRIu64\n>> diff --git a/system/vl.c b/system/vl.c\n>> index 03951be..6521ee3 100644\n>> --- a/system/vl.c\n>> +++ b/system/vl.c\n>> @@ -77,6 +77,7 @@\n>> #include \"hw/block/block.h\"\n>> #include \"hw/i386/x86.h\"\n>> #include \"hw/i386/pc.h\"\n>> +#include \"migration/cpr.h\"\n>> #include \"migration/misc.h\"\n>> #include \"migration/snapshot.h\"\n>> #include \"sysemu/tpm.h\"\n>> @@ -3713,6 +3714,8 @@ void qemu_init(int argc, char **argv)\n>> \n>> qemu_create_machine(machine_opts_dict);\n>> \n>> + cpr_state_load(&error_fatal);\n> \n> Might be good to add a rich comment here explaining the decision on why\n> loading here; I think most of the tricks lie here. E.g., it needs to be\n> before XXX and it needs to be after YYY.\n\nWill do.\n\n- Steve","headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@legolas.ozlabs.org","Authentication-Results":["legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=cD8bcGii;\n\tdkim=pass (1024-bit key;\n unprotected) header.d=oracle.onmicrosoft.com header.i=@oracle.onmicrosoft.com\n header.a=rsa-sha256 header.s=selector2-oracle-onmicrosoft-com\n header.b=QnLJfOe4;\n\tdkim-atps=neutral","legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)"],"Received":["from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WRHMH3ym0z1ySl\n\tfor ; Sun, 21 Jul 2024 05:55:07 +1000 (AEST)","from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from )\n\tid 1sVG9r-0000PL-9T; Sat, 20 Jul 2024 15:54:19 -0400","from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sVG9p-0000Oq-Ti\n for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:54:17 -0400","from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from )\n id 1sVG9n-00085g-Ep\n for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:54:17 -0400","from pps.filterd (m0246617.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 46KJZJNY030218;\n Sat, 20 Jul 2024 19:54:10 GMT","from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta03.appoci.oracle.com [130.35.103.27])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gkhbg096-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sat, 20 Jul 2024 19:54:10 +0000 (GMT)","from pps.filterd\n (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 46KHe1Pl027443; Sat, 20 Jul 2024 19:54:08 GMT","from nam10-dm6-obe.outbound.protection.outlook.com\n (mail-dm6nam10lp2100.outbound.protection.outlook.com [104.47.58.100])\n by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id\n 40g3p5q9u7-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sat, 20 Jul 2024 19:54:08 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com (2603:10b6:208:44c::10)\n by CO1PR10MB4436.namprd10.prod.outlook.com (2603:10b6:303:95::8) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7784.16; Sat, 20 Jul\n 2024 19:54:04 +0000","from IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572]) by IA1PR10MB7447.namprd10.prod.outlook.com\n ([fe80::f2fe:d6c6:70c4:4572%6]) with mapi id 15.20.7784.016; Sat, 20 Jul 2024\n 19:54:04 +0000"],"DKIM-Signature":["v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n message-id:date:subject:to:cc:references:from:in-reply-to\n :content-type:content-transfer-encoding:mime-version; s=\n corp-2023-11-20; bh=VEUN5PaTnQJ61y87NZ2KnJp6oLy0HLXSo1ULxeM4QHM=; b=\n cD8bcGiiSGwt80GcR1xydbJX2AXaF4C9oIMphHLul3Z0kYjQs9u9minWtC8p7YtG\n 3+OeHBslyTY8KSADMmU2eHfcDXdQsjPdknVMLRxEMeEPxRIjql93fonJ0k5dJlp6\n pj/2/psvYi2KWq+slg2n+hHXag6P/vA5+udj+7IcW5QZNenpLpWvsUUCHKj3O74Z\n ORKghrEDDl5SQh59VIRR/f12Pm+Zj9bRrEJ2i+qtSVK1wLxPSSSw06isyMO936mW\n QJgU74f9EINnUmxzaseFvu5QQGTswJzDw79gcnkxxRdRqv2dCIsNzXTFKIY+7S6I\n YvPUf3x1HBHSjUpzk0D1FQ==","v=1; a=rsa-sha256; c=relaxed/relaxed;\n d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=VEUN5PaTnQJ61y87NZ2KnJp6oLy0HLXSo1ULxeM4QHM=;\n b=QnLJfOe4sXLvkeAzPFk5bq8kM2JhFYWU/u4e+dFNMW5oIDHC9tD26wtSOp/cGtsz2I1DxFs67ZEOTNgyPsxCKtoHmVtrtKf6LTfaqrcTwQC/knSjqc3ScitheKrM9yVW2Z+TLVnQ3oiMnaI5JijcVuBT21+dTb52uFBz5+7L7s8="],"ARC-Seal":"i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none;\n b=NZhkIgohHN9jVNhusglxTw2flqK2bjMri5cFm+6dfL1zgHcyVEly1o0VzEFrlrrF/9dPZr5gayVA76JNa5jtcXAqKBUjKJSuOzIUR1TO3YHCo2qschBzVx00nCK1z/Xxvffzu1Xxignf6xvNbM1Fuxpf4nQcJp8u9QdbVXLfkYXX4G7X7KKusSrTEeOBXHCGq9MaJDYeuEl16Dl+94ovEeJ2NCsfpGMuPsiQNsEvFhDCakBSYtnUrSjhyNO+fvjicJT0xaOMHGsQCaggkB6JvKQegEZd5rOMme0eXsC1YwPmwdKHuH/gBth2ww4HnCS5DexGz9kjtjqwXH4mXfs/zA==","ARC-Message-Signature":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector10001;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=VEUN5PaTnQJ61y87NZ2KnJp6oLy0HLXSo1ULxeM4QHM=;\n b=R7BuZUJFTsDYuJNnsnHLRYUCroLNlhXw+0J51mBETHhHy244tktpeLwDpAZ5G2MEMU0LY5KIE0ljfViVeA3a2u83ykaex+Js6R+G9DrYPJ6DxNgWM2d4lBiX3ya4ExyWw/1EBBJlgmjUGY9A0sGeI34H1jSd7gpgVO238UJ4lynRMcYsIzP7n95orI9GOXvzCBAsK9NJoon3Feu1hCwsWw+hmYi7kRCtA9/SBc5+DlQIi/C3tG2NYS9DRmhNpHdFOQg5zVBFY7tyq5pasunkZYV9PmwDlU3OYEuVYiN5ZOVuqNh8xcU1WUqyGEAotMN5QpoAsxkUxd3EYmwHvR3X5w==","ARC-Authentication-Results":"i=1; mx.microsoft.com 1; spf=pass\n smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com;\n dkim=pass header.d=oracle.com; arc=none","Message-ID":"","Date":"Sat, 20 Jul 2024 15:53:58 -0400","User-Agent":"Mozilla Thunderbird","Subject":"Re: [PATCH V2 02/11] migration: cpr-state","To":"Peter Xu ","Cc":"qemu-devel@nongnu.org, Fabiano Rosas ,\n David Hildenbrand ,\n Marcel Apfelbaum ,\n Eduardo Habkost ,\n Philippe Mathieu-Daude ,\n Paolo Bonzini , \"Daniel P. Berrange\"\n , Markus Armbruster ","References":"<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>\n <1719776434-435013-3-git-send-email-steven.sistare@oracle.com>\n ","Content-Language":"en-US","From":"Steven Sistare ","Organization":"Oracle Corporation","In-Reply-To":"","Content-Type":"text/plain; charset=UTF-8; format=flowed","Content-Transfer-Encoding":"7bit","X-ClientProxiedBy":"LO4P265CA0183.GBRP265.PROD.OUTLOOK.COM\n (2603:10a6:600:311::6) To IA1PR10MB7447.namprd10.prod.outlook.com\n (2603:10b6:208:44c::10)","MIME-Version":"1.0","X-MS-PublicTrafficType":"Email","X-MS-TrafficTypeDiagnostic":"IA1PR10MB7447:EE_|CO1PR10MB4436:EE_","X-MS-Office365-Filtering-Correlation-Id":"7574fbc1-ffd6-490f-fd29-08dca8f5b53c","X-MS-Exchange-SenderADCheck":"1","X-MS-Exchange-AntiSpam-Relay":"0","X-Microsoft-Antispam":"BCL:0;ARA:13230040|1800799024|366016|376014|7416014;","X-Microsoft-Antispam-Message-Info":"=?utf-8?q?+lonmkwICZR8hW2hdv1kWCtw/bMAUcb?=\n\t=?utf-8?q?Jn9XJMVN7QD6Ruaik0MC9W5kxZ+KOD0o0N9kfRSOKFsKCbh4kKyCmh+ioKuB278g2?=\n\t=?utf-8?q?JyRtll2jksyfzs7pexyt521N+4oYROC0r5r/7svJUfI7c6l5jWkGTc8uusmb9I01V?=\n\t=?utf-8?q?xRIgKe1eFQ/cP3Lw7DBz5gTAEcLKsCQgzIehIXg2/HSA3HSRC7x8tvWQqS5k6jXl6?=\n\t=?utf-8?q?iSEzUX46M17FPlkHDE5qzWWinQ2eNKuL9ysPCzr/dM06MFJw87JGUFnmEM11bCYf2?=\n\t=?utf-8?q?6vfZefKUOF0NR4Vnu7JEobXXzt8cZSVPdD+vGNA6MeFpdGcr6L0b7jmnsWRXu42xA?=\n\t=?utf-8?q?7F1ywwnsFxQopGhu473XZveyDfW9wFZXb89jW0djyrDKJtfvdmSxvYC/zPDCD4Pat?=\n\t=?utf-8?q?D47yr+940AdlpG02m43sf3ZtSutcBh4/CWyntg2CRgrT5z1WOLD7ldMGDortUR6PG?=\n\t=?utf-8?q?21BdhfVAw65ZuWQW8jsCFZyUWcTD+lia3YPAd7mVbioF8Ocz5e8gUGkZ9Q9CJHyyZ?=\n\t=?utf-8?q?sSoU3BZbpRz99MRdfkO6uQCQ4nHwiHsJw9kMFSByn23lfxIkEoIsZ9Cgc+WVbcXMp?=\n\t=?utf-8?q?nzAGKRbj2t+E3Pin+Pu9I/Jv7LG5+BMrpew+zrcnAy9leErxla+uNvQNNk6j1APKY?=\n\t=?utf-8?q?szZ9HC0dw+IHDMQ/WHuNS2oYfTD093+jOGGSOIdQF5fFTY6dylGwxGdSK6Ai1nFWb?=\n\t=?utf-8?q?EtmYX8azOfn6Fh1nqBP2piV5JMBOnxSO+raGaWeAWUOfMvkyxY8qsR7qfrzI34kr0?=\n\t=?utf-8?q?SQ/t10geTrcnHqaOhkC3kQh6Hh4eWcmB1Hf5N6CaDmRNCWbwDbxIBSStpeQlSyyvJ?=\n\t=?utf-8?q?fcY+F0y4joapRHdG/494pyyYNvD9ZgGsIg5FQqFf6AGAJ3s3heP+AtDqo5GpeUeqG?=\n\t=?utf-8?q?Qw8ttnIitTVaxexws4VSRGSsZnnqb607ZIOC1AUJa1ZfusVgkXXvOdmBswAnek3cI?=\n\t=?utf-8?q?SQFcYBiOBSIHQ4j8J+vjUlglecjwaZW1DBSDv3AlKUCuEzN8+0dTIrRfjbL/A3HUM?=\n\t=?utf-8?q?oEmPDJUiLyyn1Jo3nlmAy65IFwmbEufTR7JpBCAa8NdEz2tTxLPQAIZtpynGtGd7s?=\n\t=?utf-8?q?aPA97TfHUeZICYZeYfhl9VaTgP5b3SLja+0arUNdebU1/A3t82xorikUW2/nivgsU?=\n\t=?utf-8?q?xBKj3tjdQaaVB95isyvZfeUxcqfsGN0+eosiXWrbj+RLlidnJIQb4WXBkDgQ+t/DS?=\n\t=?utf-8?q?ZoxT0ampMaOrO6YPaezLMI/SDzBXYU+COyyMZLDzqgQKirhUVuLOYGKNkFaIFOllG?=\n\t=?utf-8?q?/KrGd29yw+eaeeC+ulwgZnuTfEKMsVBkMYg=3D=3D?=","X-Forefront-Antispam-Report":"CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;\n IPV:NLI; SFV:NSPM; H:IA1PR10MB7447.namprd10.prod.outlook.com; PTR:; CAT:NONE;\n SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1101;","X-MS-Exchange-AntiSpam-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-MessageData-0":"=?utf-8?q?EphAMkoQRtMhroj5eWDwusfL6V9O?=\n\t=?utf-8?q?i7GN+2+OSCb7F2iNcagMEuXJaY76Aoi24JxfdknRf3ag4FBvL+ezLILav8Aobww1H?=\n\t=?utf-8?q?kdlN9SJN1kRfxNP1UzP266OM5IBffgORfnMsDzQ+dAjSOkgVy0hiHFRUqyPb70c0l?=\n\t=?utf-8?q?4dVza5jiOD2bVCOkSY//WwET2z0BkkCOup4Zh/N66VZchvzOkOnfF0S+sBsQXqReK?=\n\t=?utf-8?q?HYYT1dvomaM+j86LQx1hj7A2uDX6XlnsYEGlLfPaHgRmpqNreyD8sFFs40SGlvYdM?=\n\t=?utf-8?q?4PQlHuMfwC4znyvQjwp5FOBS3hWq9Vc3vSqyDgT/A1qtLW/AIFNPPoU9mEqJbi/Zo?=\n\t=?utf-8?q?IRddQX3MkE3GkmxFxWJy5ESRE808g6DEwDfP7GchO11dd1uisLbgRK9EcdxbQrMIr?=\n\t=?utf-8?q?ztUjMhyT627J0w1B5NTArO2S7IOzG+oftnR3PIYvBTPOGCFEyg/6CbuP+Yfni6y3N?=\n\t=?utf-8?q?4hyllJBZfBtDPH8THo+w6ayBaNg/nEaL2dJAMIHARm2BYLibr7Ue0iOZndHZSRX66?=\n\t=?utf-8?q?yM1DnRXgTL1AdfBzyT7seVHtnGWRxBB9J0nYKs5Yn6W5qLSjA3Pr2VWWgD/SFMu/V?=\n\t=?utf-8?q?+ep6C2vhZuBNQZQ3ysrpUvyOBvpcaq8HLWkRIp7/+Pk3wYexIJleJh/QBXdgOCy28?=\n\t=?utf-8?q?Vc+3KcAvKgs3pNMK37C2Cw6iqON/C10Xnv2oCdMzN9wX9ud7SHSvdx4RgZwEx1tIO?=\n\t=?utf-8?q?+VIGwxAZRR09oOMk4GmX91M6a0YynWaJUgtvF503hkOAMYYEkEcewQKCU5AK583NQ?=\n\t=?utf-8?q?1HTM2zKkiQhmUM6HR/73RTy1eT44cFmN89T0QjBEjDRoXmMAOiYyUvFtG+sYELO91?=\n\t=?utf-8?q?YE/cA5OLyk6Edmh3pZL7fICI3jFkGHIJxjz5YUmlneSPTt560GCmm25VREJjgC43l?=\n\t=?utf-8?q?J2joNAm344JaXa05ifZHvGu/nA51MmWz1TcQU/lwkabXYhtgZMuyYrcUAcOFFe3k2?=\n\t=?utf-8?q?uqR2WEg6GkehrOSMiQ/aNm080dUjIUZK4TSTs6USZxT++46LUWCbrcQsBb1fab1On?=\n\t=?utf-8?q?SCKojxZv6ViwmFjdV8JJvpvrK0DFr/BKSiNIMUXHD/Cvd1LKBG74Dc6ZdVA8A8De5?=\n\t=?utf-8?q?c4jIUMRWZAhE71Jmg9Yc4lDP0BZ+8xZq/k/+cI50NXA9BRoKFu2bzh186G9s0JAjq?=\n\t=?utf-8?q?yEtjEHX8Ldp3RcZGTKhGhDceuKDgKQYSJVh0MqLO7Tikvmp+k/GlmBZDhWfRz2Ily?=\n\t=?utf-8?q?kYr/H/cs8Xv2fkU8VAQpLKpeljKLAHHGfKjJeELyzXalilEQoSeQ5e47G6MsMP+/a?=\n\t=?utf-8?q?Xcdx12Y9pAj7fJH+fK3w8X8dbRQ6+V68mvGF0IuRJZrFtX+nEIwFbspwellJcVZOq?=\n\t=?utf-8?q?qvvAQQSY6RVTl6HkifOz5Wz3TAj4aTPaXAb2bnrobuEFXk8wAspTX8j+LlywVtVk+?=\n\t=?utf-8?q?YWUJRXeU0ucJdxxHeUH9K4fyPjj9jrVkN6TsHPHPBhAvdPWjfC8iIBJOurlSqr6W4?=\n\t=?utf-8?q?uFgA8nXQiquDduff5USkgC5HDVOTPh2+V/C0mJaHLJbEhuaccB8nt9VbmJmuCY8OY?=\n\t=?utf-8?q?v4pH2lM4FUWKpSIWpq0qIVv0hkj/88wa0Q=3D=3D?=","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount":"1","X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0":"\n Q/b8a1w3rEOhRpBWcfWGVOXmu36rUDd1kwMP3QetsD96nmA5JZDwej4j+BNmybLoPiRfDdXthR+EfJEmBbSk1akElsFddUJPXURhvKGnkt/1zVPQMM/4omZhHEP/q2he7ZYA3B3ffTnMx978zeAVJR68fVgVbsPEhgu0xai5rL94ZvdP4BUS87J+OME4+ab+OxTR8SOjiEpMtjuwkMcqkTkIlqYG0fxRCm8NFMp1YIWXqC3zV+BVU4kdH8cSwC3W+OLjESX0jIW4g+0uH+dKKLOELxYQYsTvO/3oIoNCPTtM4YNdT3uqJMn8K/Lg1SbuYxEU9U/GIz1mq1VIJlN5iMZEinr9nj+tWVpZBiv8n3R9SbSlIrgiiU/ZwP8aAiQaDtyp+xOvf7vfar18EiPOmoip8A3H1zRrN+T6D1pfrlnlBVmbyhSFEhjD+vW/S3/pAt0TBVvlwavn/B0pzZBDSyG3ndtGHsF0qjnzHIxB17pSFuiDSQr75CpG+2jxWg/NDqUwMzZ5/UKE+irO42ZZ05Ttplo57VI9jm6ZVZl2GDGCE8qqxT6XgayT3IfLcXqpwH9QmQdTzP7P61eSgmNB6RdDGXBsxox6ctyhSmdzxKA=","X-OriginatorOrg":"oracle.com","X-MS-Exchange-CrossTenant-Network-Message-Id":"\n 7574fbc1-ffd6-490f-fd29-08dca8f5b53c","X-MS-Exchange-CrossTenant-AuthSource":"IA1PR10MB7447.namprd10.prod.outlook.com","X-MS-Exchange-CrossTenant-AuthAs":"Internal","X-MS-Exchange-CrossTenant-OriginalArrivalTime":"20 Jul 2024 19:54:04.2407 (UTC)","X-MS-Exchange-CrossTenant-FromEntityHeader":"Hosted","X-MS-Exchange-CrossTenant-Id":"4e2c6054-71cb-48f1-bd6c-3a9705aca71b","X-MS-Exchange-CrossTenant-MailboxType":"HOSTED","X-MS-Exchange-CrossTenant-UserPrincipalName":"\n nR0daYSRU5lmBnFWQbsSR4bQ0PHcpIFIvBXrjZb10K5uVI5j80PN7rT9U5uc8y4Gk0UItz6pTGdLKcjWZuC92HJFxymSAadK72mu85C4FB4=","X-MS-Exchange-Transport-CrossTenantHeadersStamped":"CO1PR10MB4436","X-Proofpoint-Virus-Version":"vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-07-20_17,2024-07-18_01,2024-05-17_01","X-Proofpoint-Spam-Details":"rule=notspam policy=default score=0 mlxscore=0\n bulkscore=0 phishscore=0\n suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999\n classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000\n definitions=main-2407200146","X-Proofpoint-GUID":"336ol6adWQVuIO5wiaaGtf2g8_KFL4qh","X-Proofpoint-ORIG-GUID":"336ol6adWQVuIO5wiaaGtf2g8_KFL4qh","Received-SPF":"pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com","X-Spam_score_int":"-20","X-Spam_score":"-2.1","X-Spam_bar":"--","X-Spam_report":"(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,\n DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001 autolearn=ham autolearn_force=no","X-Spam_action":"no action","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.29","Precedence":"list","List-Id":"","List-Unsubscribe":",\n ","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org"}}]