Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/1954406/?format=api
{ "id": 1954406, "url": "http://patchwork.ozlabs.org/api/patches/1954406/?format=api", "web_url": "http://patchwork.ozlabs.org/project/qemu-devel/patch/1719776434-435013-12-git-send-email-steven.sistare@oracle.com/", "project": { "id": 14, "url": "http://patchwork.ozlabs.org/api/projects/14/?format=api", "name": "QEMU Development", "link_name": "qemu-devel", "list_id": "qemu-devel.nongnu.org", "list_email": "qemu-devel@nongnu.org", "web_url": "", "scm_url": "", "webscm_url": "", "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<1719776434-435013-12-git-send-email-steven.sistare@oracle.com>", "list_archive_url": null, "date": "2024-06-30T19:40:34", "name": "[V2,11/11] migration: cpr-exec mode", "commit_ref": null, "pull_url": null, "state": "new", "archived": false, "hash": "b2e6ba7b6acda92585e9efff102f074177465f14", "submitter": { "id": 71906, "url": "http://patchwork.ozlabs.org/api/people/71906/?format=api", "name": "Steve Sistare", "email": "steven.sistare@oracle.com" }, "delegate": null, "mbox": "http://patchwork.ozlabs.org/project/qemu-devel/patch/1719776434-435013-12-git-send-email-steven.sistare@oracle.com/mbox/", "series": [ { "id": 413181, "url": "http://patchwork.ozlabs.org/api/series/413181/?format=api", "web_url": "http://patchwork.ozlabs.org/project/qemu-devel/list/?series=413181", "date": "2024-06-30T19:40:26", "name": "Live update: cpr-exec", "version": 2, "mbox": "http://patchwork.ozlabs.org/series/413181/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/patches/1954406/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/1954406/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>", "X-Original-To": "incoming@patchwork.ozlabs.org", "Delivered-To": "patchwork-incoming@legolas.ozlabs.org", "Authentication-Results": [ "legolas.ozlabs.org;\n\tdkim=pass (2048-bit key;\n unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256\n header.s=corp-2023-11-20 header.b=fZmQ1ayS;\n\tdkim-atps=neutral", "legolas.ozlabs.org;\n spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org\n (client-ip=209.51.188.17; helo=lists.gnu.org;\n envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n receiver=patchwork.ozlabs.org)" ], "Received": [ "from lists.gnu.org (lists.gnu.org [209.51.188.17])\n\t(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))\n\t(No client certificate requested)\n\tby legolas.ozlabs.org (Postfix) with ESMTPS id 4WC0232jT2z1xpN\n\tfor <incoming@patchwork.ozlabs.org>; Mon, 1 Jul 2024 05:42:35 +1000 (AEST)", "from localhost ([::1] helo=lists1p.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.90_1)\n\t(envelope-from <qemu-devel-bounces@nongnu.org>)\n\tid 1sO0QO-0000G7-Qf; Sun, 30 Jun 2024 15:41:24 -0400", "from eggs.gnu.org ([2001:470:142:3::10])\n by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from <steven.sistare@oracle.com>)\n id 1sO0QB-0000As-Cs\n for qemu-devel@nongnu.org; Sun, 30 Jun 2024 15:41:12 -0400", "from mx0a-00069f02.pphosted.com ([205.220.165.32])\n by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)\n (Exim 4.90_1) (envelope-from <steven.sistare@oracle.com>)\n id 1sO0Q6-0004Pp-7W\n for qemu-devel@nongnu.org; Sun, 30 Jun 2024 15:41:11 -0400", "from pps.filterd (m0246617.ppops.net [127.0.0.1])\n by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id\n 45UIt2tN001382;\n Sun, 30 Jun 2024 19:40:52 GMT", "from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta02.appoci.oracle.com [147.154.18.20])\n by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 402att9e21-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sun, 30 Jun 2024 19:40:52 +0000 (GMT)", "from pps.filterd\n (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19)\n with ESMTP id 45UIF7Pe018995; Sun, 30 Jun 2024 19:40:50 GMT", "from pps.reinject (localhost [127.0.0.1])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id\n 4028qc16ev-1\n (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);\n Sun, 30 Jun 2024 19:40:50 +0000", "from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com\n (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1])\n by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 45UJeaSk014044;\n Sun, 30 Jun 2024 19:40:50 GMT", "from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221])\n by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with\n ESMTP id 4028qc16cc-12; Sun, 30 Jun 2024 19:40:50 +0000" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=\n from:to:cc:subject:date:message-id:in-reply-to:references; s=\n corp-2023-11-20; bh=s2nPXNUcWaNjXppRfhXzp3VvXyVUj6SXPjOtMadq9Ac=; b=\n fZmQ1aySNvNYlSxjrzZFDf0dYFq6HcZkiLb1G6ZeTsmsOWuKT9aZg15zmybe4CCX\n qMEGX9EhStLeL6blSncH6p0vzory5i21y0VDnGimNSjg2MK49DEkTv95rxH6EXBy\n QUk5uM7XmIg4zOhhWGlw1udKhrU1lD9tf/auw6PRCWkhI7I/uGq2fLhWbAzEEl8Y\n 9bcg3bz9qiueT3cAxN9HEryud0dJM1BIIma52W/EKb48/4LGOGf/KE1hHlnYg+0s\n rrmQaJGmq9k/kcQVDwT90imlT8eixWtjfwnL3rVnA/+LszTpqMtUVrneU2zaHmLD\n lWkMOOz23oY3Suw4BDPgtw==", "From": "Steve Sistare <steven.sistare@oracle.com>", "To": "qemu-devel@nongnu.org", "Cc": "Peter Xu <peterx@redhat.com>, Fabiano Rosas <farosas@suse.de>,\n David Hildenbrand <david@redhat.com>,\n Marcel Apfelbaum <marcel.apfelbaum@gmail.com>,\n Eduardo Habkost <eduardo@habkost.net>,\n Philippe Mathieu-Daude <philmd@linaro.org>,\n Paolo Bonzini <pbonzini@redhat.com>,\n \"Daniel P. Berrange\" <berrange@redhat.com>,\n Markus Armbruster <armbru@redhat.com>,\n Steve Sistare <steven.sistare@oracle.com>", "Subject": "[PATCH V2 11/11] migration: cpr-exec mode", "Date": "Sun, 30 Jun 2024 12:40:34 -0700", "Message-Id": "<1719776434-435013-12-git-send-email-steven.sistare@oracle.com>", "X-Mailer": "git-send-email 1.8.3.1", "In-Reply-To": "<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>", "References": "<1719776434-435013-1-git-send-email-steven.sistare@oracle.com>", "X-Proofpoint-Virus-Version": "vendor=baseguard\n engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16\n definitions=2024-06-30_16,2024-06-28_01,2024-05-17_01", "X-Proofpoint-Spam-Details": "rule=notspam policy=default score=0 suspectscore=0\n malwarescore=0\n adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 spamscore=0\n mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1\n engine=8.12.0-2406180000 definitions=main-2406300157", "X-Proofpoint-GUID": "5kF2f7jvzqH_aire-mOMf0vaaCopLgJ4", "X-Proofpoint-ORIG-GUID": "5kF2f7jvzqH_aire-mOMf0vaaCopLgJ4", "Received-SPF": "pass client-ip=205.220.165.32;\n envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com", "X-Spam_score_int": "-20", "X-Spam_score": "-2.1", "X-Spam_bar": "--", "X-Spam_report": "(-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001,\n DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,\n RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001,\n SPF_PASS=-0.001 autolearn=ham autolearn_force=no", "X-Spam_action": "no action", "X-BeenThere": "qemu-devel@nongnu.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "<qemu-devel.nongnu.org>", "List-Unsubscribe": "<https://lists.nongnu.org/mailman/options/qemu-devel>,\n <mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>", "List-Archive": "<https://lists.nongnu.org/archive/html/qemu-devel>", "List-Post": "<mailto:qemu-devel@nongnu.org>", "List-Help": "<mailto:qemu-devel-request@nongnu.org?subject=help>", "List-Subscribe": "<https://lists.nongnu.org/mailman/listinfo/qemu-devel>,\n <mailto:qemu-devel-request@nongnu.org?subject=subscribe>", "Errors-To": "qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org", "Sender": "qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org" }, "content": "Add the cpr-exec migration mode. Usage:\n qemu-system-$arch -machine anon-alloc=memfd ...\n migrate_set_parameter mode cpr-exec\n migrate_set_parameter cpr-exec-command \\\n <arg1> <arg2> ... -incoming <uri-1> \\\n migrate -d <uri-1>\n\nThe migrate command stops the VM, saves state to uri-1,\ndirectly exec's a new version of QEMU on the same host,\nreplacing the original process while retaining its PID, and\nloads state from uri-1. Guest RAM is preserved in place,\nalbeit with new virtual addresses.\n\nThe new QEMU process is started by exec'ing the command\nspecified by the @cpr-exec-command parameter. The first word of\nthe command is the binary, and the remaining words are its\narguments. The command may be a direct invocation of new QEMU,\nor may be a non-QEMU command that exec's the new QEMU binary.\n\nThis mode creates a second migration channel that is not visible\nto the user. At the start of migration, old QEMU saves CPR state\nto the second channel, and at the end of migration, it tells the\nmain loop to call cpr_exec. New QEMU loads CPR state early, before\nobjects are created.\n\nBecause old QEMU terminates when new QEMU starts, one cannot\nstream data between the two, so uri-1 must be a type,\nsuch as a file, that accepts all data before old QEMU exits.\nOtherwise, old QEMU may quietly block writing to the channel.\n\nMemory-backend objects must have the share=on attribute, but\nmemory-backend-epc is not supported. The VM must be started with\nthe '-machine anon-alloc=memfd' option, which allows anonymous\nmemory to be transferred in place to the new process. The memfds\nare kept open across exec by clearing the close-on-exec flag, their\nvalues are saved in CPR state, and they are mmap'd in new qemu.\n\nNote that the anon-alloc option is not related to memory-backend-memfd.\nLater patches add support for memory-backend-memfd.\n\nSigned-off-by: Steve Sistare <steven.sistare@oracle.com>\n---\n include/migration/cpr.h | 2 ++\n migration/cpr-exec.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++\n migration/cpr.c | 37 ++++++++++++++++++---\n migration/migration.c | 14 ++++++--\n migration/ram.c | 2 ++\n qapi/migration.json | 24 +++++++++++++-\n 6 files changed, 157 insertions(+), 7 deletions(-)", "diff": "diff --git a/include/migration/cpr.h b/include/migration/cpr.h\nindex 76d6ccb..c6c60f8 100644\n--- a/include/migration/cpr.h\n+++ b/include/migration/cpr.h\n@@ -30,4 +30,6 @@ QEMUFile *cpr_exec_input(Error **errp);\n void cpr_exec_persist_state(QEMUFile *f);\n bool cpr_exec_has_state(void);\n void cpr_exec_unpersist_state(void);\n+void cpr_mig_init(void);\n+void cpr_unpreserve_fds(void);\n #endif\ndiff --git a/migration/cpr-exec.c b/migration/cpr-exec.c\nindex 5c40457..fd65435 100644\n--- a/migration/cpr-exec.c\n+++ b/migration/cpr-exec.c\n@@ -11,8 +11,11 @@\n #include \"qapi/error.h\"\n #include \"io/channel-file.h\"\n #include \"io/channel-socket.h\"\n+#include \"block/block-global-state.h\"\n+#include \"qemu/main-loop.h\"\n #include \"migration/cpr.h\"\n #include \"migration/qemu-file.h\"\n+#include \"migration/migration.h\"\n #include \"migration/misc.h\"\n #include \"migration/vmstate.h\"\n #include \"sysemu/runstate.h\"\n@@ -93,3 +96,85 @@ QEMUFile *cpr_exec_input(Error **errp)\n lseek(mfd, 0, SEEK_SET);\n return qemu_file_new_fd_input(mfd, CPR_EXEC_STATE_NAME);\n }\n+\n+static int preserve_fd(int fd)\n+{\n+ qemu_clear_cloexec(fd);\n+ return 0;\n+}\n+\n+static int unpreserve_fd(int fd)\n+{\n+ qemu_set_cloexec(fd);\n+ return 0;\n+}\n+\n+static void cpr_preserve_fds(void)\n+{\n+ cpr_walk_fd(preserve_fd);\n+}\n+\n+void cpr_unpreserve_fds(void)\n+{\n+ cpr_walk_fd(unpreserve_fd);\n+}\n+\n+static void cpr_exec(char **argv)\n+{\n+ MigrationState *s = migrate_get_current();\n+ Error *err = NULL;\n+\n+ /*\n+ * Clear the close-on-exec flag for all preserved fd's. We cannot do so\n+ * earlier because they should not persist across miscellaneous fork and\n+ * exec calls that are performed during normal operation.\n+ */\n+ cpr_preserve_fds();\n+\n+ execvp(argv[0], argv);\n+\n+ cpr_unpreserve_fds();\n+\n+ error_setg_errno(&err, errno, \"execvp %s failed\", argv[0]);\n+ error_report_err(error_copy(err));\n+ migrate_set_state(&s->state, s->state, MIGRATION_STATUS_FAILED);\n+ migrate_set_error(s, err);\n+\n+ migration_call_notifiers(s, MIG_EVENT_PRECOPY_FAILED, NULL);\n+\n+ if (s->block_inactive) {\n+ Error *local_err = NULL;\n+ bdrv_activate_all(&local_err);\n+ if (local_err) {\n+ error_report_err(local_err);\n+ return;\n+ } else {\n+ s->block_inactive = false;\n+ }\n+ }\n+\n+ if (runstate_is_live(s->vm_old_state)) {\n+ vm_start();\n+ }\n+}\n+\n+static int cpr_exec_notifier(NotifierWithReturn *notifier, MigrationEvent *e,\n+ Error **errp)\n+{\n+ MigrationState *s = migrate_get_current();\n+\n+ if (e->type == MIG_EVENT_PRECOPY_DONE) {\n+ assert(s->state == MIGRATION_STATUS_COMPLETED);\n+ qemu_system_exec_request(cpr_exec, s->parameters.cpr_exec_command);\n+ } else if (e->type == MIG_EVENT_PRECOPY_FAILED) {\n+ cpr_exec_unpersist_state();\n+ }\n+ return 0;\n+}\n+\n+void cpr_mig_init(void)\n+{\n+ static NotifierWithReturn notifier;\n+ migration_add_notifier_mode(¬ifier, cpr_exec_notifier,\n+ MIG_MODE_CPR_EXEC);\n+}\ndiff --git a/migration/cpr.c b/migration/cpr.c\nindex 1c296c6..f756c15 100644\n--- a/migration/cpr.c\n+++ b/migration/cpr.c\n@@ -9,6 +9,7 @@\n #include \"qapi/error.h\"\n #include \"migration/cpr.h\"\n #include \"migration/misc.h\"\n+#include \"migration/options.h\"\n #include \"migration/qemu-file.h\"\n #include \"migration/savevm.h\"\n #include \"migration/vmstate.h\"\n@@ -160,9 +161,16 @@ int cpr_state_save(Error **errp)\n {\n int ret;\n QEMUFile *f;\n+ MigMode mode = migrate_mode();\n \n- /* set f based on mode in a later patch in this series */\n- return 0;\n+ if (mode == MIG_MODE_CPR_EXEC) {\n+ f = cpr_exec_output(errp);\n+ } else {\n+ return 0;\n+ }\n+ if (!f) {\n+ return -1;\n+ }\n \n qemu_put_be32(f, QEMU_VM_FILE_MAGIC);\n qemu_put_be32(f, QEMU_VM_FILE_VERSION);\n@@ -170,8 +178,14 @@ int cpr_state_save(Error **errp)\n ret = vmstate_save_state(f, &vmstate_cpr_state, &cpr_state, 0);\n if (ret) {\n error_setg(errp, \"vmstate_save_state error %d\", ret);\n+ goto out;\n }\n \n+ if (mode == MIG_MODE_CPR_EXEC) {\n+ cpr_exec_persist_state(f);\n+ }\n+\n+out:\n qemu_fclose(f);\n return ret;\n }\n@@ -182,8 +196,18 @@ int cpr_state_load(Error **errp)\n uint32_t v;\n QEMUFile *f;\n \n- /* set f based on mode in a later patch in this series */\n- return 0;\n+ /*\n+ * Mode will be loaded in CPR state, so cannot use it to decide which\n+ * form of state to load.\n+ */\n+ if (cpr_exec_has_state()) {\n+ f = cpr_exec_input(errp);\n+ } else {\n+ return 0;\n+ }\n+ if (!f) {\n+ return -1;\n+ }\n \n v = qemu_get_be32(f);\n if (v != QEMU_VM_FILE_MAGIC) {\n@@ -203,6 +227,11 @@ int cpr_state_load(Error **errp)\n error_setg(errp, \"vmstate_load_state error %d\", ret);\n }\n \n+ if (migrate_mode() == MIG_MODE_CPR_EXEC) {\n+ /* Set cloexec to prevent fd leaks from fork until the next cpr-exec */\n+ cpr_unpreserve_fds();\n+ }\n+\n qemu_fclose(f);\n return ret;\n }\ndiff --git a/migration/migration.c b/migration/migration.c\nindex 8a8e927..a4a020e 100644\n--- a/migration/migration.c\n+++ b/migration/migration.c\n@@ -73,9 +73,10 @@\n \n #define INMIGRATE_DEFAULT_EXIT_ON_ERROR true\n \n-static NotifierWithReturnList migration_state_notifiers[] = {\n+static NotifierWithReturnList migration_state_notifiers[MIG_MODE__MAX] = {\n NOTIFIER_ELEM_INIT(migration_state_notifiers, MIG_MODE_NORMAL),\n NOTIFIER_ELEM_INIT(migration_state_notifiers, MIG_MODE_CPR_REBOOT),\n+ NOTIFIER_ELEM_INIT(migration_state_notifiers, MIG_MODE_CPR_EXEC),\n };\n \n /* Messages sent on the return path from destination to source */\n@@ -264,6 +265,7 @@ void migration_object_init(void)\n \n ram_mig_init();\n dirty_bitmap_mig_init();\n+ cpr_mig_init();\n }\n \n typedef struct {\n@@ -1693,7 +1695,9 @@ bool migration_thread_is_self(void)\n \n bool migrate_mode_is_cpr(MigrationState *s)\n {\n- return s->parameters.mode == MIG_MODE_CPR_REBOOT;\n+ MigMode mode = s->parameters.mode;\n+ return mode == MIG_MODE_CPR_REBOOT ||\n+ mode == MIG_MODE_CPR_EXEC;\n }\n \n int migrate_init(MigrationState *s, Error **errp)\n@@ -2028,6 +2032,12 @@ static bool migrate_prepare(MigrationState *s, bool resume, Error **errp)\n return false;\n }\n \n+ if (migrate_mode() == MIG_MODE_CPR_EXEC &&\n+ !s->parameters.has_cpr_exec_command) {\n+ error_setg(errp, \"cpr-exec mode requires setting cpr-exec-command\");\n+ return false;\n+ }\n+\n if (migration_is_blocked(errp)) {\n return false;\n }\ndiff --git a/migration/ram.c b/migration/ram.c\nindex eaf3151..45b8f00 100644\n--- a/migration/ram.c\n+++ b/migration/ram.c\n@@ -216,7 +216,9 @@ static bool postcopy_preempt_active(void)\n \n bool migrate_ram_is_ignored(RAMBlock *block)\n {\n+ MigMode mode = migrate_mode();\n return !qemu_ram_is_migratable(block) ||\n+ mode == MIG_MODE_CPR_EXEC ||\n (migrate_ignore_shared() && qemu_ram_is_shared(block)\n && qemu_ram_is_named_file(block));\n }\ndiff --git a/qapi/migration.json b/qapi/migration.json\nindex 20092d2..4e626df 100644\n--- a/qapi/migration.json\n+++ b/qapi/migration.json\n@@ -604,9 +604,31 @@\n # or COLO.\n #\n # (since 8.2)\n+#\n+# @cpr-exec: The migrate command stops the VM, saves state to the\n+# migration channel, directly exec's a new version of QEMU on the\n+# same host, replacing the original process while retaining its\n+# PID, and loads state from the channel. Guest RAM is preserved\n+# in place.\n+#\n+# Old QEMU starts new QEMU by exec'ing the command specified by\n+# the @cpr-exec-command parameter. The command may be a direct\n+# invocation of new QEMU, or may be a non-QEMU command that exec's\n+# the new QEMU binary.\n+#\n+# Because old QEMU terminates when new QEMU starts, one cannot\n+# stream data between the two, so the channel must be a type,\n+# such as a file, that accepts all data before old QEMU exits.\n+# Otherwise, old QEMU may quietly block writing to the channel.\n+#\n+# Memory-backend objects must have the share=on attribute, but\n+# memory-backend-epc is not supported. The VM must be started\n+# with the '-machine anon-alloc=memfd' option.\n+#\n+# (since 9.1)\n ##\n { 'enum': 'MigMode',\n- 'data': [ 'normal', 'cpr-reboot' ] }\n+ 'data': [ 'normal', 'cpr-reboot', 'cpr-exec' ] }\n \n ##\n # @ZeroPageDetection:\n", "prefixes": [ "V2", "11/11" ] }