{"id":807499,"url":"http://patchwork.ozlabs.org/api/1.0/patches/807499/?format=json","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/1.0/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<1504081950-2528-11-git-send-email-peterx@redhat.com>","date":"2017-08-30T08:32:07","name":"[RFC,v2,10/33] migration: allow dst vm pause on postcopy","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"5f92329f8cafa747dfc3334375e178baab867c48","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/1.0/people/67717/?format=json","name":"Peter Xu","email":"peterx@redhat.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/patch/1504081950-2528-11-git-send-email-peterx@redhat.com/mbox/","series":[{"id":552,"url":"http://patchwork.ozlabs.org/api/1.0/series/552/?format=json","date":"2017-08-30T08:31:59","name":"Migration: postcopy failure recovery","version":2,"mbox":"http://patchwork.ozlabs.org/series/552/mbox/"}],"check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/807499/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xhzYS3FSqz9sRq\n\tfor ;\n\tWed, 30 Aug 2017 18:43:24 +1000 (AEST)","from localhost ([::1]:49057 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dmyao-0006xu-4G\n\tfor incoming@patchwork.ozlabs.org; Wed, 30 Aug 2017 04:43:22 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:34109)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dmyRA-0007DZ-Kx\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:25 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dmyR9-0003KY-CO\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:24 -0400","from mx1.redhat.com ([209.132.183.28]:43150)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dmyR9-0003K8-3O\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:23 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 1AFA881DF9;\n\tWed, 30 Aug 2017 08:33:22 +0000 (UTC)","from pxdev.xzpeter.org.com (dhcp-14-103.nay.redhat.com\n\t[10.66.14.103])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 11B71871DD;\n\tWed, 30 Aug 2017 08:33:16 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 1AFA881DF9","From":"Peter Xu ","To":"qemu-devel@nongnu.org","Date":"Wed, 30 Aug 2017 16:32:07 +0800","Message-Id":"<1504081950-2528-11-git-send-email-peterx@redhat.com>","In-Reply-To":"<1504081950-2528-1-git-send-email-peterx@redhat.com>","References":"<1504081950-2528-1-git-send-email-peterx@redhat.com>","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tWed, 30 Aug 2017 08:33:22 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"[Qemu-devel] [RFC v2 10/33] migration: allow dst vm pause on\n\tpostcopy","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier ,\n\tAndrea Arcangeli , \n\tJuan Quintela ,\n\tAlexey Perevalov , peterx@redhat.com,\n\t\"Dr . David Alan Gilbert\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"},"content":"When there is IO error on the incoming channel (e.g., network down),\ninstead of bailing out immediately, we allow the dst vm to switch to the\nnew POSTCOPY_PAUSE state. Currently it is still simple - it waits the\nnew semaphore, until someone poke it for another attempt.\n\nSigned-off-by: Peter Xu \n---\n migration/migration.c | 1 +\n migration/migration.h | 3 +++\n migration/savevm.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++--\n migration/trace-events | 2 ++\n 4 files changed, 64 insertions(+), 2 deletions(-)","diff":"diff --git a/migration/migration.c b/migration/migration.c\nindex 8d26ea8..80de212 100644\n--- a/migration/migration.c\n+++ b/migration/migration.c\n@@ -146,6 +146,7 @@ MigrationIncomingState *migration_incoming_get_current(void)\n memset(&mis_current, 0, sizeof(MigrationIncomingState));\n qemu_mutex_init(&mis_current.rp_mutex);\n qemu_event_init(&mis_current.main_thread_load_event, false);\n+ qemu_sem_init(&mis_current.postcopy_pause_sem_dst, 0);\n once = true;\n }\n return &mis_current;\ndiff --git a/migration/migration.h b/migration/migration.h\nindex 0c957c9..c423682 100644\n--- a/migration/migration.h\n+++ b/migration/migration.h\n@@ -60,6 +60,9 @@ struct MigrationIncomingState {\n /* The coroutine we should enter (back) after failover */\n Coroutine *migration_incoming_co;\n QemuSemaphore colo_incoming_sem;\n+\n+ /* notify PAUSED postcopy incoming migrations to try to continue */\n+ QemuSemaphore postcopy_pause_sem_dst;\n };\n \n MigrationIncomingState *migration_incoming_get_current(void);\ndiff --git a/migration/savevm.c b/migration/savevm.c\nindex 7172f14..3777124 100644\n--- a/migration/savevm.c\n+++ b/migration/savevm.c\n@@ -1488,8 +1488,8 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis,\n */\n static void *postcopy_ram_listen_thread(void *opaque)\n {\n- QEMUFile *f = opaque;\n MigrationIncomingState *mis = migration_incoming_get_current();\n+ QEMUFile *f = mis->from_src_file;\n int load_res;\n \n migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,\n@@ -1503,6 +1503,14 @@ static void *postcopy_ram_listen_thread(void *opaque)\n */\n qemu_file_set_blocking(f, true);\n load_res = qemu_loadvm_state_main(f, mis);\n+\n+ /*\n+ * This is tricky, but, mis->from_src_file can change after it\n+ * returns, when postcopy recovery happened. In the future, we may\n+ * want a wrapper for the QEMUFile handle.\n+ */\n+ f = mis->from_src_file;\n+\n /* And non-blocking again so we don't block in any cleanup */\n qemu_file_set_blocking(f, false);\n \n@@ -1581,7 +1589,7 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)\n /* Start up the listening thread and wait for it to signal ready */\n qemu_sem_init(&mis->listen_thread_sem, 0);\n qemu_thread_create(&mis->listen_thread, \"postcopy/listen\",\n- postcopy_ram_listen_thread, mis->from_src_file,\n+ postcopy_ram_listen_thread, NULL,\n QEMU_THREAD_DETACHED);\n qemu_sem_wait(&mis->listen_thread_sem);\n qemu_sem_destroy(&mis->listen_thread_sem);\n@@ -1966,11 +1974,44 @@ void qemu_loadvm_state_cleanup(void)\n }\n }\n \n+/* Return true if we should continue the migration, or false. */\n+static bool postcopy_pause_incoming(MigrationIncomingState *mis)\n+{\n+ trace_postcopy_pause_incoming();\n+\n+ migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,\n+ MIGRATION_STATUS_POSTCOPY_PAUSED);\n+\n+ assert(mis->from_src_file);\n+ qemu_file_shutdown(mis->from_src_file);\n+ qemu_fclose(mis->from_src_file);\n+ mis->from_src_file = NULL;\n+\n+ assert(mis->to_src_file);\n+ qemu_mutex_lock(&mis->rp_mutex);\n+ qemu_file_shutdown(mis->to_src_file);\n+ qemu_fclose(mis->to_src_file);\n+ mis->to_src_file = NULL;\n+ qemu_mutex_unlock(&mis->rp_mutex);\n+\n+ error_report(\"Detected IO failure for postcopy. \"\n+ \"Migration paused.\");\n+\n+ while (mis->state == MIGRATION_STATUS_POSTCOPY_PAUSED) {\n+ qemu_sem_wait(&mis->postcopy_pause_sem_dst);\n+ }\n+\n+ trace_postcopy_pause_incoming_continued();\n+\n+ return true;\n+}\n+\n static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)\n {\n uint8_t section_type;\n int ret = 0;\n \n+retry:\n while (true) {\n section_type = qemu_get_byte(f);\n \n@@ -2016,6 +2057,21 @@ static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)\n out:\n if (ret < 0) {\n qemu_file_set_error(f, ret);\n+\n+ /*\n+ * Detect whether it is:\n+ *\n+ * 1. postcopy running\n+ * 2. network failure (-EIO)\n+ *\n+ * If so, we try to wait for a recovery.\n+ */\n+ if (mis->state == MIGRATION_STATUS_POSTCOPY_ACTIVE &&\n+ ret == -EIO && postcopy_pause_incoming(mis)) {\n+ /* Reset f to point to the newly created channel */\n+ f = mis->from_src_file;\n+ goto retry;\n+ }\n }\n return ret;\n }\ndiff --git a/migration/trace-events b/migration/trace-events\nindex 907564b..7764c6f 100644\n--- a/migration/trace-events\n+++ b/migration/trace-events\n@@ -99,6 +99,8 @@ open_return_path_on_source(void) \"\"\n open_return_path_on_source_continue(void) \"\"\n postcopy_start(void) \"\"\n postcopy_pause_continued(void) \"\"\n+postcopy_pause_incoming(void) \"\"\n+postcopy_pause_incoming_continued(void) \"\"\n postcopy_start_set_run(void) \"\"\n source_return_path_thread_bad_end(void) \"\"\n source_return_path_thread_end(void) \"\"\n","prefixes":["RFC","v2","10/33"]}