{"id":807498,"url":"http://patchwork.ozlabs.org/api/1.0/patches/807498/?format=json","project":{"id":14,"url":"http://patchwork.ozlabs.org/api/1.0/projects/14/?format=json","name":"QEMU Development","link_name":"qemu-devel","list_id":"qemu-devel.nongnu.org","list_email":"qemu-devel@nongnu.org","web_url":"","scm_url":"","webscm_url":""},"msgid":"<1504081950-2528-14-git-send-email-peterx@redhat.com>","date":"2017-08-30T08:32:10","name":"[RFC,v2,13/33] migration: allow fault thread to pause","commit_ref":null,"pull_url":null,"state":"new","archived":false,"hash":"2ee88235997acdc18a63eb8ab83800f09b9d8cb3","submitter":{"id":67717,"url":"http://patchwork.ozlabs.org/api/1.0/people/67717/?format=json","name":"Peter Xu","email":"peterx@redhat.com"},"delegate":null,"mbox":"http://patchwork.ozlabs.org/project/qemu-devel/patch/1504081950-2528-14-git-send-email-peterx@redhat.com/mbox/","series":[{"id":552,"url":"http://patchwork.ozlabs.org/api/1.0/series/552/?format=json","date":"2017-08-30T08:31:59","name":"Migration: postcopy failure recovery","version":2,"mbox":"http://patchwork.ozlabs.org/series/552/mbox/"}],"check":"pending","checks":"http://patchwork.ozlabs.org/api/patches/807498/checks/","tags":{},"headers":{"Return-Path":"","X-Original-To":"incoming@patchwork.ozlabs.org","Delivered-To":"patchwork-incoming@bilbo.ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=nongnu.org\n\t(client-ip=2001:4830:134:3::11; helo=lists.gnu.org;\n\tenvelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org;\n\treceiver=)","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=peterx@redhat.com"],"Received":["from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11])\n\t(using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xhzXR1Qstz9sRq\n\tfor ;\n\tWed, 30 Aug 2017 18:42:31 +1000 (AEST)","from localhost ([::1]:49054 helo=lists.gnu.org)\n\tby lists.gnu.org with esmtp (Exim 4.71) (envelope-from\n\t)\n\tid 1dmyZw-0006DB-SZ\n\tfor incoming@patchwork.ozlabs.org; Wed, 30 Aug 2017 04:42:28 -0400","from eggs.gnu.org ([2001:4830:134:3::10]:34216)\n\tby lists.gnu.org with esmtp (Exim 4.71)\n\t(envelope-from ) id 1dmyRL-0007Ni-8S\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:36 -0400","from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)\n\t(envelope-from ) id 1dmyRK-0003P7-0j\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:35 -0400","from mx1.redhat.com ([209.132.183.28]:47882)\n\tby eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)\n\t(Exim 4.71) (envelope-from ) id 1dmyRJ-0003Ou-ON\n\tfor qemu-devel@nongnu.org; Wed, 30 Aug 2017 04:33:33 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id BC0C87E435;\n\tWed, 30 Aug 2017 08:33:32 +0000 (UTC)","from pxdev.xzpeter.org.com (dhcp-14-103.nay.redhat.com\n\t[10.66.14.103])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id A880B8479B;\n\tWed, 30 Aug 2017 08:33:29 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com BC0C87E435","From":"Peter Xu ","To":"qemu-devel@nongnu.org","Date":"Wed, 30 Aug 2017 16:32:10 +0800","Message-Id":"<1504081950-2528-14-git-send-email-peterx@redhat.com>","In-Reply-To":"<1504081950-2528-1-git-send-email-peterx@redhat.com>","References":"<1504081950-2528-1-git-send-email-peterx@redhat.com>","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tWed, 30 Aug 2017 08:33:32 +0000 (UTC)","X-detected-operating-system":"by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]\n\t[fuzzy]","X-Received-From":"209.132.183.28","Subject":"[Qemu-devel] [RFC v2 13/33] migration: allow fault thread to pause","X-BeenThere":"qemu-devel@nongnu.org","X-Mailman-Version":"2.1.21","Precedence":"list","List-Id":"","List-Unsubscribe":",\n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":",\n\t","Cc":"Laurent Vivier ,\n\tAndrea Arcangeli , \n\tJuan Quintela ,\n\tAlexey Perevalov , peterx@redhat.com,\n\t\"Dr . David Alan Gilbert\" ","Errors-To":"qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org","Sender":"\"Qemu-devel\"\n\t"},"content":"Allows the fault thread to stop handling page faults temporarily. When\nnetwork failure happened (and if we expect a recovery afterwards), we\nshould not allow the fault thread to continue sending things to source,\ninstead, it should halt for a while until the connection is rebuilt.\n\nWhen the dest main thread noticed the failure, it kicks the fault thread\nto switch to pause state.\n\nReviewed-by: Dr. David Alan Gilbert \nSigned-off-by: Peter Xu \n---\n migration/migration.c | 1 +\n migration/migration.h | 1 +\n migration/postcopy-ram.c | 50 ++++++++++++++++++++++++++++++++++++++++++++----\n migration/savevm.c | 3 +++\n migration/trace-events | 2 ++\n 5 files changed, 53 insertions(+), 4 deletions(-)","diff":"diff --git a/migration/migration.c b/migration/migration.c\nindex d42209d..722f8ac 100644\n--- a/migration/migration.c\n+++ b/migration/migration.c\n@@ -147,6 +147,7 @@ MigrationIncomingState *migration_incoming_get_current(void)\n qemu_mutex_init(&mis_current.rp_mutex);\n qemu_event_init(&mis_current.main_thread_load_event, false);\n qemu_sem_init(&mis_current.postcopy_pause_sem_dst, 0);\n+ qemu_sem_init(&mis_current.postcopy_pause_sem_fault, 0);\n once = true;\n }\n return &mis_current;\ndiff --git a/migration/migration.h b/migration/migration.h\nindex 6333391..338dfe3 100644\n--- a/migration/migration.h\n+++ b/migration/migration.h\n@@ -63,6 +63,7 @@ struct MigrationIncomingState {\n \n /* notify PAUSED postcopy incoming migrations to try to continue */\n QemuSemaphore postcopy_pause_sem_dst;\n+ QemuSemaphore postcopy_pause_sem_fault;\n };\n \n MigrationIncomingState *migration_incoming_get_current(void);\ndiff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c\nindex c28e340..026a58e 100644\n--- a/migration/postcopy-ram.c\n+++ b/migration/postcopy-ram.c\n@@ -418,6 +418,17 @@ static int ram_block_enable_notify(const char *block_name, void *host_addr,\n return 0;\n }\n \n+static bool postcopy_pause_fault_thread(MigrationIncomingState *mis)\n+{\n+ trace_postcopy_pause_fault_thread();\n+\n+ qemu_sem_wait(&mis->postcopy_pause_sem_fault);\n+\n+ trace_postcopy_pause_fault_thread_continued();\n+\n+ return true;\n+}\n+\n /*\n * Handle faults detected by the USERFAULT markings\n */\n@@ -465,6 +476,22 @@ static void *postcopy_ram_fault_thread(void *opaque)\n }\n }\n \n+ if (!mis->to_src_file) {\n+ /*\n+ * Possibly someone tells us that the return path is\n+ * broken already using the event. We should hold until\n+ * the channel is rebuilt.\n+ */\n+ if (postcopy_pause_fault_thread(mis)) {\n+ last_rb = NULL;\n+ /* Continue to read the userfaultfd */\n+ } else {\n+ error_report(\"%s: paused but don't allow to continue\",\n+ __func__);\n+ break;\n+ }\n+ }\n+\n ret = read(mis->userfault_fd, &msg, sizeof(msg));\n if (ret != sizeof(msg)) {\n if (errno == EAGAIN) {\n@@ -504,18 +531,33 @@ static void *postcopy_ram_fault_thread(void *opaque)\n qemu_ram_get_idstr(rb),\n rb_offset);\n \n+retry:\n /*\n * Send the request to the source - we want to request one\n * of our host page sizes (which is >= TPS)\n */\n if (rb != last_rb) {\n last_rb = rb;\n- migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb),\n- rb_offset, qemu_ram_pagesize(rb));\n+ ret = migrate_send_rp_req_pages(mis, qemu_ram_get_idstr(rb),\n+ rb_offset, qemu_ram_pagesize(rb));\n } else {\n /* Save some space */\n- migrate_send_rp_req_pages(mis, NULL,\n- rb_offset, qemu_ram_pagesize(rb));\n+ ret = migrate_send_rp_req_pages(mis, NULL,\n+ rb_offset, qemu_ram_pagesize(rb));\n+ }\n+\n+ if (ret) {\n+ /* May be network failure, try to wait for recovery */\n+ if (ret == -EIO && postcopy_pause_fault_thread(mis)) {\n+ /* We got reconnected somehow, try to continue */\n+ last_rb = NULL;\n+ goto retry;\n+ } else {\n+ /* This is a unavoidable fault */\n+ error_report(\"%s: migrate_send_rp_req_pages() get %d\",\n+ __func__, ret);\n+ break;\n+ }\n }\n }\n trace_postcopy_ram_fault_thread_exit();\ndiff --git a/migration/savevm.c b/migration/savevm.c\nindex 3777124..a3162c1 100644\n--- a/migration/savevm.c\n+++ b/migration/savevm.c\n@@ -1994,6 +1994,9 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis)\n mis->to_src_file = NULL;\n qemu_mutex_unlock(&mis->rp_mutex);\n \n+ /* Notify the fault thread for the invalidated file handle */\n+ postcopy_fault_thread_notify(mis);\n+\n error_report(\"Detected IO failure for postcopy. \"\n \"Migration paused.\");\n \ndiff --git a/migration/trace-events b/migration/trace-events\nindex 1a83f60..42a93d9 100644\n--- a/migration/trace-events\n+++ b/migration/trace-events\n@@ -100,6 +100,8 @@ open_return_path_on_source_continue(void) \"\"\n postcopy_start(void) \"\"\n postcopy_pause_return_path(void) \"\"\n postcopy_pause_return_path_continued(void) \"\"\n+postcopy_pause_fault_thread(void) \"\"\n+postcopy_pause_fault_thread_continued(void) \"\"\n postcopy_pause_continued(void) \"\"\n postcopy_pause_incoming(void) \"\"\n postcopy_pause_incoming_continued(void) \"\"\n","prefixes":["RFC","v2","13/33"]}