From patchwork Tue Jun 16 10:26:51 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Dr. David Alan Gilbert" X-Patchwork-Id: 484940 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 25A11140284 for ; Tue, 16 Jun 2015 20:49:29 +1000 (AEST) Received: from localhost ([::1]:39305 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4oQp-0007Qy-BQ for incoming@patchwork.ozlabs.org; Tue, 16 Jun 2015 06:49:27 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54084) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4o6e-0008L8-P3 for qemu-devel@nongnu.org; Tue, 16 Jun 2015 06:28:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Z4o6b-0001RV-92 for qemu-devel@nongnu.org; Tue, 16 Jun 2015 06:28:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34299) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Z4o6a-0001RP-Uh for qemu-devel@nongnu.org; Tue, 16 Jun 2015 06:28:33 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (Postfix) with ESMTPS id 966FDC99ED; Tue, 16 Jun 2015 10:28:32 +0000 (UTC) Received: from dgilbert-t530.redhat.com (ovpn-116-112.ams2.redhat.com [10.36.116.112]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t5GAQtaN017611; Tue, 16 Jun 2015 06:28:30 -0400 From: "Dr. David Alan Gilbert (git)" To: qemu-devel@nongnu.org Date: Tue, 16 Jun 2015 11:26:51 +0100 Message-Id: <1434450415-11339-39-git-send-email-dgilbert@redhat.com> In-Reply-To: <1434450415-11339-1-git-send-email-dgilbert@redhat.com> References: <1434450415-11339-1-git-send-email-dgilbert@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 209.132.183.28 Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, liang.z.li@intel.com, luis@cs.umu.se, amit.shah@redhat.com, pbonzini@redhat.com, david@gibson.dropbear.id.au Subject: [Qemu-devel] [PATCH v7 38/42] Start up a postcopy/listener thread ready for incoming page data X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Dr. David Alan Gilbert" The loading of a device state (during postcopy) may access guest memory that's still on the source machine and thus might need a page fill; split off a separate thread that handles the incoming page data so that the original incoming migration code can finish off the device data. Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela --- include/migration/migration.h | 4 +++ migration/migration.c | 6 ++++ migration/savevm.c | 79 ++++++++++++++++++++++++++++++++++++++++++- trace-events | 2 ++ 4 files changed, 90 insertions(+), 1 deletion(-) diff --git a/include/migration/migration.h b/include/migration/migration.h index 4f954ca..da6529d 100644 --- a/include/migration/migration.h +++ b/include/migration/migration.h @@ -75,6 +75,10 @@ struct MigrationIncomingState { QemuThread fault_thread; QemuSemaphore fault_thread_sem; + bool have_listen_thread; + QemuThread listen_thread; + QemuSemaphore listen_thread_sem; + /* For the kernel to send us notifications */ int userfault_fd; /* To tell the fault_thread to quit */ diff --git a/migration/migration.c b/migration/migration.c index 7fa982e..e908b50 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1222,6 +1222,12 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running) goto fail; } + /* + * Make sure the receiver can get incoming pages before we send the rest + * of the state + */ + qemu_savevm_send_postcopy_listen(fb); + qemu_savevm_state_complete_precopy(fb); qemu_savevm_send_ping(fb, 3); diff --git a/migration/savevm.c b/migration/savevm.c index b87238a..258c551 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1314,6 +1314,65 @@ static int loadvm_postcopy_ram_handle_discard(MigrationIncomingState *mis, return 0; } +/* + * Triggered by a postcopy_listen command; this thread takes over reading + * the input stream, leaving the main thread free to carry on loading the rest + * of the device state (from RAM). + * (TODO:This could do with being in a postcopy file - but there again it's + * just another input loop, not that postcopy specific) + */ +static void *postcopy_ram_listen_thread(void *opaque) +{ + QEMUFile *f = opaque; + MigrationIncomingState *mis = migration_incoming_get_current(); + int load_res; + + qemu_sem_post(&mis->listen_thread_sem); + trace_postcopy_ram_listen_thread_start(); + + /* + * Because we're a thread and not a coroutine we can't yield + * in qemu_file, and thus we must be blocking now. + */ + qemu_file_change_blocking(f, true); + load_res = qemu_loadvm_state_main(f, mis); + /* And non-blocking again so we don't block in any cleanup */ + qemu_file_change_blocking(f, false); + + trace_postcopy_ram_listen_thread_exit(); + if (load_res < 0) { + error_report("%s: loadvm failed: %d", __func__, load_res); + qemu_file_set_error(f, load_res); + } else { + /* + * This looks good, but it's possible that the device loading in the + * main thread hasn't finished yet, and so we might not be in 'RUN' + * state yet; wait for the end of the main thread. + */ + qemu_event_wait(&mis->main_thread_load_event); + } + postcopy_ram_incoming_cleanup(mis); + /* + * If everything has worked fine, then the main thread has waited + * for us to start, and we're the last use of the mis. + * (If something broke then qemu will have to exit anyway since it's + * got a bad migration state). + */ + migration_incoming_state_destroy(); + + if (load_res < 0) { + /* + * If something went wrong then we have a bad state so exit; + * depending how far we got it might be possible at this point + * to leave the guest running and fire MCEs for pages that never + * arrived as a desperate recovery step. + */ + exit(EXIT_FAILURE); + } + + return NULL; +} + /* After this message we must be able to immediately receive postcopy data */ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis) { @@ -1333,7 +1392,20 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis) return -1; } - /* TODO start up the postcopy listening thread */ + if (mis->have_listen_thread) { + error_report("CMD_POSTCOPY_RAM_LISTEN already has a listen thread"); + return -1; + } + + mis->have_listen_thread = true; + /* Start up the listening thread and wait for it to signal ready */ + qemu_sem_init(&mis->listen_thread_sem, 0); + qemu_thread_create(&mis->listen_thread, "postcopy/listen", + postcopy_ram_listen_thread, mis->file, + QEMU_THREAD_JOINABLE); + qemu_sem_wait(&mis->listen_thread_sem); + qemu_sem_destroy(&mis->listen_thread_sem); + return 0; } @@ -1657,6 +1729,11 @@ int qemu_loadvm_state(QEMUFile *f) qemu_event_set(&mis->main_thread_load_event); trace_qemu_loadvm_state_post_main(ret); + if (mis->have_listen_thread) { + /* Listen thread still going, can't clean up yet */ + return ret; + } + if (ret == 0) { int file_error_after_eof = qemu_file_get_error(f); diff --git a/trace-events b/trace-events index ab201f9..a6c7839 100644 --- a/trace-events +++ b/trace-events @@ -1199,6 +1199,8 @@ loadvm_postcopy_ram_handle_discard_end(void) "" loadvm_postcopy_ram_handle_discard_header(const char *ramid, uint16_t len) "%s: %ud" loadvm_process_command(uint16_t com, uint16_t len) "com=0x%x len=%d" loadvm_process_command_ping(uint32_t val) "%x" +postcopy_ram_listen_thread_exit(void) "" +postcopy_ram_listen_thread_start(void) "" qemu_savevm_send_postcopy_advise(void) "" qemu_savevm_send_postcopy_ram_discard(const char *id, uint16_t len) "%s: %ud" savevm_section_start(const char *id, unsigned int section_id) "%s, section_id %u"