From patchwork Wed Apr 25 11:27:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904142 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WHy11kPCz9s1s for ; Wed, 25 Apr 2018 21:28:25 +1000 (AEST) Received: from localhost ([::1]:35840 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIb1-0002r0-9c for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:28:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34004) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaC-0002gF-3T for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaB-0004lt-4O for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:42382 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaB-0004lB-0f for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:31 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id CBF1481A88AD for ; Wed, 25 Apr 2018 11:27:27 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id C35B7215CDC8; Wed, 25 Apr 2018 11:27:26 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:03 +0200 Message-Id: <20180425112723.1111-2-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:27 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:27 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 01/21] migration: Set error state in case of error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela --- migration/ram.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 0e90efa092..2ae560ea80 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -415,10 +415,20 @@ struct { int count; } *multifd_send_state; -static void terminate_multifd_send_threads(Error *errp) +static void terminate_multifd_send_threads(Error *err) { int i; + if (err) { + MigrationState *s = migrate_get_current(); + migrate_set_error(s, err); + if (s->state == MIGRATION_STATUS_SETUP || + s->state == MIGRATION_STATUS_ACTIVE) { + migrate_set_state(&s->state, s->state, + MIGRATION_STATUS_FAILED); + } + } + for (i = 0; i < multifd_send_state->count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -515,10 +525,20 @@ struct { int count; } *multifd_recv_state; -static void terminate_multifd_recv_threads(Error *errp) +static void terminate_multifd_recv_threads(Error *err) { int i; + if (err) { + MigrationState *s = migrate_get_current(); + migrate_set_error(s, err); + if (s->state == MIGRATION_STATUS_SETUP || + s->state == MIGRATION_STATUS_ACTIVE) { + migrate_set_state(&s->state, s->state, + MIGRATION_STATUS_FAILED); + } + } + for (i = 0; i < multifd_recv_state->count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; From patchwork Wed Apr 25 11:27:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904150 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ4s1qDdz9s0t for ; Wed, 25 Apr 2018 21:34:19 +1000 (AEST) Received: from localhost ([::1]:35872 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIgi-0007jP-U1 for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:34:17 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34001) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaC-0002gB-2T for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaB-0004lo-4L for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:42384 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaB-0004lA-0s for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:31 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 200D0814F0D2 for ; Wed, 25 Apr 2018 11:27:29 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 192B92166BC6; Wed, 25 Apr 2018 11:27:27 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:04 +0200 Message-Id: <20180425112723.1111-3-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:29 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:29 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 02/21] migration: Introduce multifd_recv_new_channel() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 3 ++- migration/ram.c | 6 ++++++ migration/ram.h | 2 ++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index 7eca65d1f0..604722cec9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -466,8 +466,9 @@ void migration_ioc_process_incoming(QIOChannel *ioc) if (!mis->from_src_file) { QEMUFile *f = qemu_fopen_channel_input(ioc); migration_fd_process_incoming(f); + return; } - /* We still only have a single channel. Nothing to do here yet */ + multifd_recv_new_channel(ioc); } /** diff --git a/migration/ram.c b/migration/ram.c index 2ae560ea80..c3c330b0e0 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -36,6 +36,7 @@ #include "xbzrle.h" #include "ram.h" #include "migration.h" +#include "socket.h" #include "migration/register.h" #include "migration/misc.h" #include "qemu-file.h" @@ -619,6 +620,11 @@ int multifd_load_setup(void) return 0; } +void multifd_recv_new_channel(QIOChannel *ioc) +{ + /* nothing to do yet */ +} + /** * save_page_header: write page header to wire * diff --git a/migration/ram.h b/migration/ram.h index 5030be110a..06dbddc2a2 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -32,6 +32,7 @@ #include "qemu-common.h" #include "qapi/qapi-types-migration.h" #include "exec/cpu-common.h" +#include "io/channel.h" extern MigrationStats ram_counters; extern XBZRLECacheStats xbzrle_counters; @@ -44,6 +45,7 @@ int multifd_save_setup(void); int multifd_save_cleanup(Error **errp); int multifd_load_setup(void); int multifd_load_cleanup(Error **errp); +void multifd_recv_new_channel(QIOChannel *ioc); uint64_t ram_pagesize_summary(void); int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len); From patchwork Wed Apr 25 11:27:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904146 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ1T0kztz9s0t for ; Wed, 25 Apr 2018 21:31:24 +1000 (AEST) Received: from localhost ([::1]:35862 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIdu-0005Wm-IZ for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:31:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34026) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaC-0002gi-LQ for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaB-0004m0-5E for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:46184 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaB-0004l7-1J for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:31 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6B755406C740 for ; Wed, 25 Apr 2018 11:27:30 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6239B215CDCF; Wed, 25 Apr 2018 11:27:29 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:05 +0200 Message-Id: <20180425112723.1111-4-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:30 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:30 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 03/21] migration: terminate_* can be called for other threads X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Once there, make count field to always be accessed with atomic operations. To make blocking operations, we need to know that the thread is running, so create a bool to indicate that. Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- Once here, s/terminate_multifd_*-threads/multifd_*_terminate_threads/ This is consistente with every other function --- migration/ram.c | 44 ++++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index c3c330b0e0..c0ad6d8e9f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -406,6 +406,7 @@ struct MultiFDSendParams { QemuThread thread; QemuSemaphore sem; QemuMutex mutex; + bool running; bool quit; }; typedef struct MultiFDSendParams MultiFDSendParams; @@ -416,7 +417,7 @@ struct { int count; } *multifd_send_state; -static void terminate_multifd_send_threads(Error *err) +static void multifd_send_terminate_threads(Error *err) { int i; @@ -430,7 +431,7 @@ static void terminate_multifd_send_threads(Error *err) } } - for (i = 0; i < multifd_send_state->count; i++) { + for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; qemu_mutex_lock(&p->mutex); @@ -448,11 +449,13 @@ int multifd_save_cleanup(Error **errp) if (!migrate_use_multifd()) { return 0; } - terminate_multifd_send_threads(NULL); - for (i = 0; i < multifd_send_state->count; i++) { + multifd_send_terminate_threads(NULL); + for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; - qemu_thread_join(&p->thread); + if (p->running) { + qemu_thread_join(&p->thread); + } qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); g_free(p->name); @@ -479,6 +482,10 @@ static void *multifd_send_thread(void *opaque) qemu_sem_wait(&p->sem); } + qemu_mutex_lock(&p->mutex); + p->running = false; + qemu_mutex_unlock(&p->mutex); + return NULL; } @@ -493,7 +500,7 @@ int multifd_save_setup(void) thread_count = migrate_multifd_channels(); multifd_send_state = g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); - multifd_send_state->count = 0; + atomic_set(&multifd_send_state->count, 0); for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -502,10 +509,11 @@ int multifd_save_setup(void) p->quit = false; p->id = i; p->name = g_strdup_printf("multifdsend_%d", i); + p->running = true; qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, QEMU_THREAD_JOINABLE); - multifd_send_state->count++; + atomic_inc(&multifd_send_state->count); } return 0; } @@ -516,6 +524,7 @@ struct MultiFDRecvParams { QemuThread thread; QemuSemaphore sem; QemuMutex mutex; + bool running; bool quit; }; typedef struct MultiFDRecvParams MultiFDRecvParams; @@ -526,7 +535,7 @@ struct { int count; } *multifd_recv_state; -static void terminate_multifd_recv_threads(Error *err) +static void multifd_recv_terminate_threads(Error *err) { int i; @@ -540,7 +549,7 @@ static void terminate_multifd_recv_threads(Error *err) } } - for (i = 0; i < multifd_recv_state->count; i++) { + for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_lock(&p->mutex); @@ -558,11 +567,13 @@ int multifd_load_cleanup(Error **errp) if (!migrate_use_multifd()) { return 0; } - terminate_multifd_recv_threads(NULL); - for (i = 0; i < multifd_recv_state->count; i++) { + multifd_recv_terminate_threads(NULL); + for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; - qemu_thread_join(&p->thread); + if (p->running) { + qemu_thread_join(&p->thread); + } qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); g_free(p->name); @@ -590,6 +601,10 @@ static void *multifd_recv_thread(void *opaque) qemu_sem_wait(&p->sem); } + qemu_mutex_lock(&p->mutex); + p->running = false; + qemu_mutex_unlock(&p->mutex); + return NULL; } @@ -604,7 +619,7 @@ int multifd_load_setup(void) thread_count = migrate_multifd_channels(); multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); - multifd_recv_state->count = 0; + atomic_set(&multifd_recv_state->count, 0); for (i = 0; i < thread_count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; @@ -613,9 +628,10 @@ int multifd_load_setup(void) p->quit = false; p->id = i; p->name = g_strdup_printf("multifdrecv_%d", i); + p->running = true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, QEMU_THREAD_JOINABLE); - multifd_recv_state->count++; + atomic_inc(&multifd_recv_state->count); } return 0; } From patchwork Wed Apr 25 11:27:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904154 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ8S2TN7z9rvt for ; Wed, 25 Apr 2018 21:37:28 +1000 (AEST) Received: from localhost ([::1]:35892 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIjm-0001qp-6g for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:37:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34035) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaD-0002h4-1Y for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaC-0004nw-86 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:33 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45560 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaC-0004n9-4V for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:32 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B51C0402290A for ; Wed, 25 Apr 2018 11:27:31 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id ACF8A215CDCF; Wed, 25 Apr 2018 11:27:30 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:06 +0200 Message-Id: <20180425112723.1111-5-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:31 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:31 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 04/21] migration: Be sure all recv channels are created X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We need them before we start migration. Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 6 +++++- migration/ram.c | 11 +++++++++++ migration/ram.h | 1 + 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index 604722cec9..98f85e982c 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -479,7 +479,11 @@ void migration_ioc_process_incoming(QIOChannel *ioc) */ bool migration_has_all_channels(void) { - return true; + bool all_channels; + + all_channels = multifd_recv_all_channels_created(); + + return all_channels; } /* diff --git a/migration/ram.c b/migration/ram.c index c0ad6d8e9f..3a01472835 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -636,6 +636,17 @@ int multifd_load_setup(void) return 0; } +bool multifd_recv_all_channels_created(void) +{ + int thread_count = migrate_multifd_channels(); + + if (!migrate_use_multifd()) { + return true; + } + + return thread_count == atomic_read(&multifd_recv_state->count); +} + void multifd_recv_new_channel(QIOChannel *ioc) { /* nothing to do yet */ diff --git a/migration/ram.h b/migration/ram.h index 06dbddc2a2..3f4b7daee8 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -45,6 +45,7 @@ int multifd_save_setup(void); int multifd_save_cleanup(Error **errp); int multifd_load_setup(void); int multifd_load_cleanup(Error **errp); +bool multifd_recv_all_channels_created(void); void multifd_recv_new_channel(QIOChannel *ioc); uint64_t ram_pagesize_summary(void); From patchwork Wed Apr 25 11:27:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904143 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ0V4vC8z9rvt for ; Wed, 25 Apr 2018 21:30:34 +1000 (AEST) Received: from localhost ([::1]:35847 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBId5-0004lh-K9 for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:30:31 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34076) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaE-0002iO-Cc for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaD-0004q3-Gs for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:34 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45562 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaD-0004pN-Cn for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:33 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0B3F94022414 for ; Wed, 25 Apr 2018 11:27:33 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 040AA215CDCF; Wed, 25 Apr 2018 11:27:31 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:07 +0200 Message-Id: <20180425112723.1111-6-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:33 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:33 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 05/21] migration: Export functions to create send channels X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- migration/socket.c | 28 +++++++++++++++++++++++++++- migration/socket.h | 7 +++++++ 2 files changed, 34 insertions(+), 1 deletion(-) diff --git a/migration/socket.c b/migration/socket.c index edf33c70cf..893a04f4cc 100644 --- a/migration/socket.c +++ b/migration/socket.c @@ -29,6 +29,28 @@ #include "trace.h" +struct SocketOutgoingArgs { + SocketAddress *saddr; +} outgoing_args; + +void socket_send_channel_create(QIOTaskFunc f, void *data) +{ + QIOChannelSocket *sioc = qio_channel_socket_new(); + qio_channel_socket_connect_async(sioc, outgoing_args.saddr, + f, data, NULL, NULL); +} + +int socket_send_channel_destroy(QIOChannel *send) +{ + /* Remove channel */ + object_unref(OBJECT(send)); + if (outgoing_args.saddr) { + qapi_free_SocketAddress(outgoing_args.saddr); + outgoing_args.saddr = NULL; + } + return 0; +} + static SocketAddress *tcp_build_address(const char *host_port, Error **errp) { SocketAddress *saddr; @@ -96,6 +118,11 @@ static void socket_start_outgoing_migration(MigrationState *s, struct SocketConnectData *data = g_new0(struct SocketConnectData, 1); data->s = s; + + /* in case previous migration leaked it */ + qapi_free_SocketAddress(outgoing_args.saddr); + outgoing_args.saddr = saddr; + if (saddr->type == SOCKET_ADDRESS_TYPE_INET) { data->hostname = g_strdup(saddr->u.inet.host); } @@ -107,7 +134,6 @@ static void socket_start_outgoing_migration(MigrationState *s, data, socket_connect_data_free, NULL); - qapi_free_SocketAddress(saddr); } void tcp_start_outgoing_migration(MigrationState *s, diff --git a/migration/socket.h b/migration/socket.h index 6b91e9db38..528c3b0202 100644 --- a/migration/socket.h +++ b/migration/socket.h @@ -16,6 +16,13 @@ #ifndef QEMU_MIGRATION_SOCKET_H #define QEMU_MIGRATION_SOCKET_H + +#include "io/channel.h" +#include "io/task.h" + +void socket_send_channel_create(QIOTaskFunc f, void *data); +int socket_send_channel_destroy(QIOChannel *send); + void tcp_start_incoming_migration(const char *host_port, Error **errp); void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, From patchwork Wed Apr 25 11:27:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904151 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ583nHrz9rvt for ; Wed, 25 Apr 2018 21:34:36 +1000 (AEST) Received: from localhost ([::1]:35878 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIh0-0007xp-HR for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:34:34 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34121) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaF-0002jp-SE for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaE-0004sx-Rn for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:35 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33566 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaE-0004sD-NC for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:34 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 55E40EC016 for ; Wed, 25 Apr 2018 11:27:34 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D9AC2166BC6; Wed, 25 Apr 2018 11:27:33 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:08 +0200 Message-Id: <20180425112723.1111-7-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:34 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:34 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 06/21] migration: Create multifd channels X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" In both sides. We still don't transmit anything through them. Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- migration/ram.c | 52 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 3a01472835..54b1f8e836 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -404,6 +404,7 @@ struct MultiFDSendParams { uint8_t id; char *name; QemuThread thread; + QIOChannel *c; QemuSemaphore sem; QemuMutex mutex; bool running; @@ -456,6 +457,8 @@ int multifd_save_cleanup(Error **errp) if (p->running) { qemu_thread_join(&p->thread); } + socket_send_channel_destroy(p->c); + p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); g_free(p->name); @@ -489,6 +492,27 @@ static void *multifd_send_thread(void *opaque) return NULL; } +static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) +{ + MultiFDSendParams *p = opaque; + QIOChannel *sioc = QIO_CHANNEL(qio_task_get_source(task)); + Error *local_err = NULL; + + if (qio_task_propagate_error(task, &local_err)) { + if (multifd_save_cleanup(&local_err) != 0) { + migrate_set_error(migrate_get_current(), local_err); + } + } else { + p->c = QIO_CHANNEL(sioc); + qio_channel_set_delay(p->c, false); + p->running = true; + qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, + QEMU_THREAD_JOINABLE); + + atomic_inc(&multifd_send_state->count); + } +} + int multifd_save_setup(void) { int thread_count; @@ -509,11 +533,7 @@ int multifd_save_setup(void) p->quit = false; p->id = i; p->name = g_strdup_printf("multifdsend_%d", i); - p->running = true; - qemu_thread_create(&p->thread, p->name, multifd_send_thread, p, - QEMU_THREAD_JOINABLE); - - atomic_inc(&multifd_send_state->count); + socket_send_channel_create(multifd_new_send_channel_async, p); } return 0; } @@ -522,6 +542,7 @@ struct MultiFDRecvParams { uint8_t id; char *name; QemuThread thread; + QIOChannel *c; QemuSemaphore sem; QemuMutex mutex; bool running; @@ -574,6 +595,8 @@ int multifd_load_cleanup(Error **errp) if (p->running) { qemu_thread_join(&p->thread); } + object_unref(OBJECT(p->c)); + p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); g_free(p->name); @@ -628,10 +651,6 @@ int multifd_load_setup(void) p->quit = false; p->id = i; p->name = g_strdup_printf("multifdrecv_%d", i); - p->running = true; - qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, - QEMU_THREAD_JOINABLE); - atomic_inc(&multifd_recv_state->count); } return 0; } @@ -649,7 +668,20 @@ bool multifd_recv_all_channels_created(void) void multifd_recv_new_channel(QIOChannel *ioc) { - /* nothing to do yet */ + MultiFDRecvParams *p; + /* we need to invent channels id's until we transmit */ + /* we will remove this on a later patch */ + static int i; + + p = &multifd_recv_state->params[i]; + i++; + p->c = ioc; + object_ref(OBJECT(ioc)); + + p->running = true; + qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, + QEMU_THREAD_JOINABLE); + atomic_inc(&multifd_recv_state->count); } /** From patchwork Wed Apr 25 11:27:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904156 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ8n4RkSz9s28 for ; Wed, 25 Apr 2018 21:37:45 +1000 (AEST) Received: from localhost ([::1]:35894 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIk3-000265-LO for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:37:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34141) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaH-0002l9-2h for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaG-0004vH-3z for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:37 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33576 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaG-0004uW-0I for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:36 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9F74D738E0 for ; Wed, 25 Apr 2018 11:27:35 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 98088215CDD1; Wed, 25 Apr 2018 11:27:34 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:09 +0200 Message-Id: <20180425112723.1111-8-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:35 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:35 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 07/21] migration: Delay start of migration main routines X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We need to make sure that we have started all the multifd threads. Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 4 ++-- migration/migration.h | 1 + migration/ram.c | 3 +++ migration/socket.c | 4 ++++ 4 files changed, 10 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 98f85e982c..9b510a809a 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -447,7 +447,7 @@ static void migration_incoming_setup(QEMUFile *f) qemu_file_set_blocking(f, false); } -static void migration_incoming_process(void) +void migration_incoming_process(void) { Coroutine *co = qemu_coroutine_create(process_incoming_migration_co, NULL); qemu_coroutine_enter(co); @@ -465,7 +465,7 @@ void migration_ioc_process_incoming(QIOChannel *ioc) if (!mis->from_src_file) { QEMUFile *f = qemu_fopen_channel_input(ioc); - migration_fd_process_incoming(f); + migration_incoming_setup(f); return; } multifd_recv_new_channel(ioc); diff --git a/migration/migration.h b/migration/migration.h index 4774ee305f..e20f680bac 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -188,6 +188,7 @@ void migrate_set_state(int *state, int old_state, int new_state); void migration_fd_process_incoming(QEMUFile *f); void migration_ioc_process_incoming(QIOChannel *ioc); +void migration_incoming_process(void); bool migration_has_all_channels(void); diff --git a/migration/ram.c b/migration/ram.c index 54b1f8e836..5a87d74862 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -682,6 +682,9 @@ void multifd_recv_new_channel(QIOChannel *ioc) qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, QEMU_THREAD_JOINABLE); atomic_inc(&multifd_recv_state->count); + if (multifd_recv_state->count == migrate_multifd_channels()) { + migration_incoming_process(); + } } /** diff --git a/migration/socket.c b/migration/socket.c index 893a04f4cc..d4a2c1e916 100644 --- a/migration/socket.c +++ b/migration/socket.c @@ -171,6 +171,10 @@ static void socket_accept_incoming_migration(QIONetListener *listener, qio_net_listener_disconnect(listener); object_unref(OBJECT(listener)); + + if (!migrate_use_multifd()) { + migration_incoming_process(); + } } } From patchwork Wed Apr 25 11:27:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904160 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJCn3tQbz9rxs for ; Wed, 25 Apr 2018 21:40:21 +1000 (AEST) Received: from localhost ([::1]:35904 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBImZ-0004FU-HN for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:40:19 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34164) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaI-0002nB-NH for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaH-0004xW-Fy for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:38 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33580 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaH-0004wc-At for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:37 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E8983EC013 for ; Wed, 25 Apr 2018 11:27:36 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id E2847215CDCE; Wed, 25 Apr 2018 11:27:35 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:10 +0200 Message-Id: <20180425112723.1111-9-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:36 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 08/21] migration: Transmit initial package through the multifd channels X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela Reviewed-by: Daniel P. Berrangé --- Be network agnostic. Add error checking for all values. --- migration/ram.c | 101 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 96 insertions(+), 5 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 5a87d74862..1aab392d8f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -52,6 +52,8 @@ #include "qemu/rcu_queue.h" #include "migration/colo.h" #include "migration/block.h" +#include "sysemu/sysemu.h" +#include "qemu/uuid.h" /***********************************************************/ /* ram save/restore */ @@ -400,6 +402,16 @@ static void compress_threads_save_setup(void) /* Multiple fd's */ +#define MULTIFD_MAGIC 0x11223344U +#define MULTIFD_VERSION 1 + +typedef struct { + uint32_t magic; + uint32_t version; + unsigned char uuid[16]; /* QemuUUID */ + uint8_t id; +} __attribute__((packed)) MultiFDInit_t; + struct MultiFDSendParams { uint8_t id; char *name; @@ -412,6 +424,65 @@ struct MultiFDSendParams { }; typedef struct MultiFDSendParams MultiFDSendParams; +static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) +{ + MultiFDInit_t msg; + int ret; + + msg.magic = cpu_to_be32(MULTIFD_MAGIC); + msg.version = cpu_to_be32(MULTIFD_VERSION); + msg.id = p->id; + memcpy(msg.uuid, &qemu_uuid.data, sizeof(msg.uuid)); + + ret = qio_channel_write_all(p->c, (char *)&msg, sizeof(msg), errp); + if (ret != 0) { + return -1; + } + return 0; +} + +static int multifd_recv_initial_packet(QIOChannel *c, Error **errp) +{ + MultiFDInit_t msg; + int ret; + + ret = qio_channel_read_all(c, (char *)&msg, sizeof(msg), errp); + if (ret != 0) { + return -1; + } + + be32_to_cpus(&msg.magic); + be32_to_cpus(&msg.version); + + if (msg.magic != MULTIFD_MAGIC) { + error_setg(errp, "multifd: received packet magic %d " + "expected %d", msg.magic, MULTIFD_MAGIC); + return -1; + } + + if (msg.version != MULTIFD_VERSION) { + error_setg(errp, "multifd: received packet version %d " + "expected %d", msg.version, MULTIFD_VERSION); + return -1; + } + + if (memcmp(msg.uuid, &qemu_uuid, sizeof(qemu_uuid))) { + char *uuid = qemu_uuid_unparse_strdup(&qemu_uuid); + error_setg(errp, "multifd: received uuid '%s' and expected " + "uuid '%s' for channel %hhd", msg.uuid, uuid, msg.id); + g_free(uuid); + return -1; + } + + if (msg.id > migrate_multifd_channels()) { + error_setg(errp, "multifd: received channel version %d " + "expected %d", msg.version, MULTIFD_VERSION); + return -1; + } + + return msg.id; +} + struct { MultiFDSendParams *params; /* number of created threads */ @@ -474,6 +545,11 @@ int multifd_save_cleanup(Error **errp) static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p = opaque; + Error *local_err = NULL; + + if (multifd_send_initial_packet(p, &local_err) < 0) { + goto out; + } while (true) { qemu_mutex_lock(&p->mutex); @@ -485,6 +561,11 @@ static void *multifd_send_thread(void *opaque) qemu_sem_wait(&p->sem); } +out: + if (local_err) { + multifd_send_terminate_threads(local_err); + } + qemu_mutex_lock(&p->mutex); p->running = false; qemu_mutex_unlock(&p->mutex); @@ -669,12 +750,22 @@ bool multifd_recv_all_channels_created(void) void multifd_recv_new_channel(QIOChannel *ioc) { MultiFDRecvParams *p; - /* we need to invent channels id's until we transmit */ - /* we will remove this on a later patch */ - static int i; + Error *local_err = NULL; + int id; - p = &multifd_recv_state->params[i]; - i++; + id = multifd_recv_initial_packet(ioc, &local_err); + if (id < 0) { + multifd_recv_terminate_threads(local_err); + return; + } + + p = &multifd_recv_state->params[id]; + if (p->c != NULL) { + error_setg(&local_err, "multifd: received id '%d' already setup'", + id); + multifd_recv_terminate_threads(local_err); + return; + } p->c = ioc; object_ref(OBJECT(ioc)); From patchwork Wed Apr 25 11:27:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904149 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ496Nq7z9rvt for ; Wed, 25 Apr 2018 21:33:45 +1000 (AEST) Received: from localhost ([::1]:35868 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIgC-0007IT-13 for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:33:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34245) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaM-0002rg-RU for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaI-0004zJ-OS for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33582 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaI-0004yg-JP for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:38 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3DAE1EC013 for ; Wed, 25 Apr 2018 11:27:38 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3775A215CDCE; Wed, 25 Apr 2018 11:27:37 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:11 +0200 Message-Id: <20180425112723.1111-10-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:38 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:38 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 09/21] migration: Define MultifdRecvParams sooner X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Once there, we don't need the struct names anywhere, just the typedefs. And now also document all fields. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 1aab392d8f..ffefa73099 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -412,17 +412,45 @@ typedef struct { uint8_t id; } __attribute__((packed)) MultiFDInit_t; -struct MultiFDSendParams { +typedef struct { + /* this fields are not changed once the thread is created */ + /* channel number */ uint8_t id; + /* channel thread name */ char *name; + /* channel thread id */ QemuThread thread; + /* communication channel */ QIOChannel *c; + /* sem where to wait for more work */ QemuSemaphore sem; + /* this mutex protects the following parameters */ QemuMutex mutex; + /* is this channel thread running */ bool running; + /* should this thread finish */ bool quit; -}; -typedef struct MultiFDSendParams MultiFDSendParams; +} MultiFDSendParams; + +typedef struct { + /* this fields are not changed once the thread is created */ + /* channel number */ + uint8_t id; + /* channel thread name */ + char *name; + /* channel thread id */ + QemuThread thread; + /* communication channel */ + QIOChannel *c; + /* sem where to wait for more work */ + QemuSemaphore sem; + /* this mutex protects the following parameters */ + QemuMutex mutex; + /* is this channel thread running */ + bool running; + /* should this thread finish */ + bool quit; +} MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) { @@ -619,18 +647,6 @@ int multifd_save_setup(void) return 0; } -struct MultiFDRecvParams { - uint8_t id; - char *name; - QemuThread thread; - QIOChannel *c; - QemuSemaphore sem; - QemuMutex mutex; - bool running; - bool quit; -}; -typedef struct MultiFDRecvParams MultiFDRecvParams; - struct { MultiFDRecvParams *params; /* number of created threads */ From patchwork Wed Apr 25 11:27:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904152 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ5R0QN4z9rvt for ; Wed, 25 Apr 2018 21:34:51 +1000 (AEST) Received: from localhost ([::1]:35879 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIhF-00087L-2s for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:34:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34242) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaM-0002rX-Q9 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaK-00051h-2q for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:42390 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaJ-000512-Uk for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:40 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8B9C081A88D2 for ; Wed, 25 Apr 2018 11:27:39 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7FEA02166BC6; Wed, 25 Apr 2018 11:27:38 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:12 +0200 Message-Id: <20180425112723.1111-11-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:39 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 10/21] migration: Create multipage support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We only create/destry the page list here. We will use it later. Signed-off-by: Juan Quintela Reviewed-by: Peter Xu --- migration/ram.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index ffefa73099..b19300992e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -412,6 +412,20 @@ typedef struct { uint8_t id; } __attribute__((packed)) MultiFDInit_t; +typedef struct { + /* number of used pages */ + uint32_t used; + /* number of allocated pages */ + uint32_t allocated; + /* global number of generated multifd packets */ + uint32_t seq; + /* offset of each page */ + ram_addr_t *offset; + /* pointer to each page */ + struct iovec *iov; + RAMBlock *block; +} MultiFDPages_t; + typedef struct { /* this fields are not changed once the thread is created */ /* channel number */ @@ -430,6 +444,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* array of pages to sent */ + MultiFDPages_t *pages; } MultiFDSendParams; typedef struct { @@ -450,6 +466,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* array of pages to receive */ + MultiFDPages_t *pages; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -511,10 +529,35 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp) return msg.id; } +static void multifd_pages_init(MultiFDPages_t **ppages, size_t size) +{ + MultiFDPages_t *pages = g_new0(MultiFDPages_t, 1); + + pages->allocated = size; + pages->iov = g_new0(struct iovec, size); + pages->offset = g_new0(ram_addr_t, size); + *ppages = pages; +} + +static void multifd_pages_clear(MultiFDPages_t *pages) +{ + pages->used = 0; + pages->allocated = 0; + pages->seq = 0; + pages->block = NULL; + g_free(pages->iov); + pages->iov = NULL; + g_free(pages->offset); + pages->offset = NULL; + g_free(pages); +} + struct { MultiFDSendParams *params; /* number of created threads */ int count; + /* array of pages to sent */ + MultiFDPages_t *pages; } *multifd_send_state; static void multifd_send_terminate_threads(Error *err) @@ -562,9 +605,13 @@ int multifd_save_cleanup(Error **errp) qemu_sem_destroy(&p->sem); g_free(p->name); p->name = NULL; + multifd_pages_clear(p->pages); + p->pages = NULL; } g_free(multifd_send_state->params); multifd_send_state->params = NULL; + multifd_pages_clear(multifd_send_state->pages); + multifd_send_state->pages = NULL; g_free(multifd_send_state); multifd_send_state = NULL; return ret; @@ -625,6 +672,7 @@ static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) int multifd_save_setup(void) { int thread_count; + uint32_t page_count = migrate_multifd_page_count(); uint8_t i; if (!migrate_use_multifd()) { @@ -634,6 +682,8 @@ int multifd_save_setup(void) multifd_send_state = g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); atomic_set(&multifd_send_state->count, 0); + multifd_pages_init(&multifd_send_state->pages, page_count); + for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -641,6 +691,7 @@ int multifd_save_setup(void) qemu_sem_init(&p->sem, 0); p->quit = false; p->id = i; + multifd_pages_init(&p->pages, page_count); p->name = g_strdup_printf("multifdsend_%d", i); socket_send_channel_create(multifd_new_send_channel_async, p); } @@ -698,6 +749,8 @@ int multifd_load_cleanup(Error **errp) qemu_sem_destroy(&p->sem); g_free(p->name); p->name = NULL; + multifd_pages_clear(p->pages); + p->pages = NULL; } g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; @@ -731,6 +784,7 @@ static void *multifd_recv_thread(void *opaque) int multifd_load_setup(void) { int thread_count; + uint32_t page_count = migrate_multifd_page_count(); uint8_t i; if (!migrate_use_multifd()) { @@ -740,6 +794,7 @@ int multifd_load_setup(void) multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); atomic_set(&multifd_recv_state->count, 0); + for (i = 0; i < thread_count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; @@ -747,6 +802,7 @@ int multifd_load_setup(void) qemu_sem_init(&p->sem, 0); p->quit = false; p->id = i; + multifd_pages_init(&p->pages, page_count); p->name = g_strdup_printf("multifdrecv_%d", i); } return 0; From patchwork Wed Apr 25 11:27:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904158 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ913RnTz9rvt for ; Wed, 25 Apr 2018 21:37:57 +1000 (AEST) Received: from localhost ([::1]:35896 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIkF-0002Gy-Fr for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:37:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34243) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaM-0002ra-Qw for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaL-00053y-Cb for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:46190 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaL-000534-7P for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:41 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D6D8E406C740 for ; Wed, 25 Apr 2018 11:27:40 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id CD6FE215CDD1; Wed, 25 Apr 2018 11:27:39 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:13 +0200 Message-Id: <20180425112723.1111-12-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:40 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:40 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 11/21] migration: Create multifd packet X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We still don't put anything there. Signed-off-by: Juan Quintela --- migration/ram.c | 137 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 136 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index b19300992e..804c83ed89 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -412,6 +412,17 @@ typedef struct { uint8_t id; } __attribute__((packed)) MultiFDInit_t; +typedef struct { + uint32_t magic; + uint32_t version; + uint32_t flags; + uint32_t size; + uint32_t used; + uint32_t seq; + char ramblock[256]; + uint64_t offset[]; +} __attribute__((packed)) MultiFDPacket_t; + typedef struct { /* number of used pages */ uint32_t used; @@ -446,6 +457,14 @@ typedef struct { bool quit; /* array of pages to sent */ MultiFDPages_t *pages; + /* packet allocated len */ + uint32_t packet_len; + /* pointer to the packet */ + MultiFDPacket_t *packet; + /* multifd flags for each packet */ + uint32_t flags; + /* global number of generated multifd packets */ + uint32_t seq; } MultiFDSendParams; typedef struct { @@ -468,6 +487,14 @@ typedef struct { bool quit; /* array of pages to receive */ MultiFDPages_t *pages; + /* packet allocated len */ + uint32_t packet_len; + /* pointer to the packet */ + MultiFDPacket_t *packet; + /* multifd flags for each packet */ + uint32_t flags; + /* global number of generated multifd packets */ + uint32_t seq; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -552,6 +579,91 @@ static void multifd_pages_clear(MultiFDPages_t *pages) g_free(pages); } +static void multifd_send_fill_packet(MultiFDSendParams *p) +{ + MultiFDPacket_t *packet = p->packet; + int i; + + packet->magic = cpu_to_be32(MULTIFD_MAGIC); + packet->version = cpu_to_be32(MULTIFD_VERSION); + packet->flags = cpu_to_be32(p->flags); + packet->size = cpu_to_be32(migrate_multifd_page_count()); + packet->used = cpu_to_be32(p->pages->used); + packet->seq = cpu_to_be32(p->seq); + + if (p->pages->block) { + strncpy(packet->ramblock, p->pages->block->idstr, 256); + } + + for (i = 0; i < p->pages->used; i++) { + packet->offset[i] = cpu_to_be64(p->pages->offset[i]); + } +} + +static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +{ + MultiFDPacket_t *packet = p->packet; + RAMBlock *block; + int i; + + /* ToDo: We can't use it until we haven't received a message */ + return 0; + + be32_to_cpus(&packet->magic); + if (packet->magic != MULTIFD_MAGIC) { + error_setg(errp, "multifd: received packet " + "version %d and expected version %d", + packet->magic, MULTIFD_VERSION); + return -1; + } + + be32_to_cpus(&packet->version); + if (packet->version != MULTIFD_VERSION) { + error_setg(errp, "multifd: received packet " + "version %d and expected version %d", + packet->version, MULTIFD_VERSION); + return -1; + } + + p->flags = be32_to_cpu(packet->flags); + + be32_to_cpus(&packet->size); + if (packet->size > migrate_multifd_page_count()) { + error_setg(errp, "multifd: received packet " + "with size %d and expected maximum size %d", + packet->size, migrate_multifd_page_count()) ; + return -1; + } + + p->pages->used = be32_to_cpu(packet->used); + if (p->pages->used > packet->size) { + error_setg(errp, "multifd: received packet " + "with size %d and expected maximum size %d", + p->pages->used, packet->size) ; + return -1; + } + + p->seq = be32_to_cpu(packet->seq); + + if (p->pages->used) { + block = qemu_ram_block_by_name(packet->ramblock); + if (!block) { + error_setg(errp, "multifd: unknown ram block %s", + packet->ramblock); + return -1; + } + } + + for (i = 0; i < p->pages->used; i++) { + ram_addr_t offset = be64_to_cpu(packet->offset[i]); + + p->pages->iov[i].iov_base = block->host + offset; + p->pages->iov[i].iov_len = TARGET_PAGE_SIZE; + } + + return 0; +} + struct { MultiFDSendParams *params; /* number of created threads */ @@ -607,6 +719,9 @@ int multifd_save_cleanup(Error **errp) p->name = NULL; multifd_pages_clear(p->pages); p->pages = NULL; + p->packet_len = 0; + g_free(p->packet); + p->packet = NULL; } g_free(multifd_send_state->params); multifd_send_state->params = NULL; @@ -628,6 +743,7 @@ static void *multifd_send_thread(void *opaque) while (true) { qemu_mutex_lock(&p->mutex); + multifd_send_fill_packet(p); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -692,6 +808,9 @@ int multifd_save_setup(void) p->quit = false; p->id = i; multifd_pages_init(&p->pages, page_count); + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(ram_addr_t) * page_count; + p->packet = g_malloc0(p->packet_len); p->name = g_strdup_printf("multifdsend_%d", i); socket_send_channel_create(multifd_new_send_channel_async, p); } @@ -751,6 +870,9 @@ int multifd_load_cleanup(Error **errp) p->name = NULL; multifd_pages_clear(p->pages); p->pages = NULL; + p->packet_len = 0; + g_free(p->packet); + p->packet = NULL; } g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; @@ -763,10 +885,20 @@ int multifd_load_cleanup(Error **errp) static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; + Error *local_err = NULL; + int ret; while (true) { qemu_mutex_lock(&p->mutex); - if (p->quit) { + if (false) { + /* ToDo: Packet reception goes here */ + + ret = multifd_recv_unfill_packet(p, &local_err); + qemu_mutex_unlock(&p->mutex); + if (ret) { + break; + } + } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } @@ -803,6 +935,9 @@ int multifd_load_setup(void) p->quit = false; p->id = i; multifd_pages_init(&p->pages, page_count); + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(ram_addr_t) * page_count; + p->packet = g_malloc0(p->packet_len); p->name = g_strdup_printf("multifdrecv_%d", i); } return 0; From patchwork Wed Apr 25 11:27:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904162 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJDK6zMhz9rxs for ; Wed, 25 Apr 2018 21:40:49 +1000 (AEST) Received: from localhost ([::1]:35909 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIn2-0004fG-2A for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:40:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34278) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaN-0002sb-Qn for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaM-00056J-Ni for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:43 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:46192 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaM-00055Y-I7 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:42 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 2E027406C740 for ; Wed, 25 Apr 2018 11:27:42 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 24F26215CDD1; Wed, 25 Apr 2018 11:27:41 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:14 +0200 Message-Id: <20180425112723.1111-13-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:42 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Wed, 25 Apr 2018 11:27:42 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 12/21] migration: Add multifd traces for start/end thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We want to know how many pages/packets each channel has sent. Add counters for those. Signed-off-by: Juan Quintela --- migration/ram.c | 20 ++++++++++++++++++++ migration/trace-events | 4 ++++ 2 files changed, 24 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 804c83ed89..0f1340b4e3 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -465,6 +465,10 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint32_t seq; + /* packets sent through this channel */ + uint32_t num_packets; + /* pages sent through this channel */ + uint32_t num_pages; } MultiFDSendParams; typedef struct { @@ -495,6 +499,10 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint32_t seq; + /* packets sent through this channel */ + uint32_t num_packets; + /* pages sent through this channel */ + uint32_t num_pages; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -737,9 +745,13 @@ static void *multifd_send_thread(void *opaque) MultiFDSendParams *p = opaque; Error *local_err = NULL; + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } + /* initial packet */ + p->num_packets = 1; while (true) { qemu_mutex_lock(&p->mutex); @@ -761,6 +773,8 @@ out: p->running = false; qemu_mutex_unlock(&p->mutex); + trace_multifd_send_thread_end(p->id, p->num_packets, p->num_pages); + return NULL; } @@ -888,6 +902,8 @@ static void *multifd_recv_thread(void *opaque) Error *local_err = NULL; int ret; + trace_multifd_recv_thread_start(p->id); + while (true) { qemu_mutex_lock(&p->mutex); if (false) { @@ -910,6 +926,8 @@ static void *multifd_recv_thread(void *opaque) p->running = false; qemu_mutex_unlock(&p->mutex); + trace_multifd_recv_thread_end(p->id, p->num_packets, p->num_pages); + return NULL; } @@ -975,6 +993,8 @@ void multifd_recv_new_channel(QIOChannel *ioc) } p->c = ioc; object_ref(OBJECT(ioc)); + /* initial packet */ + p->num_packets = 1; p->running = true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, diff --git a/migration/trace-events b/migration/trace-events index a180d7b008..e480eb050e 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -77,6 +77,10 @@ ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" +multifd_send_thread_start(uint8_t id) "%d" +multifd_send_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" +multifd_recv_thread_start(uint8_t id) "%d" +multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Wed Apr 25 11:27:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904164 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJH400XGz9rxs for ; Wed, 25 Apr 2018 21:43:11 +1000 (AEST) Received: from localhost ([::1]:35922 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIpJ-0006jD-9O for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:43:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34294) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaO-0002tg-Td for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaO-00058i-0P for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:44 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45566 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaN-000585-SY for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:43 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 781D84023155 for ; Wed, 25 Apr 2018 11:27:43 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7068A215CDD1; Wed, 25 Apr 2018 11:27:42 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:15 +0200 Message-Id: <20180425112723.1111-14-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:43 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:43 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 13/21] migration: Calculate transferred ram correctly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" On multifd we send data from more places that main channel. Signed-off-by: Juan Quintela --- migration/migration.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 9b510a809a..75d30661e9 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2246,12 +2246,19 @@ static void migration_update_counters(MigrationState *s, { uint64_t transferred, time_spent; double bandwidth; + uint64_t now; if (current_time < s->iteration_start_time + BUFFER_DELAY) { return; } - transferred = qemu_ftell(s->to_dst_file) - s->iteration_initial_bytes; + if (migrate_use_multifd()) { + now = ram_counters.normal * qemu_target_page_size() + + qemu_ftell(s->to_dst_file); + } else { + now = qemu_ftell(s->to_dst_file); + } + transferred = now - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit; @@ -2271,7 +2278,7 @@ static void migration_update_counters(MigrationState *s, qemu_file_reset_rate_limit(s->to_dst_file); s->iteration_start_time = current_time; - s->iteration_initial_bytes = qemu_ftell(s->to_dst_file); + s->iteration_initial_bytes = now; trace_migrate_transferred(transferred, time_spent, bandwidth, s->threshold_size); From patchwork Wed Apr 25 11:27:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904161 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJD74zcpz9s0v for ; Wed, 25 Apr 2018 21:40:39 +1000 (AEST) Received: from localhost ([::1]:35906 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBImq-0004UE-Sp for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:40:36 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34308) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaQ-0002v3-7C for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:47 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaP-0005An-9b for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:46 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:42398 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaP-0005A1-5h for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:45 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C07CF818AAE8 for ; Wed, 25 Apr 2018 11:27:44 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id B98562166BC6; Wed, 25 Apr 2018 11:27:43 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:16 +0200 Message-Id: <20180425112723.1111-15-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:44 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Wed, 25 Apr 2018 11:27:44 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 14/21] migration: Multifd channels always wait on the sem X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Either for quit, sync or packet, we first wake them. Signed-off-by: Juan Quintela --- migration/ram.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 0f1340b4e3..21b448c4ed 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -754,6 +754,7 @@ static void *multifd_send_thread(void *opaque) p->num_packets = 1; while (true) { + qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); multifd_send_fill_packet(p); if (p->quit) { @@ -761,7 +762,9 @@ static void *multifd_send_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + /* this is impossible */ + error_setg(&local_err, "multifd_send_thread: Unknown command"); + break; } out: @@ -905,6 +908,7 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { + qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); if (false) { /* ToDo: Packet reception goes here */ @@ -919,9 +923,14 @@ static void *multifd_recv_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + /* this is impossible */ + error_setg(&local_err, "multifd_recv_thread: Unknown command"); + break; } + if (local_err) { + multifd_recv_terminate_threads(local_err); + } qemu_mutex_lock(&p->mutex); p->running = false; qemu_mutex_unlock(&p->mutex); From patchwork Wed Apr 25 11:27:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904166 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJHH36SCz9rxs for ; Wed, 25 Apr 2018 21:43:23 +1000 (AEST) Received: from localhost ([::1]:35924 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIpV-0006t2-EM for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:43:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34319) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaR-0002wk-OJ for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaQ-0005CS-Jc for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:47 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45568 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaQ-0005Bo-E6 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:46 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 1628A402314E for ; Wed, 25 Apr 2018 11:27:46 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0F483215CDC8; Wed, 25 Apr 2018 11:27:44 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:17 +0200 Message-Id: <20180425112723.1111-16-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:46 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:46 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 15/21] migration: Add block where to send/receive packets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Once there add tracepoints. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 49 +++++++++++++++++++++++++++++++++++++----- migration/trace-events | 2 ++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 21b448c4ed..c4c185cc4c 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -455,6 +455,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* thread has work to do */ + int pending_job; /* array of pages to sent */ MultiFDPages_t *pages; /* packet allocated len */ @@ -489,6 +491,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* thread has work to do */ + bool pending_job; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -756,8 +760,28 @@ static void *multifd_send_thread(void *opaque) while (true) { qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - multifd_send_fill_packet(p); - if (p->quit) { + + if (p->pending_job) { + uint32_t used = p->pages->used; + uint32_t seq = p->seq; + uint32_t flags = p->flags; + + multifd_send_fill_packet(p); + p->flags = 0; + p->num_packets++; + p->num_pages += used; + p->pages->used = 0; + qemu_mutex_unlock(&p->mutex); + + trace_multifd_send(p->id, seq, used, flags); + + /* ToDo: send packet here */ + + qemu_mutex_lock(&p->mutex); + p->pending_job--; + qemu_mutex_unlock(&p->mutex); + continue; + } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } @@ -823,6 +847,7 @@ int multifd_save_setup(void) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); p->quit = false; + p->pending_job = 0; p->id = i; multifd_pages_init(&p->pages, page_count); p->packet_len = sizeof(MultiFDPacket_t) @@ -910,14 +935,27 @@ static void *multifd_recv_thread(void *opaque) while (true) { qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - if (false) { - /* ToDo: Packet reception goes here */ + if (p->pending_job) { + uint32_t used; + uint32_t flags; + qemu_mutex_unlock(&p->mutex); + /* ToDo: recv packet here */ + + qemu_mutex_lock(&p->mutex); ret = multifd_recv_unfill_packet(p, &local_err); - qemu_mutex_unlock(&p->mutex); if (ret) { + qemu_mutex_unlock(&p->mutex); break; } + + used = p->pages->used; + flags = p->flags; + trace_multifd_recv(p->id, p->seq, used, flags); + p->pending_job = false; + p->num_packets++; + p->num_pages += used; + qemu_mutex_unlock(&p->mutex); } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -960,6 +998,7 @@ int multifd_load_setup(void) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); p->quit = false; + p->pending_job = false; p->id = i; multifd_pages_init(&p->pages, page_count); p->packet_len = sizeof(MultiFDPacket_t) diff --git a/migration/trace-events b/migration/trace-events index e480eb050e..9eee048287 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -81,6 +81,8 @@ multifd_send_thread_start(uint8_t id) "%d" multifd_send_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_recv_thread_start(uint8_t id) "%d" multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" +multifd_send(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" +multifd_recv(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Wed Apr 25 11:27:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904153 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJ7r52Vpz9s0v for ; Wed, 25 Apr 2018 21:36:56 +1000 (AEST) Received: from localhost ([::1]:35890 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIjG-0001OX-8E for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:36:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34356) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaW-00032q-0F for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaR-0005EP-Uh for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:52 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45570 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaR-0005Dm-Oc for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:47 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 61BAA402314E for ; Wed, 25 Apr 2018 11:27:47 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 57B08215CDC8; Wed, 25 Apr 2018 11:27:46 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:18 +0200 Message-Id: <20180425112723.1111-17-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:47 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:47 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 16/21] migration: Synchronize multifd threads with main thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We synchronize all threads each RAM_SAVE_FLAG_EOS. Bitmap synchronizations don't happen inside a ram section, so we are safe about two channels trying to overwrite the same memory. Signed-off-by: Juan Quintela --- migration/ram.c | 118 +++++++++++++++++++++++++++++++++++++---- migration/trace-events | 6 +++ 2 files changed, 113 insertions(+), 11 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index c4c185cc4c..398cb0af3b 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -405,6 +405,8 @@ static void compress_threads_save_setup(void) #define MULTIFD_MAGIC 0x11223344U #define MULTIFD_VERSION 1 +#define MULTIFD_FLAG_SYNC (1 << 0) + typedef struct { uint32_t magic; uint32_t version; @@ -471,6 +473,8 @@ typedef struct { uint32_t num_packets; /* pages sent through this channel */ uint32_t num_pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; } MultiFDSendParams; typedef struct { @@ -507,6 +511,8 @@ typedef struct { uint32_t num_packets; /* pages sent through this channel */ uint32_t num_pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -682,6 +688,10 @@ struct { int count; /* array of pages to sent */ MultiFDPages_t *pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; + /* global number of generated multifd packets */ + uint32_t seq; } *multifd_send_state; static void multifd_send_terminate_threads(Error *err) @@ -727,6 +737,7 @@ int multifd_save_cleanup(Error **errp) p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; multifd_pages_clear(p->pages); @@ -735,6 +746,7 @@ int multifd_save_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_send_state->sem_sync); g_free(multifd_send_state->params); multifd_send_state->params = NULL; multifd_pages_clear(multifd_send_state->pages); @@ -744,6 +756,33 @@ int multifd_save_cleanup(Error **errp) return ret; } +static void multifd_send_sync_main(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + + trace_multifd_send_sync_main_signal(p->id); + + qemu_mutex_lock(&p->mutex); + p->flags |= MULTIFD_FLAG_SYNC; + p->pending_job++; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + + trace_multifd_send_sync_main_wait(p->id); + qemu_sem_wait(&multifd_send_state->sem_sync); + } + trace_multifd_send_sync_main(multifd_send_state->seq); +} + static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p = opaque; @@ -778,17 +817,20 @@ static void *multifd_send_thread(void *opaque) /* ToDo: send packet here */ qemu_mutex_lock(&p->mutex); + p->flags = 0; p->pending_job--; qemu_mutex_unlock(&p->mutex); - continue; + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_send_state->sem_sync); + } } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; + } else { + qemu_mutex_unlock(&p->mutex); + /* sometimes there are spurious wakeups */ } - qemu_mutex_unlock(&p->mutex); - /* this is impossible */ - error_setg(&local_err, "multifd_send_thread: Unknown command"); - break; } out: @@ -840,12 +882,14 @@ int multifd_save_setup(void) multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); atomic_set(&multifd_send_state->count, 0); multifd_pages_init(&multifd_send_state->pages, page_count); + qemu_sem_init(&multifd_send_state->sem_sync, 0); for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->sem_sync, 0); p->quit = false; p->pending_job = 0; p->id = i; @@ -863,6 +907,10 @@ struct { MultiFDRecvParams *params; /* number of created threads */ int count; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; + /* global number of generated multifd packets */ + uint32_t seq; } *multifd_recv_state; static void multifd_recv_terminate_threads(Error *err) @@ -908,6 +956,7 @@ int multifd_load_cleanup(Error **errp) p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; multifd_pages_clear(p->pages); @@ -916,6 +965,7 @@ int multifd_load_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_recv_state->sem_sync); g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; g_free(multifd_recv_state); @@ -924,6 +974,42 @@ int multifd_load_cleanup(Error **errp) return ret; } +static void multifd_recv_sync_main(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_signal(p->id); + qemu_mutex_lock(&p->mutex); + p->pending_job = true; + qemu_mutex_unlock(&p->mutex); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_wait(p->id); + qemu_sem_wait(&multifd_recv_state->sem_sync); + qemu_mutex_lock(&p->mutex); + if (multifd_recv_state->seq < p->seq) { + multifd_recv_state->seq = p->seq; + } + qemu_mutex_unlock(&p->mutex); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_signal(p->id); + + qemu_sem_post(&p->sem_sync); + } + trace_multifd_recv_sync_main(multifd_recv_state->seq); +} + static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; @@ -933,9 +1019,8 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { - qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - if (p->pending_job) { + if (true || p->pending_job) { uint32_t used; uint32_t flags; qemu_mutex_unlock(&p->mutex); @@ -956,14 +1041,18 @@ static void *multifd_recv_thread(void *opaque) p->num_packets++; p->num_pages += used; qemu_mutex_unlock(&p->mutex); + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_recv_state->sem_sync); + qemu_sem_wait(&p->sem_sync); + } } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; + } else { + qemu_mutex_unlock(&p->mutex); + /* sometimes there are spurious wakeups */ } - qemu_mutex_unlock(&p->mutex); - /* this is impossible */ - error_setg(&local_err, "multifd_recv_thread: Unknown command"); - break; } if (local_err) { @@ -991,12 +1080,14 @@ int multifd_load_setup(void) multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); atomic_set(&multifd_recv_state->count, 0); + qemu_sem_init(&multifd_recv_state->sem_sync, 0); for (i = 0; i < thread_count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->sem_sync, 0); p->quit = false; p->pending_job = false; p->id = i; @@ -2695,6 +2786,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) ram_control_before_iterate(f, RAM_CONTROL_SETUP); ram_control_after_iterate(f, RAM_CONTROL_SETUP); + multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; @@ -2770,6 +2862,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) */ ram_control_after_iterate(f, RAM_CONTROL_ROUND); + multifd_send_sync_main(); out: qemu_put_be64(f, RAM_SAVE_FLAG_EOS); ram_counters.transferred += 8; @@ -2823,6 +2916,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) rcu_read_unlock(); + multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; @@ -3253,6 +3347,7 @@ static int ram_load_postcopy(QEMUFile *f) break; case RAM_SAVE_FLAG_EOS: /* normal exit */ + multifd_recv_sync_main(); break; default: error_report("Unknown combination of migration flags: %#x" @@ -3438,6 +3533,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) break; case RAM_SAVE_FLAG_EOS: /* normal exit */ + multifd_recv_sync_main(); break; default: if (flags & RAM_SAVE_FLAG_HOOK) { diff --git a/migration/trace-events b/migration/trace-events index 9eee048287..b0ab8e2d03 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -83,6 +83,12 @@ multifd_recv_thread_start(uint8_t id) "%d" multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_send(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" multifd_recv(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" +multifd_send_sync_main(uint32_t seq) "seq %d" +multifd_send_sync_main_signal(uint8_t id) "channel %d" +multifd_send_sync_main_wait(uint8_t id) "channel %d" +multifd_recv_sync_main(uint32_t seq) "seq %d" +multifd_recv_sync_main_signal(uint8_t id) "channel %d" +multifd_recv_sync_main_wait(uint8_t id) "channel %d" # migration/migration.c await_return_path_close_on_source_close(void) "" From patchwork Wed Apr 25 11:27:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904169 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJLF4dRhz9rxs for ; Wed, 25 Apr 2018 21:45:56 +1000 (AEST) Received: from localhost ([::1]:35942 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIrw-0000wx-U3 for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:45:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34355) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaW-00032p-0J for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaT-0005GJ-5S for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:52 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51558 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaT-0005Fw-1Y for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:49 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id ACF028D76C for ; Wed, 25 Apr 2018 11:27:48 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id A3C6C215CDC8; Wed, 25 Apr 2018 11:27:47 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:19 +0200 Message-Id: <20180425112723.1111-18-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 25 Apr 2018 11:27:48 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 25 Apr 2018 11:27:48 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 17/21] migration: Create ram_multifd_page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The function still don't use multifd, but we have simplified ram_save_page, xbzrle and RDMA stuff is gone. We have added a new counter. Signed-off-by: Juan Quintela --- Add last_page parameter Add commets for done and address Remove multifd field, it is the same than normal pages Merge next patch, now we send multiple pages at a time Remove counter for multifd pages, it is identical to normal pages Use iovec's instead of creating the equivalent. Clear memory used by pages (dave) Use g_new0(danp) define MULTIFD_CONTINUE now pages member is a pointer Fix off-by-one in number of pages in one packet Remove RAM_SAVE_FLAG_MULTIFD_PAGE s/multifd_pages_t/MultiFDPages_t/ --- migration/ram.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 398cb0af3b..862ec53d32 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -54,6 +54,7 @@ #include "migration/block.h" #include "sysemu/sysemu.h" #include "qemu/uuid.h" +#include "qemu/iov.h" /***********************************************************/ /* ram save/restore */ @@ -692,8 +693,65 @@ struct { QemuSemaphore sem_sync; /* global number of generated multifd packets */ uint32_t seq; + /* send channels ready */ + QemuSemaphore channels_ready; } *multifd_send_state; +static void multifd_send_pages(void) +{ + int i; + static int next_channel; + MultiFDSendParams *p = NULL; /* make happy gcc */ + MultiFDPages_t *pages = multifd_send_state->pages; + + qemu_sem_wait(&multifd_send_state->channels_ready); + for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) { + p = &multifd_send_state->params[i]; + + qemu_mutex_lock(&p->mutex); + if (!p->pending_job) { + p->pending_job++; + next_channel = (i + 1) % migrate_multifd_channels(); + break; + } + qemu_mutex_unlock(&p->mutex); + } + p->pages->used = 0; + multifd_send_state->seq++; + p->seq = multifd_send_state->seq; + p->pages->block = NULL; + multifd_send_state->pages = p->pages; + p->pages = pages; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); +} + +static void multifd_queue_page(RAMBlock *block, ram_addr_t offset) +{ + MultiFDPages_t *pages = multifd_send_state->pages; + + if (!pages->block) { + pages->block = block; + } + + if (pages->block == block) { + pages->offset[pages->used] = offset; + pages->iov[pages->used].iov_base = block->host + offset; + pages->iov[pages->used].iov_len = TARGET_PAGE_SIZE; + pages->used++; + + if (pages->used < pages->allocated) { + return; + } + } + + multifd_send_pages(); + + if (pages->block != block) { + multifd_queue_page(block, offset); + } +} + static void multifd_send_terminate_threads(Error *err) { int i; @@ -746,6 +804,7 @@ int multifd_save_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_send_state->channels_ready); qemu_sem_destroy(&multifd_send_state->sem_sync); g_free(multifd_send_state->params); multifd_send_state->params = NULL; @@ -763,12 +822,17 @@ static void multifd_send_sync_main(void) if (!migrate_use_multifd()) { return; } + if (multifd_send_state->pages->used) { + multifd_send_pages(); + } for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; trace_multifd_send_sync_main_signal(p->id); qemu_mutex_lock(&p->mutex); + multifd_send_state->seq++; + p->seq = multifd_send_state->seq; p->flags |= MULTIFD_FLAG_SYNC; p->pending_job++; qemu_mutex_unlock(&p->mutex); @@ -824,6 +888,7 @@ static void *multifd_send_thread(void *opaque) if (flags & MULTIFD_FLAG_SYNC) { qemu_sem_post(&multifd_send_state->sem_sync); } + qemu_sem_post(&multifd_send_state->channels_ready); } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -883,6 +948,7 @@ int multifd_save_setup(void) atomic_set(&multifd_send_state->count, 0); multifd_pages_init(&multifd_send_state->pages, page_count); qemu_sem_init(&multifd_send_state->sem_sync, 0); + qemu_sem_init(&multifd_send_state->channels_ready, 0); for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -1576,6 +1642,31 @@ static int ram_save_page(RAMState *rs, PageSearchStatus *pss, bool last_stage) return pages; } +static int ram_multifd_page(RAMState *rs, PageSearchStatus *pss, + bool last_stage) +{ + int pages; + uint8_t *p; + RAMBlock *block = pss->block; + ram_addr_t offset = pss->page << TARGET_PAGE_BITS; + + p = block->host + offset; + + pages = save_zero_page(rs, block, offset); + if (pages == -1) { + ram_counters.transferred += + save_page_header(rs, rs->f, block, + offset | RAM_SAVE_FLAG_PAGE); + multifd_queue_page(block, offset); + qemu_put_buffer(rs->f, p, TARGET_PAGE_SIZE); + ram_counters.transferred += TARGET_PAGE_SIZE; + pages = 1; + ram_counters.normal++; + } + + return pages; +} + static int do_compress_ram_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset) { @@ -2004,6 +2095,8 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss, if (migrate_use_compression() && (rs->ram_bulk_stage || !migrate_use_xbzrle())) { res = ram_save_compressed_page(rs, pss, last_stage); + } else if (migrate_use_multifd()) { + res = ram_multifd_page(rs, pss, last_stage); } else { res = ram_save_page(rs, pss, last_stage); } From patchwork Wed Apr 25 11:27:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904167 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJHV4qMsz9s1P for ; Wed, 25 Apr 2018 21:43:34 +1000 (AEST) Received: from localhost ([::1]:35926 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIpg-00075s-Lp for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:43:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34354) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaW-00032o-02 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaU-0005IH-GG for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:52 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:51560 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaU-0005HX-CR for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:50 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 01BD78DC4D for ; Wed, 25 Apr 2018 11:27:50 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF5AF2166BC6; Wed, 25 Apr 2018 11:27:48 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:20 +0200 Message-Id: <20180425112723.1111-19-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 25 Apr 2018 11:27:50 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 25 Apr 2018 11:27:50 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 18/21] migration: Start sending messages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela --- migration/ram.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 862ec53d32..9adbaa81f9 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -625,9 +625,6 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) RAMBlock *block; int i; - /* ToDo: We can't use it until we haven't received a message */ - return 0; - be32_to_cpus(&packet->magic); if (packet->magic != MULTIFD_MAGIC) { error_setg(errp, "multifd: received packet " @@ -851,6 +848,7 @@ static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p = opaque; Error *local_err = NULL; + int ret; trace_multifd_send_thread_start(p->id); @@ -878,10 +876,18 @@ static void *multifd_send_thread(void *opaque) trace_multifd_send(p->id, seq, used, flags); - /* ToDo: send packet here */ + ret = qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret != 0) { + break; + } + + ret = qio_channel_writev_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { + break; + } qemu_mutex_lock(&p->mutex); - p->flags = 0; p->pending_job--; qemu_mutex_unlock(&p->mutex); @@ -1091,7 +1097,14 @@ static void *multifd_recv_thread(void *opaque) uint32_t flags; qemu_mutex_unlock(&p->mutex); - /* ToDo: recv packet here */ + ret = qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret == 0) { /* EOF */ + break; + } + if (ret == -1) { /* Error */ + break; + } qemu_mutex_lock(&p->mutex); ret = multifd_recv_unfill_packet(p, &local_err); @@ -1108,6 +1121,11 @@ static void *multifd_recv_thread(void *opaque) p->num_pages += used; qemu_mutex_unlock(&p->mutex); + ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { + break; + } + if (flags & MULTIFD_FLAG_SYNC) { qemu_sem_post(&multifd_recv_state->sem_sync); qemu_sem_wait(&p->sem_sync); From patchwork Wed Apr 25 11:27:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904159 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJCG0Cdrz9rxs for ; Wed, 25 Apr 2018 21:39:52 +1000 (AEST) Received: from localhost ([::1]:35902 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIm5-0003ns-Kx for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:39:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34376) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaW-00033i-V5 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaV-0005K8-Rc for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:52 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:33588 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaV-0005JZ-M2 for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:51 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4E886EB6E5 for ; Wed, 25 Apr 2018 11:27:51 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 44FAD215CDD1; Wed, 25 Apr 2018 11:27:50 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:21 +0200 Message-Id: <20180425112723.1111-20-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:51 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Wed, 25 Apr 2018 11:27:51 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 19/21] migration: Wait for blocking IO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We have three conditions here: - channel fails -> error - we have to quit: we close the channel and reads fails - normal read that success, we are in bussiness So forget the complications of waiting in a semaphore. Signed-off-by: Juan Quintela --- migration/ram.c | 81 ++++++++++++++++++------------------------------- 1 file changed, 29 insertions(+), 52 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 9adbaa81f9..2734f91ded 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -496,8 +496,6 @@ typedef struct { bool running; /* should this thread finish */ bool quit; - /* thread has work to do */ - bool pending_job; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -1056,14 +1054,6 @@ static void multifd_recv_sync_main(void) for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; - trace_multifd_recv_sync_main_signal(p->id); - qemu_mutex_lock(&p->mutex); - p->pending_job = true; - qemu_mutex_unlock(&p->mutex); - } - for (i = 0; i < migrate_multifd_channels(); i++) { - MultiFDRecvParams *p = &multifd_recv_state->params[i]; - trace_multifd_recv_sync_main_wait(p->id); qemu_sem_wait(&multifd_recv_state->sem_sync); qemu_mutex_lock(&p->mutex); @@ -1076,7 +1066,6 @@ static void multifd_recv_sync_main(void) MultiFDRecvParams *p = &multifd_recv_state->params[i]; trace_multifd_recv_sync_main_signal(p->id); - qemu_sem_post(&p->sem_sync); } trace_multifd_recv_sync_main(multifd_recv_state->seq); @@ -1091,51 +1080,40 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { + uint32_t used; + uint32_t flags; + + ret = qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret == 0) { /* EOF */ + break; + } + if (ret == -1) { /* Error */ + break; + } + qemu_mutex_lock(&p->mutex); - if (true || p->pending_job) { - uint32_t used; - uint32_t flags; - qemu_mutex_unlock(&p->mutex); - - ret = qio_channel_read_all_eof(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret == 0) { /* EOF */ - break; - } - if (ret == -1) { /* Error */ - break; - } - - qemu_mutex_lock(&p->mutex); - ret = multifd_recv_unfill_packet(p, &local_err); - if (ret) { - qemu_mutex_unlock(&p->mutex); - break; - } - - used = p->pages->used; - flags = p->flags; - trace_multifd_recv(p->id, p->seq, used, flags); - p->pending_job = false; - p->num_packets++; - p->num_pages += used; + ret = multifd_recv_unfill_packet(p, &local_err); + if (ret) { qemu_mutex_unlock(&p->mutex); + break; + } - ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); - if (ret != 0) { - break; - } + used = p->pages->used; + flags = p->flags; + trace_multifd_recv(p->id, p->seq, used, flags); + p->num_packets++; + p->num_pages += used; + qemu_mutex_unlock(&p->mutex); - if (flags & MULTIFD_FLAG_SYNC) { - qemu_sem_post(&multifd_recv_state->sem_sync); - qemu_sem_wait(&p->sem_sync); - } - } else if (p->quit) { - qemu_mutex_unlock(&p->mutex); + ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { break; - } else { - qemu_mutex_unlock(&p->mutex); - /* sometimes there are spurious wakeups */ + } + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_recv_state->sem_sync); + qemu_sem_wait(&p->sem_sync); } } @@ -1173,7 +1151,6 @@ int multifd_load_setup(void) qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); p->quit = false; - p->pending_job = false; p->id = i; multifd_pages_init(&p->pages, page_count); p->packet_len = sizeof(MultiFDPacket_t) From patchwork Wed Apr 25 11:27:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904163 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJGW4Gk2z9rxs for ; Wed, 25 Apr 2018 21:42:43 +1000 (AEST) Received: from localhost ([::1]:35920 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIoq-0006Hb-D8 for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:42:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34396) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaY-00034m-2I for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaX-0005MQ-4l for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:54 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:45572 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaW-0005Lh-WC for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:53 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 98B144022414 for ; Wed, 25 Apr 2018 11:27:52 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 91D37215CDD1; Wed, 25 Apr 2018 11:27:51 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:22 +0200 Message-Id: <20180425112723.1111-21-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:52 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 25 Apr 2018 11:27:52 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 20/21] migration: Remove not needed semaphore and quit X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We know quit closing the QIO. Signed-off-by: Juan Quintela --- migration/ram.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 2734f91ded..23203756b7 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -488,14 +488,10 @@ typedef struct { QemuThread thread; /* communication channel */ QIOChannel *c; - /* sem where to wait for more work */ - QemuSemaphore sem; /* this mutex protects the following parameters */ QemuMutex mutex; /* is this channel thread running */ bool running; - /* should this thread finish */ - bool quit; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -1001,8 +997,8 @@ static void multifd_recv_terminate_threads(Error *err) MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_lock(&p->mutex); - p->quit = true; - qemu_sem_post(&p->sem); + object_unref(OBJECT(p->c)); + p->c = NULL; qemu_mutex_unlock(&p->mutex); } } @@ -1025,7 +1021,6 @@ int multifd_load_cleanup(Error **errp) object_unref(OBJECT(p->c)); p->c = NULL; qemu_mutex_destroy(&p->mutex); - qemu_sem_destroy(&p->sem); qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; @@ -1148,9 +1143,7 @@ int multifd_load_setup(void) MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_init(&p->mutex); - qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); - p->quit = false; p->id = i; multifd_pages_init(&p->pages, page_count); p->packet_len = sizeof(MultiFDPacket_t) From patchwork Wed Apr 25 11:27:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 904170 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WJLL2qVVz9rxs for ; Wed, 25 Apr 2018 21:46:02 +1000 (AEST) Received: from localhost ([::1]:35945 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIs4-00014d-El for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 07:46:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34414) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBIaZ-00036C-8j for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBIaY-0005OR-DZ for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:55 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:59520 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fBIaY-0005Np-8y for qemu-devel@nongnu.org; Wed, 25 Apr 2018 07:27:54 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E20E7406E973 for ; Wed, 25 Apr 2018 11:27:53 +0000 (UTC) Received: from secure.mitica (ovpn-117-169.ams2.redhat.com [10.36.117.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id D9FFC215CDCF; Wed, 25 Apr 2018 11:27:52 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 25 Apr 2018 13:27:23 +0200 Message-Id: <20180425112723.1111-22-quintela@redhat.com> In-Reply-To: <20180425112723.1111-1-quintela@redhat.com> References: <20180425112723.1111-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 25 Apr 2018 11:27:53 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 25 Apr 2018 11:27:53 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v12 21/21] migration: Stop sending whole pages through main channel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We have to flush() the QEMUFile because now we sent really few data through that channel. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 23203756b7..f5cff2eb59 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1634,20 +1634,12 @@ static int ram_multifd_page(RAMState *rs, PageSearchStatus *pss, bool last_stage) { int pages; - uint8_t *p; RAMBlock *block = pss->block; ram_addr_t offset = pss->page << TARGET_PAGE_BITS; - p = block->host + offset; - pages = save_zero_page(rs, block, offset); if (pages == -1) { - ram_counters.transferred += - save_page_header(rs, rs->f, block, - offset | RAM_SAVE_FLAG_PAGE); multifd_queue_page(block, offset); - qemu_put_buffer(rs->f, p, TARGET_PAGE_SIZE); - ram_counters.transferred += TARGET_PAGE_SIZE; pages = 1; ram_counters.normal++; } @@ -2869,6 +2861,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); return 0; } @@ -2946,6 +2939,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) multifd_send_sync_main(); out: qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); ram_counters.transferred += 8; ret = qemu_file_get_error(f); @@ -2999,6 +2993,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); return 0; }