From patchwork Wed May 23 11:18:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918978 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVYK0qGtz9s1b for ; Wed, 23 May 2018 21:25:09 +1000 (AEST) Received: from localhost ([::1]:60702 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRtC-0002UW-Lc for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:25:06 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50806) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmj-0006Bm-Fw for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:26 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmh-0007Av-5V for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:25 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44580 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmh-0007Aq-0P for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:23 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id A827A7D83A for ; Wed, 23 May 2018 11:18:22 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 99E8B2166BB2; Wed, 23 May 2018 11:18:21 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:06 +0200 Message-Id: <20180523111817.1463-2-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:22 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:22 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 01/12] migration: Create multipage support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We only create/destry the page list here. We will use it later. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 5bcbf7a9f9..23cc5625eb 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -510,6 +510,20 @@ typedef struct { uint8_t id; } __attribute__((packed)) MultiFDInit_t; +typedef struct { + /* number of used pages */ + uint32_t used; + /* number of allocated pages */ + uint32_t allocated; + /* global number of generated multifd packets */ + uint32_t seq; + /* offset of each page */ + ram_addr_t *offset; + /* pointer to each page */ + struct iovec *iov; + RAMBlock *block; +} MultiFDPages_t; + typedef struct { /* this fields are not changed once the thread is created */ /* channel number */ @@ -528,6 +542,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* array of pages to sent */ + MultiFDPages_t *pages; } MultiFDSendParams; typedef struct { @@ -548,6 +564,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* array of pages to receive */ + MultiFDPages_t *pages; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -612,10 +630,36 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp) return msg.id; } +static MultiFDPages_t *multifd_pages_init(size_t size) +{ + MultiFDPages_t *pages = g_new0(MultiFDPages_t, 1); + + pages->allocated = size; + pages->iov = g_new0(struct iovec, size); + pages->offset = g_new0(ram_addr_t, size); + + return pages; +} + +static void multifd_pages_clear(MultiFDPages_t *pages) +{ + pages->used = 0; + pages->allocated = 0; + pages->seq = 0; + pages->block = NULL; + g_free(pages->iov); + pages->iov = NULL; + g_free(pages->offset); + pages->offset = NULL; + g_free(pages); +} + struct { MultiFDSendParams *params; /* number of created threads */ int count; + /* array of pages to sent */ + MultiFDPages_t *pages; } *multifd_send_state; static void multifd_send_terminate_threads(Error *err) @@ -665,9 +709,13 @@ int multifd_save_cleanup(Error **errp) qemu_sem_destroy(&p->sem); g_free(p->name); p->name = NULL; + multifd_pages_clear(p->pages); + p->pages = NULL; } g_free(multifd_send_state->params); multifd_send_state->params = NULL; + multifd_pages_clear(multifd_send_state->pages); + multifd_send_state->pages = NULL; g_free(multifd_send_state); multifd_send_state = NULL; return ret; @@ -728,6 +776,7 @@ static void multifd_new_send_channel_async(QIOTask *task, gpointer opaque) int multifd_save_setup(void) { int thread_count; + uint32_t page_count = migrate_multifd_page_count(); uint8_t i; if (!migrate_use_multifd()) { @@ -737,6 +786,8 @@ int multifd_save_setup(void) multifd_send_state = g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); atomic_set(&multifd_send_state->count, 0); + multifd_send_state->pages = multifd_pages_init(page_count); + for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -744,6 +795,7 @@ int multifd_save_setup(void) qemu_sem_init(&p->sem, 0); p->quit = false; p->id = i; + p->pages = multifd_pages_init(page_count); p->name = g_strdup_printf("multifdsend_%d", i); socket_send_channel_create(multifd_new_send_channel_async, p); } @@ -801,6 +853,8 @@ int multifd_load_cleanup(Error **errp) qemu_sem_destroy(&p->sem); g_free(p->name); p->name = NULL; + multifd_pages_clear(p->pages); + p->pages = NULL; } g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; @@ -834,6 +888,7 @@ static void *multifd_recv_thread(void *opaque) int multifd_load_setup(void) { int thread_count; + uint32_t page_count = migrate_multifd_page_count(); uint8_t i; if (!migrate_use_multifd()) { @@ -843,6 +898,7 @@ int multifd_load_setup(void) multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); atomic_set(&multifd_recv_state->count, 0); + for (i = 0; i < thread_count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; @@ -850,6 +906,7 @@ int multifd_load_setup(void) qemu_sem_init(&p->sem, 0); p->quit = false; p->id = i; + p->pages = multifd_pages_init(page_count); p->name = g_strdup_printf("multifdrecv_%d", i); } return 0; From patchwork Wed May 23 11:18:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918975 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVRx3xBGz9s1b for ; Wed, 23 May 2018 21:20:28 +1000 (AEST) Received: from localhost ([::1]:60677 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRof-00079f-3J for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:20:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50823) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmk-0006DH-6k for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmi-0007BU-Qe for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:26 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44600 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmi-0007B9-Kf for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:24 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 501F4857A3 for ; Wed, 23 May 2018 11:18:24 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id EB18E2157F44; Wed, 23 May 2018 11:18:22 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:07 +0200 Message-Id: <20180523111817.1463-3-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:24 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:24 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 02/12] migration: Create multifd packet X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We still don't put anything there. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- fix magic (dave) check offset/ramblock (dave) --- migration/ram.c | 145 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 144 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index 23cc5625eb..54350db8b0 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -510,6 +510,17 @@ typedef struct { uint8_t id; } __attribute__((packed)) MultiFDInit_t; +typedef struct { + uint32_t magic; + uint32_t version; + uint32_t flags; + uint32_t size; + uint32_t used; + uint32_t seq; + char ramblock[256]; + uint64_t offset[]; +} __attribute__((packed)) MultiFDPacket_t; + typedef struct { /* number of used pages */ uint32_t used; @@ -544,6 +555,14 @@ typedef struct { bool quit; /* array of pages to sent */ MultiFDPages_t *pages; + /* packet allocated len */ + uint32_t packet_len; + /* pointer to the packet */ + MultiFDPacket_t *packet; + /* multifd flags for each packet */ + uint32_t flags; + /* global number of generated multifd packets */ + uint32_t seq; } MultiFDSendParams; typedef struct { @@ -566,6 +585,14 @@ typedef struct { bool quit; /* array of pages to receive */ MultiFDPages_t *pages; + /* packet allocated len */ + uint32_t packet_len; + /* pointer to the packet */ + MultiFDPacket_t *packet; + /* multifd flags for each packet */ + uint32_t flags; + /* global number of generated multifd packets */ + uint32_t seq; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -654,6 +681,99 @@ static void multifd_pages_clear(MultiFDPages_t *pages) g_free(pages); } +static void multifd_send_fill_packet(MultiFDSendParams *p) +{ + MultiFDPacket_t *packet = p->packet; + int i; + + packet->magic = cpu_to_be32(MULTIFD_MAGIC); + packet->version = cpu_to_be32(MULTIFD_VERSION); + packet->flags = cpu_to_be32(p->flags); + packet->size = cpu_to_be32(migrate_multifd_page_count()); + packet->used = cpu_to_be32(p->pages->used); + packet->seq = cpu_to_be32(p->seq); + + if (p->pages->block) { + strncpy(packet->ramblock, p->pages->block->idstr, 256); + } + + for (i = 0; i < p->pages->used; i++) { + packet->offset[i] = cpu_to_be64(p->pages->offset[i]); + } +} + +static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +{ + MultiFDPacket_t *packet = p->packet; + RAMBlock *block; + int i; + + /* ToDo: We can't use it until we haven't received a message */ + return 0; + + be32_to_cpus(&packet->magic); + if (packet->magic != MULTIFD_MAGIC) { + error_setg(errp, "multifd: received packet " + "magic %x and expected magic %x", + packet->magic, MULTIFD_MAGIC); + return -1; + } + + be32_to_cpus(&packet->version); + if (packet->version != MULTIFD_VERSION) { + error_setg(errp, "multifd: received packet " + "version %d and expected version %d", + packet->version, MULTIFD_VERSION); + return -1; + } + + p->flags = be32_to_cpu(packet->flags); + + be32_to_cpus(&packet->size); + if (packet->size > migrate_multifd_page_count()) { + error_setg(errp, "multifd: received packet " + "with size %d and expected maximum size %d", + packet->size, migrate_multifd_page_count()) ; + return -1; + } + + p->pages->used = be32_to_cpu(packet->used); + if (p->pages->used > packet->size) { + error_setg(errp, "multifd: received packet " + "with size %d and expected maximum size %d", + p->pages->used, packet->size) ; + return -1; + } + + p->seq = be32_to_cpu(packet->seq); + + if (p->pages->used) { + /* make sure that ramblock is 0 terminated */ + packet->ramblock[255] = 0; + block = qemu_ram_block_by_name(packet->ramblock); + if (!block) { + error_setg(errp, "multifd: unknown ram block %s", + packet->ramblock); + return -1; + } + } + + for (i = 0; i < p->pages->used; i++) { + ram_addr_t offset = be64_to_cpu(packet->offset[i]); + + if (offset > (block->used_length - TARGET_PAGE_SIZE)) { + error_setg(errp, "multifd: offset too long %" PRId64 + " (max %" PRId64 ")", + offset, block->max_length); + return -1; + } + p->pages->iov[i].iov_base = block->host + offset; + p->pages->iov[i].iov_len = TARGET_PAGE_SIZE; + } + + return 0; +} + struct { MultiFDSendParams *params; /* number of created threads */ @@ -711,6 +831,9 @@ int multifd_save_cleanup(Error **errp) p->name = NULL; multifd_pages_clear(p->pages); p->pages = NULL; + p->packet_len = 0; + g_free(p->packet); + p->packet = NULL; } g_free(multifd_send_state->params); multifd_send_state->params = NULL; @@ -732,6 +855,7 @@ static void *multifd_send_thread(void *opaque) while (true) { qemu_mutex_lock(&p->mutex); + multifd_send_fill_packet(p); if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -796,6 +920,9 @@ int multifd_save_setup(void) p->quit = false; p->id = i; p->pages = multifd_pages_init(page_count); + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(ram_addr_t) * page_count; + p->packet = g_malloc0(p->packet_len); p->name = g_strdup_printf("multifdsend_%d", i); socket_send_channel_create(multifd_new_send_channel_async, p); } @@ -855,6 +982,9 @@ int multifd_load_cleanup(Error **errp) p->name = NULL; multifd_pages_clear(p->pages); p->pages = NULL; + p->packet_len = 0; + g_free(p->packet); + p->packet = NULL; } g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; @@ -867,10 +997,20 @@ int multifd_load_cleanup(Error **errp) static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; + Error *local_err = NULL; + int ret; while (true) { qemu_mutex_lock(&p->mutex); - if (p->quit) { + if (false) { + /* ToDo: Packet reception goes here */ + + ret = multifd_recv_unfill_packet(p, &local_err); + qemu_mutex_unlock(&p->mutex); + if (ret) { + break; + } + } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } @@ -907,6 +1047,9 @@ int multifd_load_setup(void) p->quit = false; p->id = i; p->pages = multifd_pages_init(page_count); + p->packet_len = sizeof(MultiFDPacket_t) + + sizeof(ram_addr_t) * page_count; + p->packet = g_malloc0(p->packet_len); p->name = g_strdup_printf("multifdrecv_%d", i); } return 0; From patchwork Wed May 23 11:18:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918982 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVcc5bWsz9s1b for ; Wed, 23 May 2018 21:28:00 +1000 (AEST) Received: from localhost ([::1]:60722 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRvy-0004yL-DQ for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:27:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50836) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRml-0006E8-43 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmk-0007Bq-4d for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:27 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44602 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmj-0007Bk-Vk for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:26 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9CBF4195478 for ; Wed, 23 May 2018 11:18:25 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 93A562144B21; Wed, 23 May 2018 11:18:24 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:08 +0200 Message-Id: <20180523111817.1463-4-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:25 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:25 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 03/12] migration: Add multifd traces for start/end thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We want to know how many pages/packets each channel has sent. Add counters for those. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- sort trace-events (dave) --- migration/ram.c | 22 ++++++++++++++++++++++ migration/trace-events | 4 ++++ 2 files changed, 26 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 54350db8b0..ab715cc148 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -563,6 +563,11 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint32_t seq; + /* thread local variables */ + /* packets sent through this channel */ + uint32_t num_packets; + /* pages sent through this channel */ + uint32_t num_pages; } MultiFDSendParams; typedef struct { @@ -593,6 +598,11 @@ typedef struct { uint32_t flags; /* global number of generated multifd packets */ uint32_t seq; + /* thread local variables */ + /* packets sent through this channel */ + uint32_t num_packets; + /* pages sent through this channel */ + uint32_t num_pages; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -849,9 +859,13 @@ static void *multifd_send_thread(void *opaque) MultiFDSendParams *p = opaque; Error *local_err = NULL; + trace_multifd_send_thread_start(p->id); + if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; } + /* initial packet */ + p->num_packets = 1; while (true) { qemu_mutex_lock(&p->mutex); @@ -873,6 +887,8 @@ out: p->running = false; qemu_mutex_unlock(&p->mutex); + trace_multifd_send_thread_end(p->id, p->num_packets, p->num_pages); + return NULL; } @@ -1000,6 +1016,8 @@ static void *multifd_recv_thread(void *opaque) Error *local_err = NULL; int ret; + trace_multifd_recv_thread_start(p->id); + while (true) { qemu_mutex_lock(&p->mutex); if (false) { @@ -1022,6 +1040,8 @@ static void *multifd_recv_thread(void *opaque) p->running = false; qemu_mutex_unlock(&p->mutex); + trace_multifd_recv_thread_end(p->id, p->num_packets, p->num_pages); + return NULL; } @@ -1087,6 +1107,8 @@ void multifd_recv_new_channel(QIOChannel *ioc) } p->c = ioc; object_ref(OBJECT(ioc)); + /* initial packet */ + p->num_packets = 1; p->running = true; qemu_thread_create(&p->thread, p->name, multifd_recv_thread, p, diff --git a/migration/trace-events b/migration/trace-events index 3c798ddd11..3c7a9a8332 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -76,6 +76,10 @@ get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_throttle(void) "" +multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" +multifd_recv_thread_start(uint8_t id) "%d" +multifd_send_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" +multifd_send_thread_start(uint8_t id) "%d" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" From patchwork Wed May 23 11:18:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918979 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVZB3Byyz9s2L for ; Wed, 23 May 2018 21:25:54 +1000 (AEST) Received: from localhost ([::1]:60706 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRtw-0003A2-1z for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:25:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50851) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmm-0006F6-7s for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRml-0007CS-D3 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:28 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44606 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRml-0007CK-99 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:27 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E7633195478 for ; Wed, 23 May 2018 11:18:26 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id DECBF2144B21; Wed, 23 May 2018 11:18:25 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:09 +0200 Message-Id: <20180523111817.1463-5-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:26 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:26 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 04/12] migration: Calculate transferred ram correctly X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" On multifd we send data from more places that main channel. Signed-off-by: Juan Quintela --- migration/migration.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 05aec2c905..55d7b64e24 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2679,12 +2679,19 @@ static void migration_update_counters(MigrationState *s, { uint64_t transferred, time_spent; double bandwidth; + uint64_t now; if (current_time < s->iteration_start_time + BUFFER_DELAY) { return; } - transferred = qemu_ftell(s->to_dst_file) - s->iteration_initial_bytes; + if (migrate_use_multifd()) { + now = ram_counters.normal * qemu_target_page_size() + + qemu_ftell(s->to_dst_file); + } else { + now = qemu_ftell(s->to_dst_file); + } + transferred = now - s->iteration_initial_bytes; time_spent = current_time - s->iteration_start_time; bandwidth = (double)transferred / time_spent; s->threshold_size = bandwidth * s->parameters.downtime_limit; @@ -2704,7 +2711,7 @@ static void migration_update_counters(MigrationState *s, qemu_file_reset_rate_limit(s->to_dst_file); s->iteration_start_time = current_time; - s->iteration_initial_bytes = qemu_ftell(s->to_dst_file); + s->iteration_initial_bytes = now; trace_migrate_transferred(transferred, time_spent, bandwidth, s->threshold_size); From patchwork Wed May 23 11:18:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918980 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVZl28jpz9s1b for ; Wed, 23 May 2018 21:26:23 +1000 (AEST) Received: from localhost ([::1]:60713 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRuO-0003hw-S9 for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:26:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50863) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmn-0006GT-Hb for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmm-0007Cj-NW for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:29 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:60020 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmm-0007Cf-JQ for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:28 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3EA7F4023826 for ; Wed, 23 May 2018 11:18:28 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 362A12144B21; Wed, 23 May 2018 11:18:27 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:10 +0200 Message-Id: <20180523111817.1463-6-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 23 May 2018 11:18:28 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 23 May 2018 11:18:28 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 05/12] migration: Multifd channels always wait on the sem X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Either for quit, sync or packet, we first wake them. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index ab715cc148..90f62144e2 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -868,6 +868,7 @@ static void *multifd_send_thread(void *opaque) p->num_packets = 1; while (true) { + qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); multifd_send_fill_packet(p); if (p->quit) { @@ -875,7 +876,9 @@ static void *multifd_send_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + /* this is impossible */ + error_setg(&local_err, "multifd_send_thread: Unknown command"); + break; } out: @@ -1019,6 +1022,7 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { + qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); if (false) { /* ToDo: Packet reception goes here */ @@ -1033,9 +1037,14 @@ static void *multifd_recv_thread(void *opaque) break; } qemu_mutex_unlock(&p->mutex); - qemu_sem_wait(&p->sem); + /* this is impossible */ + error_setg(&local_err, "multifd_recv_thread: Unknown command"); + break; } + if (local_err) { + multifd_recv_terminate_threads(local_err); + } qemu_mutex_lock(&p->mutex); p->running = false; qemu_mutex_unlock(&p->mutex); From patchwork Wed May 23 11:18:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918983 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVdT3jvtz9s2L for ; Wed, 23 May 2018 21:28:45 +1000 (AEST) Received: from localhost ([::1]:60724 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRwh-0005Z2-4e for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:28:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50890) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmq-0006JR-Nv for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmo-0007DC-1H for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:32 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:60958 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmn-0007D8-Sb for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:29 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8BDD14071181 for ; Wed, 23 May 2018 11:18:29 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 815C42144B21; Wed, 23 May 2018 11:18:28 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:11 +0200 Message-Id: <20180523111817.1463-7-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 23 May 2018 11:18:29 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 23 May 2018 11:18:29 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 06/12] migration: Add block where to send/receive packets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Once there add tracepoints. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 49 +++++++++++++++++++++++++++++++++++++----- migration/trace-events | 2 ++ 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 90f62144e2..c9a9bd79f3 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -553,6 +553,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* thread has work to do */ + int pending_job; /* array of pages to sent */ MultiFDPages_t *pages; /* packet allocated len */ @@ -588,6 +590,8 @@ typedef struct { bool running; /* should this thread finish */ bool quit; + /* thread has work to do */ + bool pending_job; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -870,8 +874,28 @@ static void *multifd_send_thread(void *opaque) while (true) { qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - multifd_send_fill_packet(p); - if (p->quit) { + + if (p->pending_job) { + uint32_t used = p->pages->used; + uint32_t seq = p->seq; + uint32_t flags = p->flags; + + multifd_send_fill_packet(p); + p->flags = 0; + p->num_packets++; + p->num_pages += used; + p->pages->used = 0; + qemu_mutex_unlock(&p->mutex); + + trace_multifd_send(p->id, seq, used, flags); + + /* ToDo: send packet here */ + + qemu_mutex_lock(&p->mutex); + p->pending_job--; + qemu_mutex_unlock(&p->mutex); + continue; + } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; } @@ -937,6 +961,7 @@ int multifd_save_setup(void) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); p->quit = false; + p->pending_job = 0; p->id = i; p->pages = multifd_pages_init(page_count); p->packet_len = sizeof(MultiFDPacket_t) @@ -1024,14 +1049,27 @@ static void *multifd_recv_thread(void *opaque) while (true) { qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - if (false) { - /* ToDo: Packet reception goes here */ + if (p->pending_job) { + uint32_t used; + uint32_t flags; + qemu_mutex_unlock(&p->mutex); + /* ToDo: recv packet here */ + + qemu_mutex_lock(&p->mutex); ret = multifd_recv_unfill_packet(p, &local_err); - qemu_mutex_unlock(&p->mutex); if (ret) { + qemu_mutex_unlock(&p->mutex); break; } + + used = p->pages->used; + flags = p->flags; + trace_multifd_recv(p->id, p->seq, used, flags); + p->pending_job = false; + p->num_packets++; + p->num_pages += used; + qemu_mutex_unlock(&p->mutex); } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -1074,6 +1112,7 @@ int multifd_load_setup(void) qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); p->quit = false; + p->pending_job = false; p->id = i; p->pages = multifd_pages_init(page_count); p->packet_len = sizeof(MultiFDPacket_t) diff --git a/migration/trace-events b/migration/trace-events index 3c7a9a8332..36e20b312d 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -76,8 +76,10 @@ get_queued_page_not_dirty(const char *block_name, uint64_t tmp_offset, unsigned migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_throttle(void) "" +multifd_recv(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_recv_thread_start(uint8_t id) "%d" +multifd_send(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" multifd_send_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_send_thread_start(uint8_t id) "%d" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" From patchwork Wed May 23 11:18:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918984 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVf83P0vz9s1b for ; Wed, 23 May 2018 21:29:20 +1000 (AEST) Received: from localhost ([::1]:60728 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRxG-00061B-4y for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:29:18 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50902) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmr-0006Jo-5i for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmp-0007Dg-GS for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:33 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44610 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmp-0007DO-B7 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:31 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0A1EE722FE for ; Wed, 23 May 2018 11:18:31 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id CEDA92166BD9; Wed, 23 May 2018 11:18:29 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:12 +0200 Message-Id: <20180523111817.1463-8-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:31 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:31 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 07/12] migration: Synchronize multifd threads with main thread X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We synchronize all threads each RAM_SAVE_FLAG_EOS. Bitmap synchronizations don't happen inside a ram section, so we are safe about two channels trying to overwrite the same memory. Signed-off-by: Juan Quintela --- migration/ram.c | 117 +++++++++++++++++++++++++++++++++++++---- migration/trace-events | 6 +++ 2 files changed, 112 insertions(+), 11 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index c9a9bd79f3..3e99d48123 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -503,6 +503,8 @@ exit: #define MULTIFD_MAGIC 0x11223344U #define MULTIFD_VERSION 1 +#define MULTIFD_FLAG_SYNC (1 << 0) + typedef struct { uint32_t magic; uint32_t version; @@ -570,6 +572,8 @@ typedef struct { uint32_t num_packets; /* pages sent through this channel */ uint32_t num_pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; } MultiFDSendParams; typedef struct { @@ -607,6 +611,8 @@ typedef struct { uint32_t num_packets; /* pages sent through this channel */ uint32_t num_pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; } MultiFDRecvParams; static int multifd_send_initial_packet(MultiFDSendParams *p, Error **errp) @@ -794,6 +800,10 @@ struct { int count; /* array of pages to sent */ MultiFDPages_t *pages; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; + /* global number of generated multifd packets */ + uint32_t seq; } *multifd_send_state; static void multifd_send_terminate_threads(Error *err) @@ -841,6 +851,7 @@ int multifd_save_cleanup(Error **errp) p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; multifd_pages_clear(p->pages); @@ -849,6 +860,7 @@ int multifd_save_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_send_state->sem_sync); g_free(multifd_send_state->params); multifd_send_state->params = NULL; multifd_pages_clear(multifd_send_state->pages); @@ -858,6 +870,33 @@ int multifd_save_cleanup(Error **errp) return ret; } +static void multifd_send_sync_main(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + + trace_multifd_send_sync_main_signal(p->id); + + qemu_mutex_lock(&p->mutex); + p->flags |= MULTIFD_FLAG_SYNC; + p->pending_job++; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + + trace_multifd_send_sync_main_wait(p->id); + qemu_sem_wait(&multifd_send_state->sem_sync); + } + trace_multifd_send_sync_main(multifd_send_state->seq); +} + static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p = opaque; @@ -894,15 +933,17 @@ static void *multifd_send_thread(void *opaque) qemu_mutex_lock(&p->mutex); p->pending_job--; qemu_mutex_unlock(&p->mutex); - continue; + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_send_state->sem_sync); + } } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; + } else { + qemu_mutex_unlock(&p->mutex); + /* sometimes there are spurious wakeups */ } - qemu_mutex_unlock(&p->mutex); - /* this is impossible */ - error_setg(&local_err, "multifd_send_thread: Unknown command"); - break; } out: @@ -954,12 +995,14 @@ int multifd_save_setup(void) multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); atomic_set(&multifd_send_state->count, 0); multifd_send_state->pages = multifd_pages_init(page_count); + qemu_sem_init(&multifd_send_state->sem_sync, 0); for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->sem_sync, 0); p->quit = false; p->pending_job = 0; p->id = i; @@ -977,6 +1020,10 @@ struct { MultiFDRecvParams *params; /* number of created threads */ int count; + /* syncs main thread and channels */ + QemuSemaphore sem_sync; + /* global number of generated multifd packets */ + uint32_t seq; } *multifd_recv_state; static void multifd_recv_terminate_threads(Error *err) @@ -1022,6 +1069,7 @@ int multifd_load_cleanup(Error **errp) p->c = NULL; qemu_mutex_destroy(&p->mutex); qemu_sem_destroy(&p->sem); + qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; multifd_pages_clear(p->pages); @@ -1030,6 +1078,7 @@ int multifd_load_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_recv_state->sem_sync); g_free(multifd_recv_state->params); multifd_recv_state->params = NULL; g_free(multifd_recv_state); @@ -1038,6 +1087,42 @@ int multifd_load_cleanup(Error **errp) return ret; } +static void multifd_recv_sync_main(void) +{ + int i; + + if (!migrate_use_multifd()) { + return; + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_signal(p->id); + qemu_mutex_lock(&p->mutex); + p->pending_job = true; + qemu_mutex_unlock(&p->mutex); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_wait(p->id); + qemu_sem_wait(&multifd_recv_state->sem_sync); + qemu_mutex_lock(&p->mutex); + if (multifd_recv_state->seq < p->seq) { + multifd_recv_state->seq = p->seq; + } + qemu_mutex_unlock(&p->mutex); + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + + trace_multifd_recv_sync_main_signal(p->id); + + qemu_sem_post(&p->sem_sync); + } + trace_multifd_recv_sync_main(multifd_recv_state->seq); +} + static void *multifd_recv_thread(void *opaque) { MultiFDRecvParams *p = opaque; @@ -1047,9 +1132,8 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { - qemu_sem_wait(&p->sem); qemu_mutex_lock(&p->mutex); - if (p->pending_job) { + if (true || p->pending_job) { uint32_t used; uint32_t flags; qemu_mutex_unlock(&p->mutex); @@ -1070,14 +1154,18 @@ static void *multifd_recv_thread(void *opaque) p->num_packets++; p->num_pages += used; qemu_mutex_unlock(&p->mutex); + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_recv_state->sem_sync); + qemu_sem_wait(&p->sem_sync); + } } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; + } else { + qemu_mutex_unlock(&p->mutex); + /* sometimes there are spurious wakeups */ } - qemu_mutex_unlock(&p->mutex); - /* this is impossible */ - error_setg(&local_err, "multifd_recv_thread: Unknown command"); - break; } if (local_err) { @@ -1105,12 +1193,14 @@ int multifd_load_setup(void) multifd_recv_state = g_malloc0(sizeof(*multifd_recv_state)); multifd_recv_state->params = g_new0(MultiFDRecvParams, thread_count); atomic_set(&multifd_recv_state->count, 0); + qemu_sem_init(&multifd_recv_state->sem_sync, 0); for (i = 0; i < thread_count; i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_init(&p->mutex); qemu_sem_init(&p->sem, 0); + qemu_sem_init(&p->sem_sync, 0); p->quit = false; p->pending_job = false; p->id = i; @@ -2847,6 +2937,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) ram_control_before_iterate(f, RAM_CONTROL_SETUP); ram_control_after_iterate(f, RAM_CONTROL_SETUP); + multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; @@ -2922,6 +3013,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) */ ram_control_after_iterate(f, RAM_CONTROL_ROUND); + multifd_send_sync_main(); out: qemu_put_be64(f, RAM_SAVE_FLAG_EOS); ram_counters.transferred += 8; @@ -2975,6 +3067,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) rcu_read_unlock(); + multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); return 0; @@ -3459,6 +3552,7 @@ static int ram_load_postcopy(QEMUFile *f) break; case RAM_SAVE_FLAG_EOS: /* normal exit */ + multifd_recv_sync_main(); break; default: error_report("Unknown combination of migration flags: %#x" @@ -3644,6 +3738,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) break; case RAM_SAVE_FLAG_EOS: /* normal exit */ + multifd_recv_sync_main(); break; default: if (flags & RAM_SAVE_FLAG_HOOK) { diff --git a/migration/trace-events b/migration/trace-events index 36e20b312d..b821041281 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -77,9 +77,15 @@ migration_bitmap_sync_start(void) "" migration_bitmap_sync_end(uint64_t dirty_pages) "dirty_pages %" PRIu64 migration_throttle(void) "" multifd_recv(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" +multifd_recv_sync_main(uint32_t seq) "seq %d" +multifd_recv_sync_main_signal(uint8_t id) "channel %d" +multifd_recv_sync_main_wait(uint8_t id) "channel %d" multifd_recv_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_recv_thread_start(uint8_t id) "%d" multifd_send(uint8_t id, uint32_t seq, uint32_t used, uint32_t flags) "channel %d seq number %d pages %d flags 0x%x" +multifd_send_sync_main(uint32_t seq) "seq %d" +multifd_send_sync_main_signal(uint8_t id) "channel %d" +multifd_send_sync_main_wait(uint8_t id) "channel %d" multifd_send_thread_end(uint8_t id, uint32_t packets, uint32_t pages) "channel %d packets %d pages %d" multifd_send_thread_start(uint8_t id) "%d" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" From patchwork Wed May 23 11:18:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918986 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVhY2484z9s1d for ; Wed, 23 May 2018 21:31:25 +1000 (AEST) Received: from localhost ([::1]:60743 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRzG-0007yR-VO for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:31:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50914) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmr-0006KS-Uj for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmq-0007EK-Py for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:33 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44612 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmq-0007Dx-LO for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:32 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 51D21195478 for ; Wed, 23 May 2018 11:18:32 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4AB752166BD9; Wed, 23 May 2018 11:18:31 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:13 +0200 Message-Id: <20180523111817.1463-9-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:32 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:32 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 08/12] migration: Create ram_save_multifd_page X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The function still don't use multifd, but we have simplified ram_save_page, xbzrle and RDMA stuff is gone. We have added a new counter. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- Add last_page parameter Add commets for done and address Remove multifd field, it is the same than normal pages Merge next patch, now we send multiple pages at a time Remove counter for multifd pages, it is identical to normal pages Use iovec's instead of creating the equivalent. Clear memory used by pages (dave) Use g_new0(danp) define MULTIFD_CONTINUE now pages member is a pointer Fix off-by-one in number of pages in one packet Remove RAM_SAVE_FLAG_MULTIFD_PAGE s/multifd_pages_t/MultiFDPages_t/ --- migration/ram.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 3e99d48123..28f5cea4d8 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -55,6 +55,7 @@ #include "sysemu/sysemu.h" #include "qemu/uuid.h" #include "savevm.h" +#include "qemu/iov.h" /***********************************************************/ /* ram save/restore */ @@ -804,8 +805,65 @@ struct { QemuSemaphore sem_sync; /* global number of generated multifd packets */ uint32_t seq; + /* send channels ready */ + QemuSemaphore channels_ready; } *multifd_send_state; +static void multifd_send_pages(void) +{ + int i; + static int next_channel; + MultiFDSendParams *p = NULL; /* make happy gcc */ + MultiFDPages_t *pages = multifd_send_state->pages; + + qemu_sem_wait(&multifd_send_state->channels_ready); + for (i = next_channel;; i = (i + 1) % migrate_multifd_channels()) { + p = &multifd_send_state->params[i]; + + qemu_mutex_lock(&p->mutex); + if (!p->pending_job) { + p->pending_job++; + next_channel = (i + 1) % migrate_multifd_channels(); + break; + } + qemu_mutex_unlock(&p->mutex); + } + p->pages->used = 0; + multifd_send_state->seq++; + p->seq = multifd_send_state->seq; + p->pages->block = NULL; + multifd_send_state->pages = p->pages; + p->pages = pages; + qemu_mutex_unlock(&p->mutex); + qemu_sem_post(&p->sem); +} + +static void multifd_queue_page(RAMBlock *block, ram_addr_t offset) +{ + MultiFDPages_t *pages = multifd_send_state->pages; + + if (!pages->block) { + pages->block = block; + } + + if (pages->block == block) { + pages->offset[pages->used] = offset; + pages->iov[pages->used].iov_base = block->host + offset; + pages->iov[pages->used].iov_len = TARGET_PAGE_SIZE; + pages->used++; + + if (pages->used < pages->allocated) { + return; + } + } + + multifd_send_pages(); + + if (pages->block != block) { + multifd_queue_page(block, offset); + } +} + static void multifd_send_terminate_threads(Error *err) { int i; @@ -860,6 +918,7 @@ int multifd_save_cleanup(Error **errp) g_free(p->packet); p->packet = NULL; } + qemu_sem_destroy(&multifd_send_state->channels_ready); qemu_sem_destroy(&multifd_send_state->sem_sync); g_free(multifd_send_state->params); multifd_send_state->params = NULL; @@ -877,12 +936,17 @@ static void multifd_send_sync_main(void) if (!migrate_use_multifd()) { return; } + if (multifd_send_state->pages->used) { + multifd_send_pages(); + } for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; trace_multifd_send_sync_main_signal(p->id); qemu_mutex_lock(&p->mutex); + multifd_send_state->seq++; + p->seq = multifd_send_state->seq; p->flags |= MULTIFD_FLAG_SYNC; p->pending_job++; qemu_mutex_unlock(&p->mutex); @@ -937,6 +1001,7 @@ static void *multifd_send_thread(void *opaque) if (flags & MULTIFD_FLAG_SYNC) { qemu_sem_post(&multifd_send_state->sem_sync); } + qemu_sem_post(&multifd_send_state->channels_ready); } else if (p->quit) { qemu_mutex_unlock(&p->mutex); break; @@ -996,6 +1061,7 @@ int multifd_save_setup(void) atomic_set(&multifd_send_state->count, 0); multifd_send_state->pages = multifd_pages_init(page_count); qemu_sem_init(&multifd_send_state->sem_sync, 0); + qemu_sem_init(&multifd_send_state->channels_ready, 0); for (i = 0; i < thread_count; i++) { MultiFDSendParams *p = &multifd_send_state->params[i]; @@ -1713,6 +1779,23 @@ static int ram_save_page(RAMState *rs, PageSearchStatus *pss, bool last_stage) return pages; } +static int ram_save_multifd_page(RAMState *rs, RAMBlock *block, + ram_addr_t offset) +{ + uint8_t *p; + + p = block->host + offset; + + ram_counters.transferred += save_page_header(rs, rs->f, block, + offset | RAM_SAVE_FLAG_PAGE); + multifd_queue_page(block, offset); + qemu_put_buffer(rs->f, p, TARGET_PAGE_SIZE); + ram_counters.transferred += TARGET_PAGE_SIZE; + ram_counters.normal++; + + return 1; +} + static int do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block, ram_addr_t offset, uint8_t *source_buf) { @@ -2116,6 +2199,8 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss, */ if (block == rs->last_sent_block && save_page_use_compression(rs)) { return compress_page_with_multi_thread(rs, block, offset); + } else if (migrate_use_multifd()) { + return ram_save_multifd_page(rs, block, offset); } return ram_save_page(rs, pss, last_stage); From patchwork Wed May 23 11:18:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918977 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVXf6K4rz9s1b for ; Wed, 23 May 2018 21:24:34 +1000 (AEST) Received: from localhost ([::1]:60699 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRse-0001yc-Et for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:24:32 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50933) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRms-0006Lk-Rx for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRms-0007En-3o for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:34 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:60962 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmr-0007Ef-VK for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:34 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9F9B44071181 for ; Wed, 23 May 2018 11:18:33 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 955222166BD9; Wed, 23 May 2018 11:18:32 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:14 +0200 Message-Id: <20180523111817.1463-10-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 23 May 2018 11:18:33 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.7]); Wed, 23 May 2018 11:18:33 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 09/12] migration: Start sending messages X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 28f5cea4d8..2584130c85 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -729,9 +729,6 @@ static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) RAMBlock *block; int i; - /* ToDo: We can't use it until we haven't received a message */ - return 0; - be32_to_cpus(&packet->magic); if (packet->magic != MULTIFD_MAGIC) { error_setg(errp, "multifd: received packet " @@ -965,6 +962,7 @@ static void *multifd_send_thread(void *opaque) { MultiFDSendParams *p = opaque; Error *local_err = NULL; + int ret; trace_multifd_send_thread_start(p->id); @@ -992,7 +990,16 @@ static void *multifd_send_thread(void *opaque) trace_multifd_send(p->id, seq, used, flags); - /* ToDo: send packet here */ + ret = qio_channel_write_all(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret != 0) { + break; + } + + ret = qio_channel_writev_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { + break; + } qemu_mutex_lock(&p->mutex); p->pending_job--; @@ -1204,7 +1211,14 @@ static void *multifd_recv_thread(void *opaque) uint32_t flags; qemu_mutex_unlock(&p->mutex); - /* ToDo: recv packet here */ + ret = qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret == 0) { /* EOF */ + break; + } + if (ret == -1) { /* Error */ + break; + } qemu_mutex_lock(&p->mutex); ret = multifd_recv_unfill_packet(p, &local_err); @@ -1221,6 +1235,11 @@ static void *multifd_recv_thread(void *opaque) p->num_pages += used; qemu_mutex_unlock(&p->mutex); + ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { + break; + } + if (flags & MULTIFD_FLAG_SYNC) { qemu_sem_post(&multifd_recv_state->sem_sync); qemu_sem_wait(&p->sem_sync); From patchwork Wed May 23 11:18:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918985 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVgr62ZLz9s1b for ; Wed, 23 May 2018 21:30:47 +1000 (AEST) Received: from localhost ([::1]:60737 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRye-0007O9-88 for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:30:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50955) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmu-0006NL-CJ for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmt-0007FX-DT for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:36 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:60022 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmt-0007FN-89 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:35 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E8E004023826 for ; Wed, 23 May 2018 11:18:34 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id E18FE2166BB2; Wed, 23 May 2018 11:18:33 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:15 +0200 Message-Id: <20180523111817.1463-11-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 23 May 2018 11:18:34 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Wed, 23 May 2018 11:18:34 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 10/12] migration: Wait for blocking IO X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We have three conditions here: - channel fails -> error - we have to quit: we close the channel and reads fails - normal read that success, we are in bussiness So forget the complications of waiting in a semaphore. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 81 ++++++++++++++++++------------------------------- 1 file changed, 29 insertions(+), 52 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 2584130c85..a707d3ae80 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -595,8 +595,6 @@ typedef struct { bool running; /* should this thread finish */ bool quit; - /* thread has work to do */ - bool pending_job; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -1170,14 +1168,6 @@ static void multifd_recv_sync_main(void) for (i = 0; i < migrate_multifd_channels(); i++) { MultiFDRecvParams *p = &multifd_recv_state->params[i]; - trace_multifd_recv_sync_main_signal(p->id); - qemu_mutex_lock(&p->mutex); - p->pending_job = true; - qemu_mutex_unlock(&p->mutex); - } - for (i = 0; i < migrate_multifd_channels(); i++) { - MultiFDRecvParams *p = &multifd_recv_state->params[i]; - trace_multifd_recv_sync_main_wait(p->id); qemu_sem_wait(&multifd_recv_state->sem_sync); qemu_mutex_lock(&p->mutex); @@ -1190,7 +1180,6 @@ static void multifd_recv_sync_main(void) MultiFDRecvParams *p = &multifd_recv_state->params[i]; trace_multifd_recv_sync_main_signal(p->id); - qemu_sem_post(&p->sem_sync); } trace_multifd_recv_sync_main(multifd_recv_state->seq); @@ -1205,51 +1194,40 @@ static void *multifd_recv_thread(void *opaque) trace_multifd_recv_thread_start(p->id); while (true) { + uint32_t used; + uint32_t flags; + + ret = qio_channel_read_all_eof(p->c, (void *)p->packet, + p->packet_len, &local_err); + if (ret == 0) { /* EOF */ + break; + } + if (ret == -1) { /* Error */ + break; + } + qemu_mutex_lock(&p->mutex); - if (true || p->pending_job) { - uint32_t used; - uint32_t flags; - qemu_mutex_unlock(&p->mutex); - - ret = qio_channel_read_all_eof(p->c, (void *)p->packet, - p->packet_len, &local_err); - if (ret == 0) { /* EOF */ - break; - } - if (ret == -1) { /* Error */ - break; - } - - qemu_mutex_lock(&p->mutex); - ret = multifd_recv_unfill_packet(p, &local_err); - if (ret) { - qemu_mutex_unlock(&p->mutex); - break; - } - - used = p->pages->used; - flags = p->flags; - trace_multifd_recv(p->id, p->seq, used, flags); - p->pending_job = false; - p->num_packets++; - p->num_pages += used; + ret = multifd_recv_unfill_packet(p, &local_err); + if (ret) { qemu_mutex_unlock(&p->mutex); + break; + } - ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); - if (ret != 0) { - break; - } + used = p->pages->used; + flags = p->flags; + trace_multifd_recv(p->id, p->seq, used, flags); + p->num_packets++; + p->num_pages += used; + qemu_mutex_unlock(&p->mutex); - if (flags & MULTIFD_FLAG_SYNC) { - qemu_sem_post(&multifd_recv_state->sem_sync); - qemu_sem_wait(&p->sem_sync); - } - } else if (p->quit) { - qemu_mutex_unlock(&p->mutex); + ret = qio_channel_readv_all(p->c, p->pages->iov, used, &local_err); + if (ret != 0) { break; - } else { - qemu_mutex_unlock(&p->mutex); - /* sometimes there are spurious wakeups */ + } + + if (flags & MULTIFD_FLAG_SYNC) { + qemu_sem_post(&multifd_recv_state->sem_sync); + qemu_sem_wait(&p->sem_sync); } } @@ -1287,7 +1265,6 @@ int multifd_load_setup(void) qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); p->quit = false; - p->pending_job = false; p->id = i; p->pages = multifd_pages_init(page_count); p->packet_len = sizeof(MultiFDPacket_t) From patchwork Wed May 23 11:18:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918990 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVlZ4f48z9s1b for ; Wed, 23 May 2018 21:34:02 +1000 (AEST) Received: from localhost ([::1]:60757 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLS1o-00019t-AG for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:34:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50979) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmv-0006OU-Hy for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:38 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmu-0007Fy-NW for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:37 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44614 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmu-0007Fl-JS for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:36 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 3E2AB195478 for ; Wed, 23 May 2018 11:18:36 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 36C8B2166BB2; Wed, 23 May 2018 11:18:35 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:16 +0200 Message-Id: <20180523111817.1463-12-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:36 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:36 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 11/12] migration: Remove not needed semaphore and quit X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We know quit closing the QIO. Signed-off-by: Juan Quintela Signed-off-by: Juan Quintela --- migration/ram.c | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index a707d3ae80..62eea9e02f 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -587,14 +587,10 @@ typedef struct { QemuThread thread; /* communication channel */ QIOChannel *c; - /* sem where to wait for more work */ - QemuSemaphore sem; /* this mutex protects the following parameters */ QemuMutex mutex; /* is this channel thread running */ bool running; - /* should this thread finish */ - bool quit; /* array of pages to receive */ MultiFDPages_t *pages; /* packet allocated len */ @@ -1115,8 +1111,8 @@ static void multifd_recv_terminate_threads(Error *err) MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_lock(&p->mutex); - p->quit = true; - qemu_sem_post(&p->sem); + object_unref(OBJECT(p->c)); + p->c = NULL; qemu_mutex_unlock(&p->mutex); } } @@ -1139,7 +1135,6 @@ int multifd_load_cleanup(Error **errp) object_unref(OBJECT(p->c)); p->c = NULL; qemu_mutex_destroy(&p->mutex); - qemu_sem_destroy(&p->sem); qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; @@ -1262,9 +1257,7 @@ int multifd_load_setup(void) MultiFDRecvParams *p = &multifd_recv_state->params[i]; qemu_mutex_init(&p->mutex); - qemu_sem_init(&p->sem, 0); qemu_sem_init(&p->sem_sync, 0); - p->quit = false; p->id = i; p->pages = multifd_pages_init(page_count); p->packet_len = sizeof(MultiFDPacket_t) From patchwork Wed May 23 11:18:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juan Quintela X-Patchwork-Id: 918987 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=redhat.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40rVjH1sBYz9s1b for ; Wed, 23 May 2018 21:32:03 +1000 (AEST) Received: from localhost ([::1]:60745 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRzs-0008JA-V1 for incoming@patchwork.ozlabs.org; Wed, 23 May 2018 07:32:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50997) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLRmy-0006RG-Js for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLRmw-0007GY-03 for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:40 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:44622 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLRmv-0007GM-SP for qemu-devel@nongnu.org; Wed, 23 May 2018 07:18:37 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8CE73195478 for ; Wed, 23 May 2018 11:18:37 +0000 (UTC) Received: from secure.mitica (ovpn-117-204.ams2.redhat.com [10.36.117.204]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7FC882166BB2; Wed, 23 May 2018 11:18:36 +0000 (UTC) From: Juan Quintela To: qemu-devel@nongnu.org Date: Wed, 23 May 2018 13:18:17 +0200 Message-Id: <20180523111817.1463-13-quintela@redhat.com> In-Reply-To: <20180523111817.1463-1-quintela@redhat.com> References: <20180523111817.1463-1-quintela@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:37 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.2]); Wed, 23 May 2018 11:18:37 +0000 (UTC) for IP:'10.11.54.6' DOMAIN:'int-mx06.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'quintela@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v13 12/12] migration: Stop sending whole pages through main channel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, dgilbert@redhat.com, peterx@redhat.com Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We have to flush() the QEMUFile because now we sent really few data through that channel. Signed-off-by: Juan Quintela Reviewed-by: Dr. David Alan Gilbert --- migration/ram.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/migration/ram.c b/migration/ram.c index 62eea9e02f..b327805403 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1771,15 +1771,7 @@ static int ram_save_page(RAMState *rs, PageSearchStatus *pss, bool last_stage) static int ram_save_multifd_page(RAMState *rs, RAMBlock *block, ram_addr_t offset) { - uint8_t *p; - - p = block->host + offset; - - ram_counters.transferred += save_page_header(rs, rs->f, block, - offset | RAM_SAVE_FLAG_PAGE); multifd_queue_page(block, offset); - qemu_put_buffer(rs->f, p, TARGET_PAGE_SIZE); - ram_counters.transferred += TARGET_PAGE_SIZE; ram_counters.normal++; return 1; @@ -3013,6 +3005,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque) multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); return 0; } @@ -3090,6 +3083,7 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) multifd_send_sync_main(); out: qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); ram_counters.transferred += 8; ret = qemu_file_get_error(f); @@ -3143,6 +3137,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque) multifd_send_sync_main(); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); + qemu_fflush(f); return 0; }