From patchwork Wed Apr 25 14:35:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 904277 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="pOuXHU9R"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WNTg6bncz9rxs for ; Thu, 26 Apr 2018 00:52:39 +1000 (AEST) Received: from localhost ([::1]:37412 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLmg-0004VH-2V for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 10:52:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:32861) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLWd-0008E4-2j for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBLWb-0007cB-Gs for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:03 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:36202) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fBLWb-0007bf-Aw for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:01 -0400 Received: by mail-pf0-x242.google.com with SMTP id g14so15397522pfh.3 for ; Wed, 25 Apr 2018 07:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=h0muVa4tgPrnd19U+fsYjX6XPExQlZ7KfFCyM3pqQ50=; b=pOuXHU9R7/eYJRpPUqsA+z6q4H4pKs9apaMH48nGsoWQEkDhrohf92OtJ7WqYP71QO qMIbeYqCf31wjGP/2EGnKhjXbK03RStRo2//aqm+Ffnzrv8nPoKeg4z2MVwJQEeAbUuu I7QD0omUOKrJipTBR1nsOI1XHDg+Cz8oeQGLz3vGu7ZQF/xM4jUBMSOmuTnptw7GJEKo lDM8JewPhjnn+1eW05SVzDDWxgGp/AL0K7+BOBz3Sc8jBHbhfRQeNVo8hvyMCCy8yqqy XcjbUpeTlT2nXloY+B/lSQ/urLvwBTab88Z2/nBoe8baoyOw7HtEzKi/0UL+rNEK3l3T 8fMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=h0muVa4tgPrnd19U+fsYjX6XPExQlZ7KfFCyM3pqQ50=; b=shuB5zl6z5bsfwJ3Oa2DXDoS5oz+txhsn8aYjxQVMrBGx8SAhklol4t2DjB4Ffurqf AAf4Rjqh1a8jJgyczIvRbUJF2ajWAwF1Vqsb1+LhbyToAoG74oQ+N4ofTx81L4G1fVHF erAH5Vd7ok2WVFsrxRB3wHKAb5xH6UQn6FvdfVSHIIBl7OkRiLUDI8fPFouV/aw/a2nb I1kiQVF+i0xQ2RivbZFqGJ1EiVvld183tYD1S4kkkMXo6p6YHyKInAfob5UNAaRSVD6M MXcvnQhnMZCnkh6kUrdS+5FgWu6sV7BfK2S3IvP/5nqxiQURRSkfFOip+qNsqPodC1FM vvrw== X-Gm-Message-State: ALQs6tDB6+GzshQJFnvaqWF9oe2XGFWioUaf/211oYPZDHl8uadYA9eL zAVm7vLWliBD4nYvwh9WI1n8iQ== X-Google-Smtp-Source: AIpwx4+asZP4pTnbZZJzCwVAbGpTgws6AOTyLElh79mYpY/zdZ30GM8wfcLk3/t3pbKPcWVreUaAqw== X-Received: by 2002:a17:902:684c:: with SMTP id f12-v6mr29711859pln.139.1524666960497; Wed, 25 Apr 2018 07:36:00 -0700 (PDT) Received: from VM_127_91_centos.localdomain ([119.28.195.137]) by smtp.gmail.com with ESMTPSA id y29sm18244004pff.42.2018.04.25.07.35.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 07:36:00 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: quintela@redhat.com, dgilbert@redhat.com Date: Wed, 25 Apr 2018 22:35:30 +0800 Message-Id: <1524666934-8064-2-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> References: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH v2 1/5] migration: disable RDMA WRITE after postcopy started X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, aviadye@mellanox.com, qemu-devel@nongnu.org, licq@mellanox.com, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" RDMA WRITE operations are performed with no notification to the destination qemu, then the destination qemu can not wakeup. This patch disable RDMA WRITE after postcopy started. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert --- migration/qemu-file.c | 8 ++++++-- migration/rdma.c | 12 ++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index bb63c77..add8c3a 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -253,8 +253,12 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, if (f->hooks && f->hooks->save_page) { int ret = f->hooks->save_page(f, f->opaque, block_offset, offset, size, bytes_sent); - f->bytes_xfer += size; - if (ret != RAM_SAVE_CONTROL_DELAYED) { + if (ret != RAM_SAVE_CONTROL_NOT_SUPP) { + f->bytes_xfer += size; + } + + if (ret != RAM_SAVE_CONTROL_DELAYED && + ret != RAM_SAVE_CONTROL_NOT_SUPP) { if (bytes_sent && *bytes_sent > 0) { qemu_update_position(f, *bytes_sent); } else if (ret < 0) { diff --git a/migration/rdma.c b/migration/rdma.c index da474fc..a22be43 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2927,6 +2927,10 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return RAM_SAVE_CONTROL_NOT_SUPP; + } + qemu_fflush(f); if (size > 0) { @@ -3482,6 +3486,10 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return 0; + } + trace_qemu_rdma_registration_start(flags); qemu_put_be64(f, RAM_SAVE_FLAG_HOOK); qemu_fflush(f); @@ -3504,6 +3512,10 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return 0; + } + qemu_fflush(f); ret = qemu_rdma_drain_cq(f, rdma); From patchwork Wed Apr 25 14:35:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 904280 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DWUWLOUY"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WNXT4MJmz9rxs for ; Thu, 26 Apr 2018 00:55:05 +1000 (AEST) Received: from localhost ([::1]:37422 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLp1-0006FT-LU for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 10:55:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:32880) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLWf-0008G5-3Q for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBLWd-0007eL-Uu for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:05 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:33657) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fBLWd-0007dd-Lx for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:36:03 -0400 Received: by mail-pf0-x244.google.com with SMTP id f15so15404370pfn.0 for ; Wed, 25 Apr 2018 07:36:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Okab3uZssiMUqQpyzMHwmA3VM5Quh0vhCKInirNnY4w=; b=DWUWLOUY2vKicgajf9V9PoF3G/TrG0XYaKcZ2rGcWkbb+cr17VVFQFrovOaC9ydNF3 CKGc4Y1N0s3VUVKXG9CMKuH+oreqhQYR9VLOpYRickIEdf4E8SyuyRGetFGR21ErB+5C 9HR7C+wCgUY/0tmGRUP5QU4KfOrycNdsnpufKgqXSJvgMEeyja5rQIb0HMQVNFvkH6KR AZ9hIhkk/avKVeKMgtdtA4zf/JpQewIYnG8LnwQy/xvp4UJh2YXWk93nGEx+TyFooeQd RsQP1N49dBx4W7PBsaZqAGQTg2/6SynT3HHFEp9a3XhpGOKittv1ba/IJPLobjwZZGDM eD5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Okab3uZssiMUqQpyzMHwmA3VM5Quh0vhCKInirNnY4w=; b=ONOIc6TG1lPOhPmkNtpqBtE2gR4NZJsiB7+d5fkH/QKnJODIV/zaWUo2mtY+BEECHs +/vksLQlOcRIEzP1v0CRbzSBz98htP0Zwfr3NIwKtm94UcPIFa6NTQUIxRfFMbLJ04V+ Tz1YezusMsNHrrRj9gmRMwE2Hlvpf1HHNI8w6D8UT7sbYCMrrWmYHVm6URoGsFf1ZoZH RPnrX5dilHBCsIVKGJ/U8ARiyiG/UTaILkgoX4B+OhQoF0CXaGPMAzF6wJAm2JmZwgKE JdmaFIiToCH551mphP3vScNctFdHU+91zCeW6nZ7Y76ZfiOCE385UeuC/8ZTM+kOYk8B uPyA== X-Gm-Message-State: ALQs6tCRVdy796ynXp2eJAff5u8f46Sp05NPBGpwr7CRQd5ccHjcfuqw qYwcp3bp6hUNiYIdrVBSeXc= X-Google-Smtp-Source: AB8JxZrx04PaSgVxvoYClASUM6riqJdO5RoTASLmywcFWckfcEcPUnCD0Pwj/smAenktayqf2VSMlw== X-Received: by 2002:a17:902:aa4b:: with SMTP id c11-v6mr2265666plr.17.1524666962554; Wed, 25 Apr 2018 07:36:02 -0700 (PDT) Received: from VM_127_91_centos.localdomain ([119.28.195.137]) by smtp.gmail.com with ESMTPSA id y29sm18244004pff.42.2018.04.25.07.36.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 07:36:02 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: quintela@redhat.com, dgilbert@redhat.com Date: Wed, 25 Apr 2018 22:35:31 +0800 Message-Id: <1524666934-8064-3-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> References: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH v2 2/5] migration: create a dedicated connection for rdma return path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, aviadye@mellanox.com, qemu-devel@nongnu.org, licq@mellanox.com, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" If start a RDMA migration with postcopy enabled, the source qemu establish a dedicated connection for return path. Signed-off-by: Lidong Chen Signed-off-by: Dr. David Alan Gilbert Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 91 insertions(+), 3 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index a22be43..c745427 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -387,6 +387,10 @@ typedef struct RDMAContext { uint64_t unregistrations[RDMA_SIGNALED_SEND_MAX]; GHashTable *blockmap; + + /* the RDMAContext for return path */ + struct RDMAContext *return_path; + bool is_return_path; } RDMAContext; #define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma" @@ -2329,10 +2333,22 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) rdma_destroy_id(rdma->cm_id); rdma->cm_id = NULL; } + + /* the destination side, listen_id and channel is shared */ if (rdma->listen_id) { - rdma_destroy_id(rdma->listen_id); + if (!rdma->is_return_path) { + rdma_destroy_id(rdma->listen_id); + } rdma->listen_id = NULL; + + if (rdma->channel) { + if (!rdma->is_return_path) { + rdma_destroy_event_channel(rdma->channel); + } + rdma->channel = NULL; + } } + if (rdma->channel) { rdma_destroy_event_channel(rdma->channel); rdma->channel = NULL; @@ -2561,6 +2577,25 @@ err_dest_init_create_listen_id: } +static void qemu_rdma_return_path_dest_init(RDMAContext *rdma_return_path, + RDMAContext *rdma) +{ + int idx; + + for (idx = 0; idx < RDMA_WRID_MAX; idx++) { + rdma_return_path->wr_data[idx].control_len = 0; + rdma_return_path->wr_data[idx].control_curr = NULL; + } + + /*the CM channel and CM id is shared*/ + rdma_return_path->channel = rdma->channel; + rdma_return_path->listen_id = rdma->listen_id; + + rdma->return_path = rdma_return_path; + rdma_return_path->return_path = rdma; + rdma_return_path->is_return_path = true; +} + static void *qemu_rdma_data_init(const char *host_port, Error **errp) { RDMAContext *rdma = NULL; @@ -3018,6 +3053,8 @@ err: return ret; } +static void rdma_accept_incoming_migration(void *opaque); + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3112,7 +3149,14 @@ static int qemu_rdma_accept(RDMAContext *rdma) } } - qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); + /* Accept the second connection request for return path */ + if (migrate_postcopy() && !rdma->is_return_path) { + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, + NULL, + (void *)(intptr_t)rdma->return_path); + } else { + qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); + } ret = rdma_accept(rdma->cm_id, &conn_param); if (ret) { @@ -3693,6 +3737,10 @@ static void rdma_accept_incoming_migration(void *opaque) trace_qemu_rdma_accept_incoming_migration_accepted(); + if (rdma->is_return_path) { + return; + } + f = qemu_fopen_rdma(rdma, "rb"); if (f == NULL) { ERROR(errp, "could not qemu_fopen_rdma!"); @@ -3707,7 +3755,7 @@ static void rdma_accept_incoming_migration(void *opaque) void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; - RDMAContext *rdma; + RDMAContext *rdma, *rdma_return_path; Error *local_err = NULL; trace_rdma_start_incoming_migration(); @@ -3734,12 +3782,24 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp) trace_rdma_start_incoming_migration_after_rdma_listen(); + /* initialize the RDMAContext for return path */ + if (migrate_postcopy()) { + rdma_return_path = qemu_rdma_data_init(host_port, &local_err); + + if (rdma_return_path == NULL) { + goto err; + } + + qemu_rdma_return_path_dest_init(rdma_return_path, rdma); + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; err: error_propagate(errp, local_err); g_free(rdma); + g_free(rdma_return_path); } void rdma_start_outgoing_migration(void *opaque, @@ -3747,6 +3807,7 @@ void rdma_start_outgoing_migration(void *opaque, { MigrationState *s = opaque; RDMAContext *rdma = qemu_rdma_data_init(host_port, errp); + RDMAContext *rdma_return_path = NULL; int ret = 0; if (rdma == NULL) { @@ -3767,6 +3828,32 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } + /* RDMA postcopy need a seprate queue pair for return path */ + if (migrate_postcopy()) { + rdma_return_path = qemu_rdma_data_init(host_port, errp); + + if (rdma_return_path == NULL) { + goto err; + } + + ret = qemu_rdma_source_init(rdma_return_path, + s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); + + if (ret) { + goto err; + } + + ret = qemu_rdma_connect(rdma_return_path, errp); + + if (ret) { + goto err; + } + + rdma->return_path = rdma_return_path; + rdma_return_path->return_path = rdma; + rdma_return_path->is_return_path = true; + } + trace_rdma_start_outgoing_migration_after_rdma_connect(); s->to_dst_file = qemu_fopen_rdma(rdma, "wb"); @@ -3774,4 +3861,5 @@ void rdma_start_outgoing_migration(void *opaque, return; err: g_free(rdma); + g_free(rdma_return_path); } From patchwork Wed Apr 25 14:35:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 904272 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="iopdLUjz"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WNGP5zgJz9rxs for ; Thu, 26 Apr 2018 00:42:53 +1000 (AEST) Received: from localhost ([::1]:37344 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLdD-00054O-SB for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 10:42:51 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33369) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLXd-0000dv-2X for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBLXZ-00088F-UQ for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:05 -0400 Received: from mail-pf0-x244.google.com ([2607:f8b0:400e:c00::244]:42483) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fBLXZ-00087u-OH for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:01 -0400 Received: by mail-pf0-x244.google.com with SMTP id o16so15391988pfk.9 for ; Wed, 25 Apr 2018 07:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=R9FLfX1tL4VaDyok72m55Q1F0jlLSepx6JB/Y4hCFCM=; b=iopdLUjzXjH0IwYmtN0YXEHBUiFTxYz4ky3MyOXQjJN0j1QIGRSTa85LQpq/lHXK1r OFXWPmI5/0xxNgyJW3xcicB/9hTScgpWV+ClpGQn7WKKjqjGNZUacPE9fsXevkBY0Dqx 6m4UGwogajGKNRftdTk+pGEO3zcPKb8nnLUMTXYVC00E3yfvoG0kCUHwMdFAWDbM+iM4 bULZqErQDqy2XZB854lxfV1VIYPF0TxPBCWOoDF9urwWHSZZ0aUaf+ma5/5VGZPyGcWQ ldYApMWcbWxWUPJuacaQAMh+f7XPT8rJ0O0jjlxzPv+LbyS6UWHUifKsL4yapwQzG3Od ymrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=R9FLfX1tL4VaDyok72m55Q1F0jlLSepx6JB/Y4hCFCM=; b=M9j2DAFxVlK9FY9FfUL6AncR12gNX/V4Xd29ycVKBQOvbMDkUP8EQsO+J45rbwOd24 lp7TZ4LlEBw5RbWz02iwcX8mPWghfNzZfMFCb8z8iFsvgutRXlBJsfCw1J78+ums7Uxv CQjrdwf6Nz4C3ZpoNp9SYSblzop/kFe8iOdq9/+WG86uC4+2r3QGAqrwWptCEzmE1bd4 ojVbedDqeHD6V9xI/13hbIsJaVXRxins8ct/hes/7FOFyLvkYLGqFqYV5JVaLjlXSn5h ZEQvwES4lWREyxKg09l3GlStM4E48hm5u9WGbrBj/LESDvt85uHjmBceo7UBbB+X/68Z Nxvw== X-Gm-Message-State: ALQs6tAmRB3jp/mAnz7c05QGOf7wM0eADTYZ38PQoryJHOQQuE2+wRah bWNHbOCARlHjNN0P4Aw6znE= X-Google-Smtp-Source: AB8JxZq15JwXWMjE/oMnT85TGJifVjgXJbojnjZJxG7NF5S6ciFwTrAa2U+/8oHzoXiZcnNkG/SUzg== X-Received: by 10.99.127.23 with SMTP id a23mr6943199pgd.99.1524667020974; Wed, 25 Apr 2018 07:37:00 -0700 (PDT) Received: from VM_127_91_centos.localdomain ([119.28.195.137]) by smtp.gmail.com with ESMTPSA id y29sm18244004pff.42.2018.04.25.07.36.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 07:37:00 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: quintela@redhat.com, dgilbert@redhat.com Date: Wed, 25 Apr 2018 22:35:32 +0800 Message-Id: <1524666934-8064-4-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> References: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::244 Subject: [Qemu-devel] [PATCH v2 3/5] migration: remove unnecessary variables len in QIOChannelRDMA X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, aviadye@mellanox.com, qemu-devel@nongnu.org, licq@mellanox.com, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Because qio_channel_rdma_writev and qio_channel_rdma_readv maybe invoked by different threads concurrently, this patch removes unnecessary variables len in QIOChannelRDMA and use local variable instead. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Daniel P. Berrangé --- migration/rdma.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index c745427..f5c1d02 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -404,7 +404,6 @@ struct QIOChannelRDMA { QIOChannel parent; RDMAContext *rdma; QEMUFile *file; - size_t len; bool blocking; /* XXX we don't actually honour this yet */ }; @@ -2640,6 +2639,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, int ret; ssize_t done = 0; size_t i; + size_t len = 0; CHECK_ERROR_STATE(); @@ -2659,10 +2659,10 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, while (remaining) { RDMAControlHeader head; - rioc->len = MIN(remaining, RDMA_SEND_INCREMENT); - remaining -= rioc->len; + len = MIN(remaining, RDMA_SEND_INCREMENT); + remaining -= len; - head.len = rioc->len; + head.len = len; head.type = RDMA_CONTROL_QEMU_FILE; ret = qemu_rdma_exchange_send(rdma, &head, data, NULL, NULL, NULL); @@ -2672,8 +2672,8 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, return ret; } - data += rioc->len; - done += rioc->len; + data += len; + done += len; } } @@ -2768,8 +2768,7 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, } } } - rioc->len = done; - return rioc->len; + return done; } /* From patchwork Wed Apr 25 14:35:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 904269 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="fo1WFuj1"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WN8m1xCnz9rxs for ; Thu, 26 Apr 2018 00:38:00 +1000 (AEST) Received: from localhost ([::1]:37306 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLYU-0000nI-Av for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 10:37:58 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33392) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLXe-0000gc-GB for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBLXc-0008AJ-LS for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:06 -0400 Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]:38370) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fBLXc-0008A5-CN for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:04 -0400 Received: by mail-pf0-x242.google.com with SMTP id o76so10688172pfi.5 for ; Wed, 25 Apr 2018 07:37:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ql+dRZCtZHYCN2tS0PM886LoJ19LT7BTN2dHi55o/Nw=; b=fo1WFuj1b1K81GqjkHBKl8TWPgQxxgaNhNqxdIDFekpJufsLwWD2FBZ2uB33S1BeFu qQq7P7+poTKTz7Jme5GCO2+uYWKN/TVh/DDJYpsS3u359gPXI45+0iFWEYS2gwKTYq7s ITeyJRvASNT+W0XHRMkAOUSyJRBcon6jg9nYt0IUwPonfA0qO7EEoC6jTAO5xmyyOx6+ Eb2O05ytbw1J7pXcfWKkS0VhEvfxFZ8a8Z5Yj7ynMIf3D2rovmAQXMV6cO3iu/gzRdPt 89KGPTeYGMNN9pevo9rHsyTp6WH/qxi9Gr+GlVHgBmUzCY7Ymebvfzc6oxB4OZBIPnbs Msog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ql+dRZCtZHYCN2tS0PM886LoJ19LT7BTN2dHi55o/Nw=; b=txXuWux7ZNa+ATTU9ttOLXbLJtUNN3ZuWu9rnNbuMrkTuN6XXSUxolv0Bu7ss7K3T4 1jhQqcogAAd7/fXtsC1MPjyYU9uOMikRYnv2N8ZQdSw7TKWeZco+lkYcIP0WM1W3PtKN inzacGGUD+KnSOLd08vmxsLiQ4jQ0P6W5ziJUzQ2vDlEimr4y1lEDMTP7Vz/9+zmgm60 dnuzNiUeKJTJDmr9s/MzjzpPjZzM1WsT+FYfXtP2I8K585XgKRH6w9SCAXTZNW6tWvnE gjDkaRrBO3ZRMbf/wDsHiBg4TU1zC/QXIVoA0zwtmDSaQwiZk9FCxdAtxB/3NQHcnLXU uFoA== X-Gm-Message-State: ALQs6tB6P9d9HcHF86jZ/vcuUPGp9FCvcHnNtPBkkEl+OXDVuNqQ9aY+ L9xLh1WeC8adDYyFKvqwzC4= X-Google-Smtp-Source: AIpwx4/jsijUXxvjQqT3I00b326Uru9F+yybXmUhzDeSOJm+zfZdX9Dv5FP+jLzvUiMNLxeAFzlArw== X-Received: by 10.99.163.9 with SMTP id s9mr23650539pge.187.1524667023381; Wed, 25 Apr 2018 07:37:03 -0700 (PDT) Received: from VM_127_91_centos.localdomain ([119.28.195.137]) by smtp.gmail.com with ESMTPSA id y29sm18244004pff.42.2018.04.25.07.37.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 07:37:02 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: quintela@redhat.com, dgilbert@redhat.com Date: Wed, 25 Apr 2018 22:35:33 +0800 Message-Id: <1524666934-8064-5-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> References: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::242 Subject: [Qemu-devel] [PATCH v2 4/5] migration: implement bi-directional RDMA QIOChannel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, aviadye@mellanox.com, qemu-devel@nongnu.org, licq@mellanox.com, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch implements bi-directional RDMA QIOChannel. Because different threads may access RDMAQIOChannel concurrently, this patch use RCU to protect it. Signed-off-by: Lidong Chen --- migration/rdma.c | 162 +++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 146 insertions(+), 16 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index f5c1d02..0652224 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -86,6 +86,7 @@ static uint32_t known_capabilities = RDMA_CAPABILITY_PIN_ALL; " to abort!"); \ rdma->error_reported = 1; \ } \ + rcu_read_unlock(); \ return rdma->error_state; \ } \ } while (0) @@ -405,6 +406,7 @@ struct QIOChannelRDMA { RDMAContext *rdma; QEMUFile *file; bool blocking; /* XXX we don't actually honour this yet */ + QemuMutex lock; }; /* @@ -2635,12 +2637,29 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); QEMUFile *f = rioc->file; - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; int ret; ssize_t done = 0; size_t i; size_t len = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + + if (rdma->listen_id) { + rdma = rdma->return_path; + } + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); /* @@ -2650,6 +2669,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, ret = qemu_rdma_write_flush(f, rdma); if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -2669,6 +2689,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -2677,6 +2698,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, } } + rcu_read_unlock(); return done; } @@ -2710,12 +2732,29 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, Error **errp) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; RDMAControlHeader head; int ret = 0; ssize_t i; size_t done = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + + if (!rdma->listen_id) { + rdma = rdma->return_path; + } + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); for (i = 0; i < niov; i++) { @@ -2727,7 +2766,7 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, * were given and dish out the bytes until we run * out of bytes. */ - ret = qemu_rdma_fill(rioc->rdma, data, want, 0); + ret = qemu_rdma_fill(rdma, data, want, 0); done += ret; want -= ret; /* Got what we needed, so go to next iovec */ @@ -2749,25 +2788,28 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } /* * SEND was received with new bytes, now try again. */ - ret = qemu_rdma_fill(rioc->rdma, data, want, 0); + ret = qemu_rdma_fill(rdma, data, want, 0); done += ret; want -= ret; /* Still didn't get enough, so lets just return */ if (want) { if (done == 0) { + rcu_read_unlock(); return QIO_CHANNEL_ERR_BLOCK; } else { break; } } } + rcu_read_unlock(); return done; } @@ -2823,6 +2865,16 @@ qio_channel_rdma_source_prepare(GSource *source, GIOCondition cond = 0; *timeout = -1; + if ((rdma->listen_id && rsource->condition == G_IO_OUT) || + (!rdma->listen_id && rsource->condition == G_IO_IN)) { + rdma = rdma->return_path; + } + + if (!rdma) { + error_report("RDMAContext is NULL when prepare Gsource"); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } @@ -2838,6 +2890,16 @@ qio_channel_rdma_source_check(GSource *source) RDMAContext *rdma = rsource->rioc->rdma; GIOCondition cond = 0; + if ((rdma->listen_id && rsource->condition == G_IO_OUT) || + (!rdma->listen_id && rsource->condition == G_IO_IN)) { + rdma = rdma->return_path; + } + + if (!rdma) { + error_report("RDMAContext is NULL when check Gsource"); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } @@ -2856,6 +2918,16 @@ qio_channel_rdma_source_dispatch(GSource *source, RDMAContext *rdma = rsource->rioc->rdma; GIOCondition cond = 0; + if ((rdma->listen_id && rsource->condition == G_IO_OUT) || + (!rdma->listen_id && rsource->condition == G_IO_IN)) { + rdma = rdma->return_path; + } + + if (!rdma) { + error_report("RDMAContext is NULL when dispatch Gsource"); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } @@ -2905,15 +2977,29 @@ static int qio_channel_rdma_close(QIOChannel *ioc, Error **errp) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext *rdma; trace_qemu_rdma_close(); - if (rioc->rdma) { - if (!rioc->rdma->error_state) { - rioc->rdma->error_state = qemu_file_get_error(rioc->file); - } - qemu_rdma_cleanup(rioc->rdma); - g_free(rioc->rdma); - rioc->rdma = NULL; + + qemu_mutex_lock(&rioc->lock); + rdma = rioc->rdma; + if (!rdma) { + qemu_mutex_unlock(&rioc->lock); + return 0; + } + atomic_rcu_set(&rioc->rdma, NULL); + qemu_mutex_unlock(&rioc->lock); + + if (!rdma->error_state) { + rdma->error_state = qemu_file_get_error(rioc->file); + } + qemu_rdma_cleanup(rdma); + + if (rdma->return_path) { + qemu_rdma_cleanup(rdma->return_path); + g_free(rdma->return_path); } + + g_free(rdma); return 0; } @@ -2956,12 +3042,21 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, size_t size, uint64_t *bytes_sent) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; int ret; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return RAM_SAVE_CONTROL_NOT_SUPP; } @@ -3046,9 +3141,11 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, } } + rcu_read_unlock(); return RAM_SAVE_CONTROL_DELAYED; err: rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -3224,8 +3321,8 @@ static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque) RDMAControlHeader blocks = { .type = RDMA_CONTROL_RAM_BLOCKS_RESULT, .repeat = 1 }; QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; - RDMALocalBlocks *local = &rdma->local_ram_blocks; + RDMAContext *rdma; + RDMALocalBlocks *local; RDMAControlHeader head; RDMARegister *reg, *registers; RDMACompress *comp; @@ -3238,8 +3335,17 @@ static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque) int count = 0; int i = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); + local = &rdma->local_ram_blocks; do { trace_qemu_rdma_registration_handle_wait(); @@ -3469,6 +3575,7 @@ out: if (ret < 0) { rdma->error_state = ret; } + rcu_read_unlock(); return ret; } @@ -3525,11 +3632,19 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, uint64_t flags, void *data) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; + + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return 0; } @@ -3537,6 +3652,7 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, qemu_put_be64(f, RAM_SAVE_FLAG_HOOK); qemu_fflush(f); + rcu_read_unlock(); return 0; } @@ -3549,13 +3665,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, { Error *local_err = NULL, **errp = &local_err; QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; RDMAControlHeader head = { .len = 0, .repeat = 1 }; int ret = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdma); + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return 0; } @@ -3587,6 +3711,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, qemu_rdma_reg_whole_ram_blocks : NULL); if (ret < 0) { ERROR(errp, "receiving remote info!"); + rcu_read_unlock(); return ret; } @@ -3610,6 +3735,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, "not identical on both the source and destination.", local->nb_blocks, nb_dest_blocks); rdma->error_state = -EINVAL; + rcu_read_unlock(); return -EINVAL; } @@ -3626,6 +3752,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, local->block[i].length, rdma->dest_blocks[i].length); rdma->error_state = -EINVAL; + rcu_read_unlock(); return -EINVAL; } local->block[i].remote_host_addr = @@ -3643,9 +3770,11 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, goto err; } + rcu_read_unlock(); return 0; err: rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -3707,6 +3836,7 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode) rioc = QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); rioc->rdma = rdma; + qemu_mutex_init(&rioc->lock); if (mode[0] == 'w') { rioc->file = qemu_fopen_channel_output(QIO_CHANNEL(rioc)); From patchwork Wed Apr 25 14:35:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 904274 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="cDZKMqRS"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40WNLQ35RLz9rxs for ; Thu, 26 Apr 2018 00:46:22 +1000 (AEST) Received: from localhost ([::1]:37373 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLga-00089o-FY for incoming@patchwork.ozlabs.org; Wed, 25 Apr 2018 10:46:20 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33407) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fBLXf-0000he-Fz for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fBLXe-0008BG-Ii for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:07 -0400 Received: from mail-pf0-x241.google.com ([2607:f8b0:400e:c00::241]:43569) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fBLXe-0008Al-DT for qemu-devel@nongnu.org; Wed, 25 Apr 2018 10:37:06 -0400 Received: by mail-pf0-x241.google.com with SMTP id j11so15393712pff.10 for ; Wed, 25 Apr 2018 07:37:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PwHP1AzcBwZHKc6SCBOYyzB9IuaEtpVaqkyyam+SpBo=; b=cDZKMqRSLVny/MLjXVB3LcI8MwQYrxCeI71w2hzB84zWDK0M/friF0mntMTz/Q1o6b 5fTgVfDsV/oBmSN9noD/jsWvbCBBMQh4vECQNcI4sAZsaQourrKK5kn9D3elPKqjj96d 07jfkPjPTbIcDUgomy2Pjfz2vMTHvo21uw9BmUea/mUIVi23oMr1sq6ASPYSGlX6YSff MPwMZcmt2zb1113DpMRsUjX3SaB294WTX6a/YUwDlYnuucKvCJW1dssOE1mRutyV3FOD U6fYs+V32LTW53rg+vJ90yYSzirU7mB57Osw3na30WzFqhy7vFrvofuXSdAifojfmcWl e4gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PwHP1AzcBwZHKc6SCBOYyzB9IuaEtpVaqkyyam+SpBo=; b=nl/L7hJjuAHyriI53Qodc0cmniLUA6UlpAd1D1b5IR5cxMr/Uke5y9klzW0Ts7jnfo Ocqrs0d0/NNMP/5TOIccpkUQC/4uEze5Qb/6qPvkVLEuxTmtLdDUrlNtU6sArXjHxOgf GptXDZkubg5ganMn3xl/ive3mgtQpfCUKx7SFg4tn6KSf/flaay4pgH8851Rt/euEu2T oZ703iyIt7qcJOQOi3SrEZ5oGG5bQ+cXtgNy7ZVQCOrezigtTVbiXT7ixl87Opyp3nuV d48tEHmy6hqtayDJCkEN40D/R7bTmJF+TF+TGBetdOOzc3L2ZdPf3HeTXYbaUOo1yUVp hhRA== X-Gm-Message-State: ALQs6tCUUlWyTMBFm/7jZX8Xe7r/duS8LWqtyRuVJfKD/OOKaXlI8w7t X1dzYAVsS0fzIrHm2d/lp6Q= X-Google-Smtp-Source: AIpwx48cRsKrUWjKBfXwucIsbO/S7zuVHRUalRhWmKklkpuVNTsObq9QDNelkaAfmkRZsreQXWYG7g== X-Received: by 10.99.163.77 with SMTP id v13mr19614133pgn.224.1524667025564; Wed, 25 Apr 2018 07:37:05 -0700 (PDT) Received: from VM_127_91_centos.localdomain ([119.28.195.137]) by smtp.gmail.com with ESMTPSA id y29sm18244004pff.42.2018.04.25.07.37.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Apr 2018 07:37:05 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: quintela@redhat.com, dgilbert@redhat.com Date: Wed, 25 Apr 2018 22:35:34 +0800 Message-Id: <1524666934-8064-6-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> References: <1524666934-8064-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c00::241 Subject: [Qemu-devel] [PATCH v2 5/5] migration: Stop rdma yielding during incoming postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: galsha@mellanox.com, adido@mellanox.com, aviadye@mellanox.com, qemu-devel@nongnu.org, licq@mellanox.com, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" During incoming postcopy, the destination qemu will invoke qemu_rdma_wait_comp_channel in a seprate thread. So does not use rdma yield, and poll the completion channel fd instead. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/migration/rdma.c b/migration/rdma.c index 0652224..4ba9fe2 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -1490,11 +1490,13 @@ static int qemu_rdma_wait_comp_channel(RDMAContext *rdma) * Coroutine doesn't start until migration_fd_process_incoming() * so don't yield unless we know we're running inside of a coroutine. */ - if (rdma->migration_started_on_destination) { + if (rdma->migration_started_on_destination && + migration_incoming_get_current()->state == MIGRATION_STATUS_ACTIVE) { yield_until_fd_readable(rdma->comp_channel->fd); } else { /* This is the source side, we're in a separate thread * or destination prior to migration_fd_process_incoming() + * after postcopy, the destination also in a seprate thread. * we can't yield; so we have to poll the fd. * But we need to be able to handle 'cancel' or an error * without hanging forever.