From patchwork Mon Aug 6 13:29:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953869 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="mbfUw0fG"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdsG3Sd3z9rxx for ; Mon, 6 Aug 2018 23:33:54 +1000 (AEST) Received: from localhost ([::1]:34127 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfdw-0006L7-3a for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:33:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49102) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa0-0003FO-HS for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfZz-0008DD-9V for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:48 -0400 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]:46296) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfZz-0008Cw-3S for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:47 -0400 Received: by mail-pf1-x430.google.com with SMTP id u24-v6so6854132pfn.13 for ; Mon, 06 Aug 2018 06:29:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Gw1eRPs9Eiy6kcsN8PtO8YyrNNK7Uhup63R4pYFlYZQ=; b=mbfUw0fGMEjeBkDZhlLjE0SwWQTjBUZHIeIY4rRdnL4oTksmAtL6LGRUt+Le5kTayq J6Z/+xXK3mgymzEOsay+z9r5TSYn4C4I8Pbr9+m16/wX7iVlNshSqa/u9g7oneZxUWnR rXRLECpvcHu3blLxkRmzd9QRponJ6BVnUw3Z/qdkOkFndKm04k2Y/dxEe68i1KWPoqZ+ gvXnBVz0d3o4XUIsNbHTsK1jd3s+l/zvcVf4eHw3cU1leznIV5GQk086rNIdwb7An+dm 0Jn4EjbT+pCuh8cubPc5aaOYeLSdMRTTz82SOXoiIDR0yrd7xLscL/lEIqSe167cKDCA 4W5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Gw1eRPs9Eiy6kcsN8PtO8YyrNNK7Uhup63R4pYFlYZQ=; b=QkmJqzA3Ll1+t0qrKdN4H6TgiQBfxvY2MJJqTMtpWAeZ4Ip4apI4l1urECqWgPvTQY H7ykiWnpjcQbppUNtKv9scqkxD2W40JbcokC+V9TCLPNO+npc8u0DzehO2wU4RQoNFRH gmps6jCaHfCHmonJqr7e+s+rv5VACMSDeNfYf+8SLWDwe87FChj1t0IBRpOk1cHJ/sKx x2fidb/JIHlYhrn7C77E0m9HXwCe06C3qGwYd6pITZuP3rIU2/1MmbpjQsf0dYQy2RFu 85DTm0V++08+Wm0wWIFZ5mi7RNnFqxmxCp5sQebbHDQPyscLvLqXS+8MKDywG1WVBpN5 duuw== X-Gm-Message-State: AOUpUlGBycPe1t6GkWwrkM4JIK2sUI8ev8cdyWuKpN+ujQ5oUTLjkkfp kZEhsqtXfrM7YfB0vmeDI7s= X-Google-Smtp-Source: AAOMgpdR9ruIpfGwMY2UzJytk2jHfsmZzFzVvzaAqd2xRIOIm83jkXcG/sfQlB8pr98PrywJRf3mDg== X-Received: by 2002:a65:4384:: with SMTP id m4-v6mr14613001pgp.265.1533562186214; Mon, 06 Aug 2018 06:29:46 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:45 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:27 +0800 Message-Id: <1533562177-16447-2-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::430 Subject: [Qemu-devel] [PATCH v6 01/11] migration: disable RDMA WRITE after postcopy started X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen RDMA WRITE operations are performed with no notification to the destination qemu, then the destination qemu can not wakeup. This patch disable RDMA WRITE after postcopy started. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela --- migration/qemu-file.c | 8 ++++++-- migration/rdma.c | 12 ++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/migration/qemu-file.c b/migration/qemu-file.c index 0463f4c..977b9ae 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -253,8 +253,12 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset, if (f->hooks && f->hooks->save_page) { int ret = f->hooks->save_page(f, f->opaque, block_offset, offset, size, bytes_sent); - f->bytes_xfer += size; - if (ret != RAM_SAVE_CONTROL_DELAYED) { + if (ret != RAM_SAVE_CONTROL_NOT_SUPP) { + f->bytes_xfer += size; + } + + if (ret != RAM_SAVE_CONTROL_DELAYED && + ret != RAM_SAVE_CONTROL_NOT_SUPP) { if (bytes_sent && *bytes_sent > 0) { qemu_update_position(f, *bytes_sent); } else if (ret < 0) { diff --git a/migration/rdma.c b/migration/rdma.c index 8bd7159..76424a5 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2921,6 +2921,10 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return RAM_SAVE_CONTROL_NOT_SUPP; + } + qemu_fflush(f); if (size > 0) { @@ -3480,6 +3484,10 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return 0; + } + trace_qemu_rdma_registration_start(flags); qemu_put_be64(f, RAM_SAVE_FLAG_HOOK); qemu_fflush(f); @@ -3502,6 +3510,10 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, CHECK_ERROR_STATE(); + if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + return 0; + } + qemu_fflush(f); ret = qemu_rdma_drain_cq(f, rdma); From patchwork Mon Aug 6 13:29:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953866 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OTN2y4tH"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdnZ2pXcz9s0R for ; Mon, 6 Aug 2018 23:30:42 +1000 (AEST) Received: from localhost ([::1]:34112 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfap-0003HN-VJ for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:30:39 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49131) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa2-0003FZ-9L for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:51 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa1-0008EO-1K for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:50 -0400 Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]:33631) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa0-0008Dk-OU for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:48 -0400 Received: by mail-pg1-x543.google.com with SMTP id r5-v6so6250167pgv.0 for ; Mon, 06 Aug 2018 06:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xaKjbDdmaFRrENVSxuPNBBGbclo+QaleQR6T32e6rxo=; b=OTN2y4tHNU2crTgfb4tufzx4D7EW2SB6Hafj0ZN99rofIVvHLx48p2hSeu2SxKQnHi H6NSFFMGtkKO4qHBO2UG3+HuiAmIBqMdUd1bCw6VI8A7IuS84cekwB+Kq4IKwEpiQcPB ctq3u9gQB82vzCLY0qhgly+GODPXsjBJZNDHniJ6Z6GjrWAaAh4hpY/RjzJ4HhchJMs1 wq5yKfk/nL7r/KStyH9wFf4PfeohoL3DwQfxpRcX8PvfXBB0OCPZFiEvhN8ObZGtrCA9 Btw/4WM4FgmtVOoWyPaIymROfVrIACCS6FgQIowYYkyOcej9ORcXqJqfVv1IwvTxIMSR ZzBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xaKjbDdmaFRrENVSxuPNBBGbclo+QaleQR6T32e6rxo=; b=mTJ82ytp2fLTIPWTEBQXA+wXsbHuwPxDBOMmRKm1GKcHlc67FHIc61kfSq46uN8vdn 4xiXHB8+rXzLJ4sg8fSXUNtoumbALKETZ6T/QruqZoOiQFOMtY8EMXyOC5UJPJTG21Rs b7kPqgwe8Npkbg7aY9EpJp03r/LkejSpQzlexHSzpnjPgx0JJ3v9JHXd+81RS42JC0fb s6lliJa1Xbzi8rWVCcrV12BrbBDT9wyHE8Yoh9mvG3Co0aoZGnvDbaFBB8oPJI1NGi01 B+2YFPpCPAV9FZZAQJlk7/awJ7MoHcjkkQOGDJEqqytPo6+Pa8Dslf+vn3eLWE0ATpvk CvIw== X-Gm-Message-State: AOUpUlF92tFkYBl1jHetxLf2JDGIsC1W4uutth7yVo4ks0BnVW7ai80J G9ZTCMBEcEtRXJobG8tx+xU6xYE5n9o= X-Google-Smtp-Source: AAOMgpf8atAxiWIU9zo7kEl5e/BWSzMlZbdJgzRZTbPN1ykFi8HX6F9996hgLQ5KnBPxLtSKmNCTEA== X-Received: by 2002:a63:db4f:: with SMTP id x15-v6mr14424419pgi.214.1533562187919; Mon, 06 Aug 2018 06:29:47 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:47 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:28 +0800 Message-Id: <1533562177-16447-3-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::543 Subject: [Qemu-devel] [PATCH v6 02/11] migration: create a dedicated connection for rdma return path X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen If start a RDMA migration with postcopy enabled, the source qemu establish a dedicated connection for return path. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert --- migration/rdma.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 91 insertions(+), 3 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index 76424a5..57af5ed 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -387,6 +387,10 @@ typedef struct RDMAContext { uint64_t unregistrations[RDMA_SIGNALED_SEND_MAX]; GHashTable *blockmap; + + /* the RDMAContext for return path */ + struct RDMAContext *return_path; + bool is_return_path; } RDMAContext; #define TYPE_QIO_CHANNEL_RDMA "qio-channel-rdma" @@ -2323,10 +2327,22 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) rdma_destroy_id(rdma->cm_id); rdma->cm_id = NULL; } + + /* the destination side, listen_id and channel is shared */ if (rdma->listen_id) { - rdma_destroy_id(rdma->listen_id); + if (!rdma->is_return_path) { + rdma_destroy_id(rdma->listen_id); + } rdma->listen_id = NULL; + + if (rdma->channel) { + if (!rdma->is_return_path) { + rdma_destroy_event_channel(rdma->channel); + } + rdma->channel = NULL; + } } + if (rdma->channel) { rdma_destroy_event_channel(rdma->channel); rdma->channel = NULL; @@ -2555,6 +2571,25 @@ err_dest_init_create_listen_id: } +static void qemu_rdma_return_path_dest_init(RDMAContext *rdma_return_path, + RDMAContext *rdma) +{ + int idx; + + for (idx = 0; idx < RDMA_WRID_MAX; idx++) { + rdma_return_path->wr_data[idx].control_len = 0; + rdma_return_path->wr_data[idx].control_curr = NULL; + } + + /*the CM channel and CM id is shared*/ + rdma_return_path->channel = rdma->channel; + rdma_return_path->listen_id = rdma->listen_id; + + rdma->return_path = rdma_return_path; + rdma_return_path->return_path = rdma; + rdma_return_path->is_return_path = true; +} + static void *qemu_rdma_data_init(const char *host_port, Error **errp) { RDMAContext *rdma = NULL; @@ -3012,6 +3047,8 @@ err: return ret; } +static void rdma_accept_incoming_migration(void *opaque); + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3106,7 +3143,14 @@ static int qemu_rdma_accept(RDMAContext *rdma) } } - qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); + /* Accept the second connection request for return path */ + if (migrate_postcopy() && !rdma->is_return_path) { + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, + NULL, + (void *)(intptr_t)rdma->return_path); + } else { + qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); + } ret = rdma_accept(rdma->cm_id, &conn_param); if (ret) { @@ -3691,6 +3735,10 @@ static void rdma_accept_incoming_migration(void *opaque) trace_qemu_rdma_accept_incoming_migration_accepted(); + if (rdma->is_return_path) { + return; + } + f = qemu_fopen_rdma(rdma, "rb"); if (f == NULL) { ERROR(errp, "could not qemu_fopen_rdma!"); @@ -3705,7 +3753,7 @@ static void rdma_accept_incoming_migration(void *opaque) void rdma_start_incoming_migration(const char *host_port, Error **errp) { int ret; - RDMAContext *rdma; + RDMAContext *rdma, *rdma_return_path; Error *local_err = NULL; trace_rdma_start_incoming_migration(); @@ -3732,12 +3780,24 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp) trace_rdma_start_incoming_migration_after_rdma_listen(); + /* initialize the RDMAContext for return path */ + if (migrate_postcopy()) { + rdma_return_path = qemu_rdma_data_init(host_port, &local_err); + + if (rdma_return_path == NULL) { + goto err; + } + + qemu_rdma_return_path_dest_init(rdma_return_path, rdma); + } + qemu_set_fd_handler(rdma->channel->fd, rdma_accept_incoming_migration, NULL, (void *)(intptr_t)rdma); return; err: error_propagate(errp, local_err); g_free(rdma); + g_free(rdma_return_path); } void rdma_start_outgoing_migration(void *opaque, @@ -3745,6 +3805,7 @@ void rdma_start_outgoing_migration(void *opaque, { MigrationState *s = opaque; RDMAContext *rdma = qemu_rdma_data_init(host_port, errp); + RDMAContext *rdma_return_path = NULL; int ret = 0; if (rdma == NULL) { @@ -3765,6 +3826,32 @@ void rdma_start_outgoing_migration(void *opaque, goto err; } + /* RDMA postcopy need a seprate queue pair for return path */ + if (migrate_postcopy()) { + rdma_return_path = qemu_rdma_data_init(host_port, errp); + + if (rdma_return_path == NULL) { + goto err; + } + + ret = qemu_rdma_source_init(rdma_return_path, + s->enabled_capabilities[MIGRATION_CAPABILITY_RDMA_PIN_ALL], errp); + + if (ret) { + goto err; + } + + ret = qemu_rdma_connect(rdma_return_path, errp); + + if (ret) { + goto err; + } + + rdma->return_path = rdma_return_path; + rdma_return_path->return_path = rdma; + rdma_return_path->is_return_path = true; + } + trace_rdma_start_outgoing_migration_after_rdma_connect(); s->to_dst_file = qemu_fopen_rdma(rdma, "wb"); @@ -3772,4 +3859,5 @@ void rdma_start_outgoing_migration(void *opaque, return; err: g_free(rdma); + g_free(rdma_return_path); } From patchwork Mon Aug 6 13:29:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953874 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nrB4ubnL"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdwg6vWvz9s4c for ; Mon, 6 Aug 2018 23:36:51 +1000 (AEST) Received: from localhost ([::1]:34149 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfgn-0000CS-Ih for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:36:49 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49147) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa5-0003HS-3f for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa2-0008FB-U7 for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:53 -0400 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:36854) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa2-0008Ev-Ky for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:50 -0400 Received: by mail-pg1-x536.google.com with SMTP id h12-v6so6242887pgs.3 for ; Mon, 06 Aug 2018 06:29:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yDPXchnWZ/0b3ESluGi75juNF44UfWzFS4nvpj5oPcQ=; b=nrB4ubnLwRgYXfznIHi9oWGLv9dGFe7dP0vZiCNUCr4gMw75kNb7d7M+POFVH6zuz6 38LNzr9zL/ZMDFh6VVjM0r8dLPhjkP6byiVHNxbMTMFpsByrTEaFOoko6ktuxlObOQnr jJVJBveYEj5nhaERUx/vGKc/5J5/KXhiIk1pmlySnSDNouanU2I4TCcX0PwvKv/gd5Si wXaH5uHcpwqBUxh8EyaPKP9txMWpYapJNfnnuNse3LF6g7DXj9XwmbLJ+Ym3cBMYhjYv GeDVoSJCU6fv5SHYTq984IzkZ3FNSzB+zBHOrt3u4IpDP8dDOrZXi6J9IFNOuk+9ZSsv 3wVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yDPXchnWZ/0b3ESluGi75juNF44UfWzFS4nvpj5oPcQ=; b=JphFkeMM6OnLk1GvndZnKm1GhloY/0jZfoTPEqMkOB2uq3a9kH6jIjhRJxax+ngNcj +YNh6GtGssVVItEALJfQIOkBT0Fgc7A4Ykc9iP7SU2Bznk/+t+swg4I0SZ6un2mAbj2z BxA8LPR+DurRp6a0dqSgV1/MVnjGodyWV+sqX9Tj5I6INCR1ckCfhbcLlL4tgCLxTgMm 7kq47OgoYfOtKuLEuNmKNzkxxa6DGt4bWpGT5SXMoHsbs9XZC68R0TwS/ynrI0C56iui VO0eLEPWfF5l/x6ZUfUJedwoDATPJiORp5DMdcYskmzyyZDJTE6qrZ6e70uxYBLUcshz 7Z5Q== X-Gm-Message-State: AOUpUlHd/qa6blWXoH5Ulq20hxjZa3yDl1nh2uyWixwm1nMoEC+jiisV Mh/6akxU818Tq8cqZa+Yzx8= X-Google-Smtp-Source: AAOMgpdNHVFsXAenhEOndFdnbzqBH2i3CD/kmCJWyWL/mwYJxXsIT7JMaYBW9UrlznzX0ukdTjUd2A== X-Received: by 2002:a63:3685:: with SMTP id d127-v6mr14632226pga.231.1533562189744; Mon, 06 Aug 2018 06:29:49 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:49 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:29 +0800 Message-Id: <1533562177-16447-4-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::536 Subject: [Qemu-devel] [PATCH v6 03/11] migration: implement bi-directional RDMA QIOChannel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen This patch implements bi-directional RDMA QIOChannel. Because different threads may access RDMAQIOChannel currently, this patch use RCU to protect it. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert --- migration/colo.c | 2 + migration/migration.c | 2 + migration/postcopy-ram.c | 2 + migration/ram.c | 4 + migration/rdma.c | 196 ++++++++++++++++++++++++++++++++++++++++------- migration/savevm.c | 3 + 6 files changed, 183 insertions(+), 26 deletions(-) diff --git a/migration/colo.c b/migration/colo.c index 4381067..88936f5 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -534,6 +534,7 @@ void *colo_process_incoming_thread(void *opaque) uint64_t value; Error *local_err = NULL; + rcu_register_thread(); qemu_sem_init(&mis->colo_incoming_sem, 0); migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE, @@ -666,5 +667,6 @@ out: } migration_incoming_exit_colo(); + rcu_unregister_thread(); return NULL; } diff --git a/migration/migration.c b/migration/migration.c index b7d9854..df0c2cf 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2104,6 +2104,7 @@ static void *source_return_path_thread(void *opaque) int res; trace_source_return_path_thread_entry(); + rcu_register_thread(); retry: while (!ms->rp_state.error && !qemu_file_get_error(rp) && @@ -2243,6 +2244,7 @@ out: trace_source_return_path_thread_end(); ms->rp_state.from_dst_file = NULL; qemu_fclose(rp); + rcu_unregister_thread(); return NULL; } diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 932f188..3952d78 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -853,6 +853,7 @@ static void *postcopy_ram_fault_thread(void *opaque) RAMBlock *rb = NULL; trace_postcopy_ram_fault_thread_entry(); + rcu_register_thread(); mis->last_rb = NULL; /* last RAMBlock we sent part of */ qemu_sem_post(&mis->fault_thread_sem); @@ -1059,6 +1060,7 @@ retry: } } } + rcu_unregister_thread(); trace_postcopy_ram_fault_thread_exit(); g_free(pfd); return NULL; diff --git a/migration/ram.c b/migration/ram.c index 24dea27..4da0930 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -988,6 +988,7 @@ static void *multifd_send_thread(void *opaque) int ret; trace_multifd_send_thread_start(p->id); + rcu_register_thread(); if (multifd_send_initial_packet(p, &local_err) < 0) { goto out; @@ -1050,6 +1051,7 @@ out: p->running = false; qemu_mutex_unlock(&p->mutex); + rcu_unregister_thread(); trace_multifd_send_thread_end(p->id, p->num_packets, p->num_pages); return NULL; @@ -1219,6 +1221,7 @@ static void *multifd_recv_thread(void *opaque) int ret; trace_multifd_recv_thread_start(p->id); + rcu_register_thread(); while (true) { uint32_t used; @@ -1265,6 +1268,7 @@ static void *multifd_recv_thread(void *opaque) p->running = false; qemu_mutex_unlock(&p->mutex); + rcu_unregister_thread(); trace_multifd_recv_thread_end(p->id, p->num_packets, p->num_pages); return NULL; diff --git a/migration/rdma.c b/migration/rdma.c index 57af5ed..a5535fb 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -86,6 +86,7 @@ static uint32_t known_capabilities = RDMA_CAPABILITY_PIN_ALL; " to abort!"); \ rdma->error_reported = 1; \ } \ + rcu_read_unlock(); \ return rdma->error_state; \ } \ } while (0) @@ -402,7 +403,8 @@ typedef struct QIOChannelRDMA QIOChannelRDMA; struct QIOChannelRDMA { QIOChannel parent; - RDMAContext *rdma; + RDMAContext *rdmain; + RDMAContext *rdmaout; QEMUFile *file; bool blocking; /* XXX we don't actually honour this yet */ }; @@ -2630,12 +2632,20 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); QEMUFile *f = rioc->file; - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; int ret; ssize_t done = 0; size_t i; size_t len = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmaout); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); /* @@ -2645,6 +2655,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, ret = qemu_rdma_write_flush(f, rdma); if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -2664,6 +2675,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -2672,6 +2684,7 @@ static ssize_t qio_channel_rdma_writev(QIOChannel *ioc, } } + rcu_read_unlock(); return done; } @@ -2705,12 +2718,20 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, Error **errp) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; RDMAControlHeader head; int ret = 0; ssize_t i; size_t done = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmain); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); for (i = 0; i < niov; i++) { @@ -2722,7 +2743,7 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, * were given and dish out the bytes until we run * out of bytes. */ - ret = qemu_rdma_fill(rioc->rdma, data, want, 0); + ret = qemu_rdma_fill(rdma, data, want, 0); done += ret; want -= ret; /* Got what we needed, so go to next iovec */ @@ -2744,25 +2765,28 @@ static ssize_t qio_channel_rdma_readv(QIOChannel *ioc, if (ret < 0) { rdma->error_state = ret; + rcu_read_unlock(); return ret; } /* * SEND was received with new bytes, now try again. */ - ret = qemu_rdma_fill(rioc->rdma, data, want, 0); + ret = qemu_rdma_fill(rdma, data, want, 0); done += ret; want -= ret; /* Still didn't get enough, so lets just return */ if (want) { if (done == 0) { + rcu_read_unlock(); return QIO_CHANNEL_ERR_BLOCK; } else { break; } } } + rcu_read_unlock(); return done; } @@ -2814,15 +2838,29 @@ qio_channel_rdma_source_prepare(GSource *source, gint *timeout) { QIOChannelRDMASource *rsource = (QIOChannelRDMASource *)source; - RDMAContext *rdma = rsource->rioc->rdma; + RDMAContext *rdma; GIOCondition cond = 0; *timeout = -1; + rcu_read_lock(); + if (rsource->condition == G_IO_IN) { + rdma = atomic_rcu_read(&rsource->rioc->rdmain); + } else { + rdma = atomic_rcu_read(&rsource->rioc->rdmaout); + } + + if (!rdma) { + error_report("RDMAContext is NULL when prepare Gsource"); + rcu_read_unlock(); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } cond |= G_IO_OUT; + rcu_read_unlock(); return cond & rsource->condition; } @@ -2830,14 +2868,28 @@ static gboolean qio_channel_rdma_source_check(GSource *source) { QIOChannelRDMASource *rsource = (QIOChannelRDMASource *)source; - RDMAContext *rdma = rsource->rioc->rdma; + RDMAContext *rdma; GIOCondition cond = 0; + rcu_read_lock(); + if (rsource->condition == G_IO_IN) { + rdma = atomic_rcu_read(&rsource->rioc->rdmain); + } else { + rdma = atomic_rcu_read(&rsource->rioc->rdmaout); + } + + if (!rdma) { + error_report("RDMAContext is NULL when check Gsource"); + rcu_read_unlock(); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } cond |= G_IO_OUT; + rcu_read_unlock(); return cond & rsource->condition; } @@ -2848,14 +2900,28 @@ qio_channel_rdma_source_dispatch(GSource *source, { QIOChannelFunc func = (QIOChannelFunc)callback; QIOChannelRDMASource *rsource = (QIOChannelRDMASource *)source; - RDMAContext *rdma = rsource->rioc->rdma; + RDMAContext *rdma; GIOCondition cond = 0; + rcu_read_lock(); + if (rsource->condition == G_IO_IN) { + rdma = atomic_rcu_read(&rsource->rioc->rdmain); + } else { + rdma = atomic_rcu_read(&rsource->rioc->rdmaout); + } + + if (!rdma) { + error_report("RDMAContext is NULL when dispatch Gsource"); + rcu_read_unlock(); + return FALSE; + } + if (rdma->wr_data[0].control_len) { cond |= G_IO_IN; } cond |= G_IO_OUT; + rcu_read_unlock(); return (*func)(QIO_CHANNEL(rsource->rioc), (cond & rsource->condition), user_data); @@ -2900,15 +2966,32 @@ static int qio_channel_rdma_close(QIOChannel *ioc, Error **errp) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext *rdmain, *rdmaout; trace_qemu_rdma_close(); - if (rioc->rdma) { - if (!rioc->rdma->error_state) { - rioc->rdma->error_state = qemu_file_get_error(rioc->file); - } - qemu_rdma_cleanup(rioc->rdma); - g_free(rioc->rdma); - rioc->rdma = NULL; + + rdmain = rioc->rdmain; + if (rdmain) { + atomic_rcu_set(&rioc->rdmain, NULL); + } + + rdmaout = rioc->rdmaout; + if (rdmaout) { + atomic_rcu_set(&rioc->rdmaout, NULL); } + + synchronize_rcu(); + + if (rdmain) { + qemu_rdma_cleanup(rdmain); + } + + if (rdmaout) { + qemu_rdma_cleanup(rdmaout); + } + + g_free(rdmain); + g_free(rdmaout); + return 0; } @@ -2951,12 +3034,21 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, size_t size, uint64_t *bytes_sent) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; int ret; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmaout); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return RAM_SAVE_CONTROL_NOT_SUPP; } @@ -3041,9 +3133,11 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque, } } + rcu_read_unlock(); return RAM_SAVE_CONTROL_DELAYED; err: rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -3219,8 +3313,8 @@ static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque) RDMAControlHeader blocks = { .type = RDMA_CONTROL_RAM_BLOCKS_RESULT, .repeat = 1 }; QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; - RDMALocalBlocks *local = &rdma->local_ram_blocks; + RDMAContext *rdma; + RDMALocalBlocks *local; RDMAControlHeader head; RDMARegister *reg, *registers; RDMACompress *comp; @@ -3233,8 +3327,17 @@ static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque) int count = 0; int i = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmain); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); + local = &rdma->local_ram_blocks; do { trace_qemu_rdma_registration_handle_wait(); @@ -3468,6 +3571,7 @@ out: if (ret < 0) { rdma->error_state = ret; } + rcu_read_unlock(); return ret; } @@ -3481,10 +3585,18 @@ out: static int rdma_block_notification_handle(QIOChannelRDMA *rioc, const char *name) { - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; int curr; int found = -1; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmain); + + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + /* Find the matching RAMBlock in our local list */ for (curr = 0; curr < rdma->local_ram_blocks.nb_blocks; curr++) { if (!strcmp(rdma->local_ram_blocks.block[curr].block_name, name)) { @@ -3495,6 +3607,7 @@ rdma_block_notification_handle(QIOChannelRDMA *rioc, const char *name) if (found == -1) { error_report("RAMBlock '%s' not found on destination", name); + rcu_read_unlock(); return -ENOENT; } @@ -3502,6 +3615,7 @@ rdma_block_notification_handle(QIOChannelRDMA *rioc, const char *name) trace_rdma_block_notification_handle(name, rdma->next_src_index); rdma->next_src_index++; + rcu_read_unlock(); return 0; } @@ -3524,11 +3638,19 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, uint64_t flags, void *data) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; + + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmaout); + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return 0; } @@ -3536,6 +3658,7 @@ static int qemu_rdma_registration_start(QEMUFile *f, void *opaque, qemu_put_be64(f, RAM_SAVE_FLAG_HOOK); qemu_fflush(f); + rcu_read_unlock(); return 0; } @@ -3548,13 +3671,21 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, { Error *local_err = NULL, **errp = &local_err; QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque); - RDMAContext *rdma = rioc->rdma; + RDMAContext *rdma; RDMAControlHeader head = { .len = 0, .repeat = 1 }; int ret = 0; + rcu_read_lock(); + rdma = atomic_rcu_read(&rioc->rdmaout); + if (!rdma) { + rcu_read_unlock(); + return -EIO; + } + CHECK_ERROR_STATE(); if (migrate_get_current()->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) { + rcu_read_unlock(); return 0; } @@ -3586,6 +3717,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, qemu_rdma_reg_whole_ram_blocks : NULL); if (ret < 0) { ERROR(errp, "receiving remote info!"); + rcu_read_unlock(); return ret; } @@ -3609,6 +3741,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, "not identical on both the source and destination.", local->nb_blocks, nb_dest_blocks); rdma->error_state = -EINVAL; + rcu_read_unlock(); return -EINVAL; } @@ -3625,6 +3758,7 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, local->block[i].length, rdma->dest_blocks[i].length); rdma->error_state = -EINVAL; + rcu_read_unlock(); return -EINVAL; } local->block[i].remote_host_addr = @@ -3642,9 +3776,11 @@ static int qemu_rdma_registration_stop(QEMUFile *f, void *opaque, goto err; } + rcu_read_unlock(); return 0; err: rdma->error_state = ret; + rcu_read_unlock(); return ret; } @@ -3662,10 +3798,15 @@ static const QEMUFileHooks rdma_write_hooks = { static void qio_channel_rdma_finalize(Object *obj) { QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(obj); - if (rioc->rdma) { - qemu_rdma_cleanup(rioc->rdma); - g_free(rioc->rdma); - rioc->rdma = NULL; + if (rioc->rdmain) { + qemu_rdma_cleanup(rioc->rdmain); + g_free(rioc->rdmain); + rioc->rdmain = NULL; + } + if (rioc->rdmaout) { + qemu_rdma_cleanup(rioc->rdmaout); + g_free(rioc->rdmaout); + rioc->rdmaout = NULL; } } @@ -3705,13 +3846,16 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode) } rioc = QIO_CHANNEL_RDMA(object_new(TYPE_QIO_CHANNEL_RDMA)); - rioc->rdma = rdma; if (mode[0] == 'w') { rioc->file = qemu_fopen_channel_output(QIO_CHANNEL(rioc)); + rioc->rdmaout = rdma; + rioc->rdmain = rdma->return_path; qemu_file_set_hooks(rioc->file, &rdma_write_hooks); } else { rioc->file = qemu_fopen_channel_input(QIO_CHANNEL(rioc)); + rioc->rdmain = rdma; + rioc->rdmaout = rdma->return_path; qemu_file_set_hooks(rioc->file, &rdma_read_hooks); } diff --git a/migration/savevm.c b/migration/savevm.c index 7f92567..13e51f0 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1622,6 +1622,7 @@ static void *postcopy_ram_listen_thread(void *opaque) qemu_sem_post(&mis->listen_thread_sem); trace_postcopy_ram_listen_thread_start(); + rcu_register_thread(); /* * Because we're a thread and not a coroutine we can't yield * in qemu_file, and thus we must be blocking now. @@ -1662,6 +1663,7 @@ static void *postcopy_ram_listen_thread(void *opaque) * to leave the guest running and fire MCEs for pages that never * arrived as a desperate recovery step. */ + rcu_unregister_thread(); exit(EXIT_FAILURE); } @@ -1676,6 +1678,7 @@ static void *postcopy_ram_listen_thread(void *opaque) migration_incoming_state_destroy(); qemu_loadvm_state_cleanup(); + rcu_unregister_thread(); return NULL; } From patchwork Mon Aug 6 13:29:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953870 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="T+w7eGnw"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdsR39wgz9s0n for ; Mon, 6 Aug 2018 23:34:03 +1000 (AEST) Received: from localhost ([::1]:34129 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfe3-0006R3-St for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:34:00 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49149) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa5-0003Hc-Cc for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa4-0008Fg-Ei for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:53 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:38300) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa4-0008FR-8V for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:52 -0400 Received: by mail-pg1-x542.google.com with SMTP id k3-v6so6239760pgq.5 for ; Mon, 06 Aug 2018 06:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BhTjWSxI5vI0JqfpTA2MCjzO8Oc/jwHf7sGsu2PlpTU=; b=T+w7eGnwSRBGCRhNi3IdZg9VTrjeAOR/CjAsKCNwM78dkyorHIpf/PyYcD8WVd4nUU qleG4Pr2eSzmf6ove4Z8Ce56rlPLCCq5QVWNw7Xe7IxSIJciIbtxuXqws9fOruyP7KOO bWTV50NenZgB+elz588TIlDu1HEqkZ4UngmLgJV1X6H3ptgvG6M7KvjDN8ET8WKMygxY XWo7CeyE9uZ0lLAYuKi1hag7f7zZBZAeLNi2vuK33+UkVk0efzaBlgnzGgXHjZemg3FQ UUj6bgcyXqZcKUyPOT3MyDkABveBYX2flyQFuEUnwsRs1ljJWmfqN9nJwBis/cqqc3zR 5WNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BhTjWSxI5vI0JqfpTA2MCjzO8Oc/jwHf7sGsu2PlpTU=; b=L+NHOJS4B+TnkdZTgscZAPBCEmmO0SxCSEW5uOeH3pePkbuO7l0OZOaPJhUpPeLeYE P8Oog2jNQF28JKHTv0L1/N5wJubqBHsiKhUfSZfmej7ajdFdhHjq5E2UnJwU0C6ZtK4f xH4wruCh0+TEWhIdyHbl7Vm3mZBpSFclshgF6MYc77ec2Qv8V5HX4V2dzSDDpsKaGoAE mCk7H0pRxRqjzpjAoRQE67boADpCPJCcX4Ta4Ed0WDC4ClMR0NeJAy5a3mhCX97UXA7v WDbWk1j7qpRJpK/8v/nW9dZFHHxIo2nP172ytnGGxGFtylx/Nvjlsl6E774XZ56lgc7p diFg== X-Gm-Message-State: AOUpUlGnxeY23W0rVjGvxi9JJDSU9BLPre5uww1cluAE8monU38Rkfwk 8/KOFdsQO8iTo4lGVK0l0J0= X-Google-Smtp-Source: AAOMgpexjONPNLEzw9mYpTNHsukZynEO3bgEnGGDQ4Z6AhKvL6jA3AhJobx9dNNN52V/lCNsUlaIxg== X-Received: by 2002:a63:788b:: with SMTP id t133-v6mr14561889pgc.329.1533562191462; Mon, 06 Aug 2018 06:29:51 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:51 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:30 +0800 Message-Id: <1533562177-16447-5-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 Subject: [Qemu-devel] [PATCH v6 04/11] migration: Stop rdma yielding during incoming postcopy X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen During incoming postcopy, the destination qemu will invoke qemu_rdma_wait_comp_channel in a seprate thread. So does not use rdma yield, and poll the completion channel fd instead. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela --- migration/rdma.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/migration/rdma.c b/migration/rdma.c index a5535fb..cfb0671 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -1493,11 +1493,13 @@ static int qemu_rdma_wait_comp_channel(RDMAContext *rdma) * Coroutine doesn't start until migration_fd_process_incoming() * so don't yield unless we know we're running inside of a coroutine. */ - if (rdma->migration_started_on_destination) { + if (rdma->migration_started_on_destination && + migration_incoming_get_current()->state == MIGRATION_STATUS_ACTIVE) { yield_until_fd_readable(rdma->comp_channel->fd); } else { /* This is the source side, we're in a separate thread * or destination prior to migration_fd_process_incoming() + * after postcopy, the destination also in a seprate thread. * we can't yield; so we have to poll the fd. * But we need to be able to handle 'cancel' or an error * without hanging forever. From patchwork Mon Aug 6 13:29:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953873 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="iS+8Vg3h"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdwW2YYzz9s0R for ; Mon, 6 Aug 2018 23:36:41 +1000 (AEST) Received: from localhost ([::1]:34146 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfgc-00005c-Jo for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:36:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49171) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa9-0003JB-0s for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa6-0008GS-8I for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:57 -0400 Received: from mail-pl0-x233.google.com ([2607:f8b0:400e:c01::233]:44032) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa6-0008G9-2X for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:54 -0400 Received: by mail-pl0-x233.google.com with SMTP id ba4-v6so5655936plb.11 for ; Mon, 06 Aug 2018 06:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=i+9cu6YEw6HZ98kuAAxb7164x+SFub9Zft6Br89mO6c=; b=iS+8Vg3huZzx+eJGQKc1pji34nxMVzUxJHM0ps6N22WovhQyRvsiXhHGaQ1ME95RGf dQLfvtC7Bdj140gFmQ9jhNlxZosWOxtk5uNUuYDRWuhLmBFEg/jpHHmm7c2QU8P5HWfG sGWO8Fzio83DrUrOl7iBt/2Zd2JZM1dh8fBGS03WH1eF7uJ4h4rfltdoAkZyYRqcjsmw bs6PT8AtVf3YRhMPR3RVvf6ubIVT/EtufReplPmxxTh2HsbuEEh8F04NYGIZk5tTLjLI 5qTaLesEbS8MdGCAYh190L1Lumkai1VW6gPCcgwACNrmGbzwN9eXIsji7CUPutfCleRs xxpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=i+9cu6YEw6HZ98kuAAxb7164x+SFub9Zft6Br89mO6c=; b=nywCx4/s8vhOOOkE0S72LWJYK18wDdWNYrKLM13QjVtNraN1ck+g706IQEvuZ2CUkD 9UnQWnETRqunIkNimjQxafIr6PeiGh19+LBOA0iKUQvmoDWf5iQ4e6LCjBUqBesS8D88 RMj+Gp/TbR6USrhjDtaNJXFFbJs5dQg2fj3UiI8a6zo7FqP5+YjyJ+BXukXb4RSERxI3 iGnYmr/d5JW+wuvtSGbjbiQJyhcahQBR78nVMKApou7q9vv5NxzqzvcYKSqod/rqP9Xk Xk/JUOP3BIHcY9BmJHNtpEBo4Fc2YQmfZzJJgv64kEnx6Fx/kTDc873pCdxJ2v8JJHCN AePQ== X-Gm-Message-State: AOUpUlGz9eKWKcPXafjDWolL0utC67s+zIX16LCBycfRRREo55JXqIKe 4gtGEGLqPDnG12FlcXRqsyA= X-Google-Smtp-Source: AAOMgpfAjStctzjgbD4qa5dAuIE7O646EhwB2+xRU0JSHDc9/SX6GQXTgBgiFySq3Sq0TrHmz6eb0g== X-Received: by 2002:a17:902:d24:: with SMTP id 33-v6mr13849580plu.211.1533562193258; Mon, 06 Aug 2018 06:29:53 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:52 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:31 +0800 Message-Id: <1533562177-16447-6-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:400e:c01::233 Subject: [Qemu-devel] [PATCH v6 05/11] migration: implement io_set_aio_fd_handler function for RDMA QIOChannel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen if qio_channel_rdma_readv return QIO_CHANNEL_ERR_BLOCK, the destination qemu crash. The backtrace is: (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00000000008db50e in qio_channel_set_aio_fd_handler (ioc=0x38111e0, ctx=0x3726080, io_read=0x8db841 , io_write=0x0, opaque=0x38111e0) at io/channel.c: #2 0x00000000008db952 in qio_channel_set_aio_fd_handlers (ioc=0x38111e0) at io/channel.c:438 #3 0x00000000008dbab4 in qio_channel_yield (ioc=0x38111e0, condition=G_IO_IN) at io/channel.c:47 #4 0x00000000007a870b in channel_get_buffer (opaque=0x38111e0, buf=0x440c038 "", pos=0, size=327 at migration/qemu-file-channel.c:83 #5 0x00000000007a70f6 in qemu_fill_buffer (f=0x440c000) at migration/qemu-file.c:299 #6 0x00000000007a79d0 in qemu_peek_byte (f=0x440c000, offset=0) at migration/qemu-file.c:562 #7 0x00000000007a7a22 in qemu_get_byte (f=0x440c000) at migration/qemu-file.c:575 #8 0x00000000007a7c78 in qemu_get_be32 (f=0x440c000) at migration/qemu-file.c:655 #9 0x00000000007a0508 in qemu_loadvm_state (f=0x440c000) at migration/savevm.c:2126 #10 0x0000000000794141 in process_incoming_migration_co (opaque=0x0) at migration/migration.c:366 #11 0x000000000095c598 in coroutine_trampoline (i0=84033984, i1=0) at util/coroutine-ucontext.c:1 #12 0x00007f9c0db56d40 in ?? () from /lib64/libc.so.6 #13 0x00007f96fe858760 in ?? () #14 0x0000000000000000 in ?? () RDMA QIOChannel not implement io_set_aio_fd_handler. so qio_channel_set_aio_fd_handler will access NULL pointer. Signed-off-by: Lidong Chen Reviewed-by: Juan Quintela Reviewed-by: Juan Quintela --- migration/rdma.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index cfb0671..d6bbf28 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2963,6 +2963,21 @@ static GSource *qio_channel_rdma_create_watch(QIOChannel *ioc, return source; } +static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, + AioContext *ctx, + IOHandler *io_read, + IOHandler *io_write, + void *opaque) +{ + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + if (io_read) { + aio_set_fd_handler(ctx, rioc->rdmain->comp_channel->fd, + false, io_read, io_write, NULL, opaque); + } else { + aio_set_fd_handler(ctx, rioc->rdmaout->comp_channel->fd, + false, io_read, io_write, NULL, opaque); + } +} static int qio_channel_rdma_close(QIOChannel *ioc, Error **errp) @@ -3822,6 +3837,7 @@ static void qio_channel_rdma_class_init(ObjectClass *klass, ioc_klass->io_set_blocking = qio_channel_rdma_set_blocking; ioc_klass->io_close = qio_channel_rdma_close; ioc_klass->io_create_watch = qio_channel_rdma_create_watch; + ioc_klass->io_set_aio_fd_handler = qio_channel_rdma_set_aio_fd_handler; } static const TypeInfo qio_channel_rdma_info = { From patchwork Mon Aug 6 13:29:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Hrp62WMm"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdnf2w4yz9rxx for ; Mon, 6 Aug 2018 23:30:46 +1000 (AEST) Received: from localhost ([::1]:34115 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfau-0003Ld-0o for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:30:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49172) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfa9-0003JC-0q for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa7-0008Gt-WA for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:57 -0400 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:45777) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa7-0008Gl-Oz for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:55 -0400 Received: by mail-pg1-x536.google.com with SMTP id f1-v6so6224119pgq.12 for ; Mon, 06 Aug 2018 06:29:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=V6VZn/TLDJ0rcpWXxTwI8ayG4VlrvGWwAP//SDZcvS8=; b=Hrp62WMmoXHqXZ8Y3igaW83kYSNRL0LNViqNYoTzZaaXGNnGeOPY/GSJoZkUqzovFa MlISneU/Jbgoa0lReBvIzmuWqAJBxsZc207JPGaXZOD1d+smhWQmkm3bZzPWzPb7FL/q OLuHA93CGFTHiytZ0+aQ81+Bt16oAQYGNyOuXEcoDAab4UfUvOrbzllGg9MmATX6io1M hc3R9L4HYzrajFz1icwuezDuvfns2om+f0CsKjikxBs/0fZRcXEsZuhBNmeVITRVHfmH xxQYboDNcHkeP2hmpKqaai3mLWwuCGhZvlALXKyXCnW+6xLDtWyLYFRvb9Pvpq2Wowf6 jaUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=V6VZn/TLDJ0rcpWXxTwI8ayG4VlrvGWwAP//SDZcvS8=; b=Eh06mBip5tPbjw3HRXdNctsz02TNvvxEF/ElVyxpMvU1+0crUbb6KPkJ89QYS6EJro M4JFArLgzl0VNGsq6nMDZXGZmMT02JoVtZJSmOZqrBfh2kFJ7tO6sbvwio1sB/mpZdCL cav42EpH3pKLPNAaZmZXsYnBJhwQ9hAQHPd6di0GiNvhKqQ5mFLW6Y1i4iYAL39PQLnQ 3vwXZDSQ069ZPHehkem7Z+8kYF6ANHee8qqvxUUWMtVqG7ZgrFxgZuqVwHbK8KXOG4hy 6lb76NuS9DUZ7CLyFRlLGZxX1h+oDmi+6H04hfLFaGccysWXMBhqdNibXKijgHLmUCYA pPUQ== X-Gm-Message-State: AOUpUlFiVYnEGLvKr445UnkzuNtM1XcPbOjTsyr1cyWvJ3qy5FGac+h/ Qte+YnmB6AQjAtUp+1ao1UqfqgM5H2U= X-Google-Smtp-Source: AAOMgpcPi9cu2pcWPnyzuVsXWSbf6OO60tss2r5TQwwtnMy0IGhD59ZGu5xyAbx9wR2T1WWVjE2axA== X-Received: by 2002:a62:aa02:: with SMTP id e2-v6mr16939868pff.211.1533562194980; Mon, 06 Aug 2018 06:29:54 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:54 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:32 +0800 Message-Id: <1533562177-16447-7-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::536 Subject: [Qemu-devel] [PATCH v6 06/11] migration: invoke qio_channel_yield only when qemu_in_coroutine() X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen when qio_channel_read return QIO_CHANNEL_ERR_BLOCK, the source qemu crash. The backtrace is: (gdb) bt #0 0x00007fb20aba91d7 in raise () from /lib64/libc.so.6 #1 0x00007fb20abaa8c8 in abort () from /lib64/libc.so.6 #2 0x00007fb20aba2146 in __assert_fail_base () from /lib64/libc.so.6 #3 0x00007fb20aba21f2 in __assert_fail () from /lib64/libc.so.6 #4 0x00000000008dba2d in qio_channel_yield (ioc=0x22f9e20, condition=G_IO_IN) at io/channel.c:460 #5 0x00000000007a870b in channel_get_buffer (opaque=0x22f9e20, buf=0x3d54038 "", pos=0, size=32768) at migration/qemu-file-channel.c:83 #6 0x00000000007a70f6 in qemu_fill_buffer (f=0x3d54000) at migration/qemu-file.c:299 #7 0x00000000007a79d0 in qemu_peek_byte (f=0x3d54000, offset=0) at migration/qemu-file.c:562 #8 0x00000000007a7a22 in qemu_get_byte (f=0x3d54000) at migration/qemu-file.c:575 #9 0x00000000007a7c46 in qemu_get_be16 (f=0x3d54000) at migration/qemu-file.c:647 #10 0x0000000000796db7 in source_return_path_thread (opaque=0x2242280) at migration/migration.c:1794 #11 0x00000000009428fa in qemu_thread_start (args=0x3e58420) at util/qemu-thread-posix.c:504 #12 0x00007fb20af3ddc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fb20ac6b74d in clone () from /lib64/libc.so.6 This patch fixed by invoke qio_channel_yield only when qemu_in_coroutine(). Signed-off-by: Lidong Chen Reviewed-by: Juan Quintela Reviewed-by: Juan Quintela --- migration/qemu-file-channel.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/migration/qemu-file-channel.c b/migration/qemu-file-channel.c index e202d73..8e639eb 100644 --- a/migration/qemu-file-channel.c +++ b/migration/qemu-file-channel.c @@ -49,7 +49,11 @@ static ssize_t channel_writev_buffer(void *opaque, ssize_t len; len = qio_channel_writev(ioc, local_iov, nlocal_iov, NULL); if (len == QIO_CHANNEL_ERR_BLOCK) { - qio_channel_wait(ioc, G_IO_OUT); + if (qemu_in_coroutine()) { + qio_channel_yield(ioc, G_IO_OUT); + } else { + qio_channel_wait(ioc, G_IO_OUT); + } continue; } if (len < 0) { @@ -80,7 +84,11 @@ static ssize_t channel_get_buffer(void *opaque, ret = qio_channel_read(ioc, (char *)buf, size, NULL); if (ret < 0) { if (ret == QIO_CHANNEL_ERR_BLOCK) { - qio_channel_yield(ioc, G_IO_IN); + if (qemu_in_coroutine()) { + qio_channel_yield(ioc, G_IO_IN); + } else { + qio_channel_wait(ioc, G_IO_IN); + } } else { /* XXX handle Error * object */ return -EIO; From patchwork Mon Aug 6 13:29:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953875 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="txaRGJzm"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdyt2JXHz9s0R for ; Mon, 6 Aug 2018 23:38:46 +1000 (AEST) Received: from localhost ([::1]:34160 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfid-0001dz-Gq for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:38:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49188) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfaA-0003KT-Oi for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfa9-0008Hb-S3 for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:58 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:42904) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfa9-0008HI-M4 for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:57 -0400 Received: by mail-pg1-x542.google.com with SMTP id y4-v6so6233467pgp.9 for ; Mon, 06 Aug 2018 06:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NJ8AlY4mjuV9hYO17hGvZ8dmi6Tk0gA05ynUsTBv4Fw=; b=txaRGJzmNB37ACeFnpbG7H6NsYX7i2+dFDAyUTU+ELuwbulyGvQerZiocajKK3R8G7 giC1YyC7B2fpT0n00pYh4bmRUGWrpj+yns8HrGdFGtQt/9cPIYo6UMJ1cgDrwHcjvaF9 4RtAtInU9TB5RoN6BoLnHx2XMsZCZxHg3mpXHYsfQ7G5Lth27yqcD2mEKb5rlMLAa2wS 8LLz61dHQyHbLE/NcBmpQCCWZxqkwf5xPLxXtTz7ET3giQz1a+pMAR+nv6VHRKVhXECR PrM7IOgFjP8qw8kCRKF+Cygv3gw8KPSJxOQ7XV6Xd7ST1USQWTtGmGEtNbew77wNj/uY pT9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NJ8AlY4mjuV9hYO17hGvZ8dmi6Tk0gA05ynUsTBv4Fw=; b=clEsZSK0zEuOWZ8nDOcJnoccrRiW7/hIwtY6PMRuDoSXyvnZCn1GnO8OFba/W13kaN S41CB4R95BXIJIGutI8iZQzTgacoY9N2c0sLP5jNSAPiHPC532QjUfFE4yj/D4FZ+eeJ ceDNxNI/ii+5yfec6LquDJAAOp14PL7BKp7YpUqcw04FfUz4RVniz4IUxgLJmsV8hFHJ slCp4cLdJW2cVIXe0BYhEsslwkAgUjpChyRCUQbtGiAn8TdtfRy159g158olW1VJ7FAt ceQP+JhBNp+TAaFYpzn4IuurhrQbn6V/+LZ5+cQnU9TqIEOu8lT8Wsq28AslU/tAzTHQ eTxQ== X-Gm-Message-State: AOUpUlEq2amVKDFbj86SBqm5RyNRt/kPPokv1X5klEBC5VQfolm+PSge pvGM4eDrXxux/qR9fPU2CqI= X-Google-Smtp-Source: AAOMgpfSZ7KNh7BU2M9tXaJaueKbDPjMLRHKwHaX5Q5l4Dcy4m43jBR0ysCdsa1qKJ4puYpQR5O6tA== X-Received: by 2002:a62:1b4a:: with SMTP id b71-v6mr16797446pfb.221.1533562196950; Mon, 06 Aug 2018 06:29:56 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:56 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:33 +0800 Message-Id: <1533562177-16447-8-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 Subject: [Qemu-devel] [PATCH v6 07/11] migration: poll the cm event while wait RDMA work request completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gal Shachaf , Lidong Chen , Aviad Yehezkel , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen If the peer qemu is crashed, the qemu_rdma_wait_comp_channel function maybe loop forever. so we should also poll the cm event fd, and when receive RDMA_CM_EVENT_DISCONNECTED and RDMA_CM_EVENT_DEVICE_REMOVAL, we consider some error happened. Signed-off-by: Lidong Chen Signed-off-by: Gal Shachaf Signed-off-by: Aviad Yehezkel Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela --- migration/rdma.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index d6bbf28..673f126 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -1489,6 +1489,9 @@ static uint64_t qemu_rdma_poll(RDMAContext *rdma, uint64_t *wr_id_out, */ static int qemu_rdma_wait_comp_channel(RDMAContext *rdma) { + struct rdma_cm_event *cm_event; + int ret = -1; + /* * Coroutine doesn't start until migration_fd_process_incoming() * so don't yield unless we know we're running inside of a coroutine. @@ -1505,13 +1508,37 @@ static int qemu_rdma_wait_comp_channel(RDMAContext *rdma) * without hanging forever. */ while (!rdma->error_state && !rdma->received_error) { - GPollFD pfds[1]; + GPollFD pfds[2]; pfds[0].fd = rdma->comp_channel->fd; pfds[0].events = G_IO_IN | G_IO_HUP | G_IO_ERR; + pfds[0].revents = 0; + + pfds[1].fd = rdma->channel->fd; + pfds[1].events = G_IO_IN | G_IO_HUP | G_IO_ERR; + pfds[1].revents = 0; + /* 0.1s timeout, should be fine for a 'cancel' */ - switch (qemu_poll_ns(pfds, 1, 100 * 1000 * 1000)) { + switch (qemu_poll_ns(pfds, 2, 100 * 1000 * 1000)) { + case 2: case 1: /* fd active */ - return 0; + if (pfds[0].revents) { + return 0; + } + + if (pfds[1].revents) { + ret = rdma_get_cm_event(rdma->channel, &cm_event); + if (!ret) { + rdma_ack_cm_event(cm_event); + } + + error_report("receive cm event while wait comp channel," + "cm event is %d", cm_event->event); + if (cm_event->event == RDMA_CM_EVENT_DISCONNECTED || + cm_event->event == RDMA_CM_EVENT_DEVICE_REMOVAL) { + return -EPIPE; + } + } + break; case 0: /* Timeout, go around again */ break; From patchwork Mon Aug 6 13:29:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953871 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="fCo/GFr+"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdsX2qN9z9rxx for ; Mon, 6 Aug 2018 23:34:08 +1000 (AEST) Received: from localhost ([::1]:34131 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfe8-0006Ub-QI for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:34:04 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49200) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfaC-0003Md-RL for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfaB-0008IC-OE for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:00 -0400 Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]:33953) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfaB-0008I0-GB for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:29:59 -0400 Received: by mail-pg1-x535.google.com with SMTP id y5-v6so6251375pgv.1 for ; Mon, 06 Aug 2018 06:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UaUc+NfHOvncluzDJkjXLlo2mSi7ZcHceZZz9mI6xM4=; b=fCo/GFr+J/6bt2OPOLXZQDnwfWAaVX5epXI9I5GOUbd7Qz6RhdpAIa2lq5HYcuPHiZ n6a//5mvmu3WhJkB4FY0Y76bIoM/0Oy6C4YhXAA1SNsgzNdyIG0G1fYEfXr80x32Fj8b VPM6j7Av/hpOYqzNKfKr7DaDwWwMtqJBgKUSU6kDkbudBDFzFPrQxtnIxjrMQbFGBSWA cGfTLiVnZaPr4g976iL0uhVZMxERrjWMm6ZaiG0rTA9Z3C2xa+K9IN99x9G1gwd9m9Iu z80axT09CIXhm3r3qeoQuNnpDCEkyHWt4dErgNplU6z/kNQB5SvsbOtUAMiaHCtvda46 k36g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UaUc+NfHOvncluzDJkjXLlo2mSi7ZcHceZZz9mI6xM4=; b=K5cG44lrF57UCxJgwXTAjH30vVQvVlUP0+N19hi7Pwol/MC6CS1yye3f9oNisyZQAX PJFigiZiV0tt+GuDotnUxed+IcYO/Pq4CbEfCdao3KI5cmLBdEHElFYCC77hz2+WQz1a /wHoGNlWpfgNPGG2jmxOz4rj7J1ers1guFgZ6nsjgjolxJi84bJUhKsFK82JmKe5mYr0 WFLgPQ/LSZHXp9FYKW8RNDtqafUgvhsLqMOMEfrz7HgN4sPfclbXmjbPYjKScZTbai/L HgZO/kczWHILr3xry6PiemMJSjJazEDJ8vs/pwIjI2ZUSLQHbh9pBF5G2kCA3zb3Aj96 2dZg== X-Gm-Message-State: AOUpUlF/8SxsgkTfPDp388R+yon34jzXl8QT7/bU01vYCPm1Vuu0eXpd efVUV3NAo5o9ITv1m1T0Gbg= X-Google-Smtp-Source: AAOMgpe45al1Ai27lRzScf9F5kjPAKnK8NYZZIU7WK4nkm16InwQPzi4cs8XrVq7c1u65dSPWv+DRQ== X-Received: by 2002:aa7:8713:: with SMTP id b19-v6mr17081155pfo.151.1533562198669; Mon, 06 Aug 2018 06:29:58 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:58 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:34 +0800 Message-Id: <1533562177-16447-9-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::535 Subject: [Qemu-devel] [PATCH v6 08/11] migration: implement the shutdown for RDMA QIOChannel X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lidong Chen , qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Lidong Chen Because RDMA QIOChannel not implement shutdown function, If the to_dst_file was set error, the return path thread will wait forever. and the migration thread will wait return path thread exit. the backtrace of return path thread is: (gdb) bt #0 0x00007f372a76bb0f in ppoll () from /lib64/libc.so.6 #1 0x000000000071dc24 in qemu_poll_ns (fds=0x7ef7091d0580, nfds=2, timeout=100000000) at qemu-timer.c:325 #2 0x00000000006b2fba in qemu_rdma_wait_comp_channel (rdma=0xd424000) at migration/rdma.c:1501 #3 0x00000000006b3191 in qemu_rdma_block_for_wrid (rdma=0xd424000, wrid_requested=4000, byte_len=0x7ef7091d0640) at migration/rdma.c:1580 #4 0x00000000006b3638 in qemu_rdma_exchange_get_response (rdma=0xd424000, head=0x7ef7091d0720, expecting=3, idx=0) at migration/rdma.c:1726 #5 0x00000000006b3ad6 in qemu_rdma_exchange_recv (rdma=0xd424000, head=0x7ef7091d0720, expecting=3) at migration/rdma.c:1903 #6 0x00000000006b5d03 in qemu_rdma_get_buffer (opaque=0x6a57dc0, buf=0x5c80030 "", pos=8, size=32768) at migration/rdma.c:2714 #7 0x00000000006a9635 in qemu_fill_buffer (f=0x5c80000) at migration/qemu-file.c:232 #8 0x00000000006a9ecd in qemu_peek_byte (f=0x5c80000, offset=0) at migration/qemu-file.c:502 #9 0x00000000006a9f1f in qemu_get_byte (f=0x5c80000) at migration/qemu-file.c:515 #10 0x00000000006aa162 in qemu_get_be16 (f=0x5c80000) at migration/qemu-file.c:591 #11 0x00000000006a46d3 in source_return_path_thread ( opaque=0xd826a0 ) at migration/migration.c:1331 #12 0x00007f372aa49e25 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f372a77635d in clone () from /lib64/libc.so.6 the backtrace of migration thread is: (gdb) bt #0 0x00007f372aa4af57 in pthread_join () from /lib64/libpthread.so.0 #1 0x00000000007d5711 in qemu_thread_join (thread=0xd826f8 ) at util/qemu-thread-posix.c:504 #2 0x00000000006a4bc5 in await_return_path_close_on_source ( ms=0xd826a0 ) at migration/migration.c:1460 #3 0x00000000006a53e4 in migration_completion (s=0xd826a0 , current_active_state=4, old_vm_running=0x7ef7089cf976, start_time=0x7ef7089cf980) at migration/migration.c:1695 #4 0x00000000006a5c54 in migration_thread (opaque=0xd826a0 ) at migration/migration.c:1837 #5 0x00007f372aa49e25 in start_thread () from /lib64/libpthread.so.0 #6 0x00007f372a77635d in clone () from /lib64/libc.so.6 Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Juan Quintela --- migration/rdma.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/migration/rdma.c b/migration/rdma.c index 673f126..1affc46 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3039,6 +3039,45 @@ static int qio_channel_rdma_close(QIOChannel *ioc, return 0; } +static int +qio_channel_rdma_shutdown(QIOChannel *ioc, + QIOChannelShutdown how, + Error **errp) +{ + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext *rdmain, *rdmaout; + + rcu_read_lock(); + + rdmain = atomic_rcu_read(&rioc->rdmain); + rdmaout = atomic_rcu_read(&rioc->rdmain); + + switch (how) { + case QIO_CHANNEL_SHUTDOWN_READ: + if (rdmain) { + rdmain->error_state = -1; + } + break; + case QIO_CHANNEL_SHUTDOWN_WRITE: + if (rdmaout) { + rdmaout->error_state = -1; + } + break; + case QIO_CHANNEL_SHUTDOWN_BOTH: + default: + if (rdmain) { + rdmain->error_state = -1; + } + if (rdmaout) { + rdmaout->error_state = -1; + } + break; + } + + rcu_read_unlock(); + return 0; +} + /* * Parameters: * @offset == 0 : @@ -3865,6 +3904,7 @@ static void qio_channel_rdma_class_init(ObjectClass *klass, ioc_klass->io_close = qio_channel_rdma_close; ioc_klass->io_create_watch = qio_channel_rdma_create_watch; ioc_klass->io_set_aio_fd_handler = qio_channel_rdma_set_aio_fd_handler; + ioc_klass->io_shutdown = qio_channel_rdma_shutdown; } static const TypeInfo qio_channel_rdma_info = { From patchwork Mon Aug 6 13:29:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953877 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="E7OEjXXe"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kf0q4f5Wz9rxx for ; Mon, 6 Aug 2018 23:40:27 +1000 (AEST) Received: from localhost ([::1]:34168 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfkH-0002zr-BZ for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:40:25 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49217) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfaE-0003Ou-7X for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfaD-0008Ip-8m for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:02 -0400 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]:33943) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfaD-0008IY-2t for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:01 -0400 Received: by mail-pg1-x52a.google.com with SMTP id y5-v6so6251425pgv.1 for ; Mon, 06 Aug 2018 06:30:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mbB7jKMgu1mG/LYryAL3c2j/fz/WLPMVJBJA6CzcPqo=; b=E7OEjXXeJxtIbtD/92mqjQF/LVidRto4mCaWPu9bws7C5uEQcmaq2/HEwA1GqU30aB k6MxNGr+CK9NCAA+lDbgbLw634xqPOMF99TKRYLwtaEVKRGL/M5pX6+ZHutNGRjRiVWl OFBTY/8H/sbFB7XmXvVf94cYh2BqVfFDOO3KP7Ts77vYW6biqs6cbHo4AscdsQ0HJAME 2fcM9Y8Rz+x+LMS3QhfBUBqkVft4hJ6eYeAfVLhpSBYx6dI7iPuoz41YEfC4RJXiCyaS tXhV8CjobGFDQLzRmWB2P9cqDgo4SmaUyM/JHShoMVPzdjr7y6MUXpsOMq5YFICCXhlZ qVbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mbB7jKMgu1mG/LYryAL3c2j/fz/WLPMVJBJA6CzcPqo=; b=OrUFvTzSIZefXFLgfc9PvbfxbbvPmdgbIDLLsQKGajTxe5j3LQ2vyYtiFcdGJy7SQp MqGOm1yLQeOIObr4PRDsLQ51HuWn3UEFacXuG3uGi7Oe8CKcHVzKBUj9zrc++32kjMD3 OThTbaYYpwQTbpSf1cdjdHkuI4PlsUN7NAXfKNRXnQEEobiU5XTgDN8MAobF9eEiganl 9wuhELbQMqL0Z17rMLYLVZaVwAqQIaGb6si339Ztp/pMkr5ooqfyqrfdZeteDKxfKx8U gO/zF7CbT9kCrbetxrikKIq6MMGgLeqrljONWnnutLj/F1deAvkQJCzuvAYFTFKG/N1/ YaEw== X-Gm-Message-State: AOUpUlEOCL06Z/8z8hoCle9II1Ud3a9MLfD2Zcb/v1D+cbnkPy4ybDdl 24tv6ZJLoZKDlQ0PR+43gFo= X-Google-Smtp-Source: AAOMgpdJC3hw7HmJaYrWqpfjhqJ0+g5JVHRmMnGOAoMLBTkBle1zkSDhW/NysRicfgH978I1/36eFA== X-Received: by 2002:a63:5542:: with SMTP id f2-v6mr14710114pgm.37.1533562200275; Mon, 06 Aug 2018 06:30:00 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.29.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:29:59 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:35 +0800 Message-Id: <1533562177-16447-10-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::52a Subject: [Qemu-devel] [PATCH v6 09/11] migration: poll the cm event for destination qemu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The destination qemu only poll the comp_channel->fd in qemu_rdma_wait_comp_channel. But when source qemu disconnnect the rdma connection, the destination qemu should be notified. Signed-off-by: Lidong Chen Reviewed-by: Dr. David Alan Gilbert --- migration/migration.c | 3 ++- migration/rdma.c | 32 +++++++++++++++++++++++++++++++- 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index df0c2cf..f7d6e26 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -389,6 +389,7 @@ static void process_incoming_migration_co(void *opaque) int ret; assert(mis->from_src_file); + mis->migration_incoming_co = qemu_coroutine_self(); mis->largest_page_size = qemu_ram_pagesize_largest(); postcopy_state_set(POSTCOPY_INCOMING_NONE); migrate_set_state(&mis->state, MIGRATION_STATUS_NONE, @@ -418,7 +419,6 @@ static void process_incoming_migration_co(void *opaque) /* we get COLO info, and know if we are in COLO mode */ if (!ret && migration_incoming_enable_colo()) { - mis->migration_incoming_co = qemu_coroutine_self(); qemu_thread_create(&mis->colo_incoming_thread, "COLO incoming", colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE); mis->have_colo_incoming_thread = true; @@ -442,6 +442,7 @@ static void process_incoming_migration_co(void *opaque) } mis->bh = qemu_bh_new(process_incoming_migration_bh, mis); qemu_bh_schedule(mis->bh); + mis->migration_incoming_co = NULL; } static void migration_incoming_setup(QEMUFile *f) diff --git a/migration/rdma.c b/migration/rdma.c index 1affc46..ae07515 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -3226,6 +3226,35 @@ err: static void rdma_accept_incoming_migration(void *opaque); +static void rdma_cm_poll_handler(void *opaque) +{ + RDMAContext *rdma = opaque; + int ret; + struct rdma_cm_event *cm_event; + MigrationIncomingState *mis = migration_incoming_get_current(); + + ret = rdma_get_cm_event(rdma->channel, &cm_event); + if (ret) { + error_report("get_cm_event failed %d", errno); + return; + } + rdma_ack_cm_event(cm_event); + + if (cm_event->event == RDMA_CM_EVENT_DISCONNECTED || + cm_event->event == RDMA_CM_EVENT_DEVICE_REMOVAL) { + error_report("receive cm event, cm event is %d", cm_event->event); + rdma->error_state = -EPIPE; + if (rdma->return_path) { + rdma->return_path->error_state = -EPIPE; + } + + if (mis->migration_incoming_co) { + qemu_coroutine_enter(mis->migration_incoming_co); + } + return; + } +} + static int qemu_rdma_accept(RDMAContext *rdma) { RDMACapabilities cap; @@ -3326,7 +3355,8 @@ static int qemu_rdma_accept(RDMAContext *rdma) NULL, (void *)(intptr_t)rdma->return_path); } else { - qemu_set_fd_handler(rdma->channel->fd, NULL, NULL, NULL); + qemu_set_fd_handler(rdma->channel->fd, rdma_cm_poll_handler, + NULL, rdma); } ret = rdma_accept(rdma->cm_id, &conn_param); From patchwork Mon Aug 6 13:29:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953868 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="guz/MirT"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdq96DQfz9s0n for ; Mon, 6 Aug 2018 23:32:05 +1000 (AEST) Received: from localhost ([::1]:34121 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfcB-0004yN-FI for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:32:03 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49238) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfaH-0003Rl-12 for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfaE-0008Ja-PZ for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:05 -0400 Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]:38965) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfaE-0008JF-Iy for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:02 -0400 Received: by mail-pg1-x542.google.com with SMTP id a11-v6so6234614pgw.6 for ; Mon, 06 Aug 2018 06:30:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WDpE7Whx2w0dBjEFqd4GAu9cy4lWySzxy6UXPm/z6eY=; b=guz/MirTCDKgHst5RGll/pVMMuyE16X4yEyOc/h5LuqONqhNpTVhFwHjMyqzKtN18c JZRbilHqvWIanfGXW38OwTdi8zi1tno3e8pN2D2oHrgzKSgBH7Eob2mI/82y2BUXLrhj dgkjnv6SeMXAQvdbxWEBLNvBC91aLy7ZbpclNMvQpx4FvtgdDOSaoOSJkmK6VgS+yU2C 6t/8NsknVrH277ZY7gpXuca9jX+LMbzM/NiVBV4Vp++QF6NJoOtmgee8VT8kD5zrs/Rz 0wwGlN3DFCnA+fY9oto5ItGXx9IXsoeARmqmcBaiY1oeCuhJiLryfFfo72O+pCn5pFN6 Kfqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WDpE7Whx2w0dBjEFqd4GAu9cy4lWySzxy6UXPm/z6eY=; b=LeUjM/8L92GVMMG7bW+YPD4uD/GLNP4R68btW3+N56qY2W9eCMW1bibOkKJM/AbYiI 0dNiRh05ES3EbcM99oq5zhuWuWStYbf+GSg/BAxZk8Qh6jWU16GZnOq3Ux9klc5T+J/W R1Iu8ZqHdeGegCl+rlY3Y1qLLyvkUrKSkymPYFt3c2xgWLAOLiuKndRSr6X8mHpdzXG8 teXlkm5R5o4rtOzhLX/C5fD/Ou62GQHafTeRId2Gp/VE07ytjHUMVNUa3j4N7r6Rrt2V F6CDFAtJlkvjgslfYMvtDv5pVu6tUS9y8MuDkrY+5xpQ4V+XY1hOSpR3XQ7GFqez08nt VE3g== X-Gm-Message-State: AOUpUlHAg8YaIuJIZNFKsu0Jcs1aHLNGhHeGz3quX/d7ub0mIdXgXoY2 JhbfllXLmvdOlWO07Nl1Ql0= X-Google-Smtp-Source: AAOMgpcF5zT3kxb86VrFTSNuDcjjiAuucLp7mQYGbj03j7UegpdTH87M5Aw6KSmYYBk/3cBaKzrg6g== X-Received: by 2002:a63:1644:: with SMTP id 4-v6mr14873284pgw.103.1533562201877; Mon, 06 Aug 2018 06:30:01 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.30.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:30:01 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:36 +0800 Message-Id: <1533562177-16447-11-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::542 Subject: [Qemu-devel] [PATCH v6 10/11] migration: remove the unnecessary RDMA_CONTROL_ERROR message X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" It's not necessary to send RDMA_CONTROL_ERROR when clean up rdma resource. If rdma->error_state is ture, the message may not send successfully. and the cm event can also notify the peer qemu. Signed-off-by: Lidong Chen --- migration/rdma.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/migration/rdma.c b/migration/rdma.c index ae07515..e1498f2 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2305,17 +2305,6 @@ static void qemu_rdma_cleanup(RDMAContext *rdma) int idx; if (rdma->cm_id && rdma->connected) { - if ((rdma->error_state || - migrate_get_current()->state == MIGRATION_STATUS_CANCELLING) && - !rdma->received_error) { - RDMAControlHeader head = { .len = 0, - .type = RDMA_CONTROL_ERROR, - .repeat = 1, - }; - error_report("Early error. Sending error."); - qemu_rdma_post_send_control(rdma, NULL, &head); - } - rdma_disconnect(rdma->cm_id); trace_qemu_rdma_cleanup_disconnect(); rdma->connected = false; From patchwork Mon Aug 6 13:29:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: 858585 jemmy X-Patchwork-Id: 953876 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OJmknSzm"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41kdz06Gn7z9s0R for ; Mon, 6 Aug 2018 23:38:52 +1000 (AEST) Received: from localhost ([::1]:34162 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfik-0001kx-J0 for incoming@patchwork.ozlabs.org; Mon, 06 Aug 2018 09:38:50 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49246) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fmfaH-0003SC-Er for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:06 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fmfaG-0008L4-GZ for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:05 -0400 Received: from mail-pf1-x441.google.com ([2607:f8b0:4864:20::441]:33210) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fmfaG-0008Ku-9A for qemu-devel@nongnu.org; Mon, 06 Aug 2018 09:30:04 -0400 Received: by mail-pf1-x441.google.com with SMTP id d4-v6so6891253pfn.0 for ; Mon, 06 Aug 2018 06:30:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=itMHUa+JDC7eH0BikBkcGSlr7LwmQHk0uMms6ma4D0U=; b=OJmknSzmrrEOkPS6MRxsbcLFDxPV8aKykObVEEUqwC+OfFUSt7xsdIq4jIlhxjPyfQ 7AurmiUvl300BOGjHDwNX27nZFzRu40247FHCBFX2xBDeMNF3kU+HZZ1q3BRhoJjSRk9 fiGonO3jWbDzTWJFGzJisAOc0K3SGqdJIu6GZZbbnDlqzACXNFS3W7+SJu7+Xf4Ib1Sl fOZadI/voOWu6+Lid8gSUf7LXr+hPjyrqiy4OaX8Omiisw/Q3MMp00rvvwvTRK8sO0wd Efpz7r+/PqWxMT83wjEv9yyVwGXPdv0hIwQ0AP1lSUjxWp0z5toPz7/fGcmjkgAiG3gv davg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=itMHUa+JDC7eH0BikBkcGSlr7LwmQHk0uMms6ma4D0U=; b=L1S/z+SkOWbcy8T2dHrPJjz04stWMuDdJogFuETUUw1UbBLTP8PwbdbPxRUHRG2Di+ RCa36xKUoDfF1UEU/hFTioovpouE1yhVD44GE2fabhErMaGlq6Rmrz5ADO1JHtBzBYLS yl0O5cdUsg34WdESKm/O+WBxsQKQvhxPIgGs1pXsmfIg6aZoO1Jmd2AB/F//N8w/n6H7 zpnDlQgG7MAhap86ctoc84mKUmNV1uOAjR8/xTuTHbkQJ5fP5s8LtL2+HtV9HaBzPph+ vPE/02MuLeX5pwBLpo+10GYw/qUredD0a9w8TWMRCigCDILgRIZFkT0aU4Uc9VoSFezD YDzg== X-Gm-Message-State: AOUpUlHwJjb3gcJSGvQvO9hRx/Wowmfn+3E9OrkM/L91sJ6Hy4j7qKMR yYUVS6m8u1RlKi+I1lWhWE9voTgUI1w= X-Google-Smtp-Source: AAOMgpd/A6Jvi0XYsSdWsPDTGBI/ZgmeaPWPheqQDd2rZ+HBj4pdRS1r7XA0lmEEZTnkrh7Iq0ap1g== X-Received: by 2002:a63:5660:: with SMTP id g32-v6mr14722719pgm.227.1533562203452; Mon, 06 Aug 2018 06:30:03 -0700 (PDT) Received: from VM_120_46_centos.localdomain ([119.28.87.64]) by smtp.gmail.com with ESMTPSA id f11-v6sm11756302pfa.131.2018.08.06.06.30.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Aug 2018 06:30:03 -0700 (PDT) From: Lidong Chen X-Google-Original-From: Lidong Chen To: zhang.zhanghailiang@huawei.com, quintela@redhat.com, dgilbert@redhat.com Date: Mon, 6 Aug 2018 21:29:37 +0800 Message-Id: <1533562177-16447-12-git-send-email-lidongchen@tencent.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> References: <1533562177-16447-1-git-send-email-lidongchen@tencent.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::441 Subject: [Qemu-devel] [PATCH v6 11/11] migration: create a dedicated thread to release rdma resource X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: qemu-devel@nongnu.org, Lidong Chen Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" ibv_dereg_mr wait for a long time for big memory size virtual server. The test result is: 10GB 326ms 20GB 699ms 30GB 1021ms 40GB 1387ms 50GB 1712ms 60GB 2034ms 70GB 2457ms 80GB 2807ms 90GB 3107ms 100GB 3474ms 110GB 3735ms 120GB 4064ms 130GB 4567ms 140GB 4886ms this will cause the guest os hang for a while when migration finished. So create a dedicated thread to release rdma resource. Signed-off-by: Lidong Chen --- migration/migration.c | 6 ++++++ migration/migration.h | 3 +++ migration/rdma.c | 47 +++++++++++++++++++++++++++++++---------------- 3 files changed, 40 insertions(+), 16 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index f7d6e26..25d9009 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1499,6 +1499,7 @@ void migrate_init(MigrationState *s) s->vm_was_running = false; s->iteration_initial_bytes = 0; s->threshold_size = 0; + s->rdma_cleanup_thread_quit = true; } static GSList *migration_blockers; @@ -1660,6 +1661,10 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc, return false; } + if (s->rdma_cleanup_thread_quit != true) { + return false; + } + if (runstate_check(RUN_STATE_INMIGRATE)) { error_setg(errp, "Guest is waiting for an incoming migration"); return false; @@ -3213,6 +3218,7 @@ static void migration_instance_init(Object *obj) ms->state = MIGRATION_STATUS_NONE; ms->mbps = -1; + ms->rdma_cleanup_thread_quit = true; qemu_sem_init(&ms->pause_sem, 0); qemu_mutex_init(&ms->error_mutex); diff --git a/migration/migration.h b/migration/migration.h index 64a7b33..60138dd 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -224,6 +224,9 @@ struct MigrationState * do not trigger spurious decompression errors. */ bool decompress_error_check; + + /* Set this when rdma resource have released */ + bool rdma_cleanup_thread_quit; }; void migrate_set_state(int *state, int old_state, int new_state); diff --git a/migration/rdma.c b/migration/rdma.c index e1498f2..3282f35 100644 --- a/migration/rdma.c +++ b/migration/rdma.c @@ -2995,35 +2995,50 @@ static void qio_channel_rdma_set_aio_fd_handler(QIOChannel *ioc, } } -static int qio_channel_rdma_close(QIOChannel *ioc, - Error **errp) +static void *qio_channel_rdma_close_thread(void *arg) { - QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); - RDMAContext *rdmain, *rdmaout; - trace_qemu_rdma_close(); - - rdmain = rioc->rdmain; - if (rdmain) { - atomic_rcu_set(&rioc->rdmain, NULL); - } + RDMAContext **rdma = arg; + RDMAContext *rdmain = rdma[0]; + RDMAContext *rdmaout = rdma[1]; + MigrationState *s = migrate_get_current(); - rdmaout = rioc->rdmaout; - if (rdmaout) { - atomic_rcu_set(&rioc->rdmaout, NULL); - } + rcu_register_thread(); synchronize_rcu(); - if (rdmain) { qemu_rdma_cleanup(rdmain); } - if (rdmaout) { qemu_rdma_cleanup(rdmaout); } g_free(rdmain); g_free(rdmaout); + g_free(rdma); + + rcu_unregister_thread(); + s->rdma_cleanup_thread_quit = true; + return NULL; +} + +static int qio_channel_rdma_close(QIOChannel *ioc, + Error **errp) +{ + QemuThread t; + QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(ioc); + RDMAContext **rdma = g_new0(RDMAContext*, 2); + MigrationState *s = migrate_get_current(); + + trace_qemu_rdma_close(); + if (rioc->rdmain || rioc->rdmaout) { + rdma[0] = rioc->rdmain; + rdma[1] = rioc->rdmaout; + atomic_rcu_set(&rioc->rdmain, NULL); + atomic_rcu_set(&rioc->rdmaout, NULL); + s->rdma_cleanup_thread_quit = false; + qemu_thread_create(&t, "rdma cleanup", qio_channel_rdma_close_thread, + rdma, QEMU_THREAD_DETACHED); + } return 0; }