From patchwork Thu Jan 31 13:08:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuval Shaia X-Patchwork-Id: 1034194 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.b="jzKd3aWT"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43r10m14T9z9sDB for ; Fri, 1 Feb 2019 00:14:32 +1100 (AEDT) Received: from localhost ([127.0.0.1]:54580 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gpCAo-0002jC-4r for incoming@patchwork.ozlabs.org; Thu, 31 Jan 2019 08:14:30 -0500 Received: from eggs.gnu.org ([209.51.188.92]:48074) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gpC5b-00076n-1M for qemu-devel@nongnu.org; Thu, 31 Jan 2019 08:09:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gpC5Z-0000HO-W3 for qemu-devel@nongnu.org; Thu, 31 Jan 2019 08:09:07 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:53816) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gpC5Z-0000H8-OH for qemu-devel@nongnu.org; Thu, 31 Jan 2019 08:09:05 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id x0VD4BT7164992; Thu, 31 Jan 2019 13:09:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=ZDaGt8Jm1RccJffYUjL4UMx8RohYX54nfkp7TUV3sdQ=; b=jzKd3aWTXGhBl8OSI4VdJS3/HirNM7EHCtKr27AQ67TKtWdBKq4bXo9zBhu69GcwMXnR BcP+54U2R63dVVCaYm1LWNgyyHOHBAdMZZBWreozXRq/lqUkueazcBdco1//wXtXLsDJ C/GC7YjGJyr4nRPsdI2FqDEPZxlfX4fBfUIdLJ737qfm2hZV5WWWLUkyHmdQxBqqgbMT PEQEGcVFKLA5jcRS57XQmp2LauXYqfUXHJ++UI6XLDXyZ2MmVxGN2y0PPJsQcuz6R3F0 FJYRhKRvGynPnX7IcVn/d/JpTcYwlETa0lpUs+PVcJQQ0vd4sWGAa6R5o+vOFWFqyjEj Ig== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2130.oracle.com with ESMTP id 2q8eyurm1s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 31 Jan 2019 13:09:03 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x0VD93pa031920 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 31 Jan 2019 13:09:03 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x0VD93oI027592; Thu, 31 Jan 2019 13:09:03 GMT Received: from localhost.localdomain (/77.138.186.148) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 31 Jan 2019 05:09:02 -0800 From: Yuval Shaia To: dgilbert@redhat.com, yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, armbru@redhat.com, qemu-devel@nongnu.org Date: Thu, 31 Jan 2019 15:08:42 +0200 Message-Id: <20190131130850.6850-3-yuval.shaia@oracle.com> X-Mailer: git-send-email 2.17.2 In-Reply-To: <20190131130850.6850-1-yuval.shaia@oracle.com> References: <20190131130850.6850-1-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9152 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1901310103 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 156.151.31.86 Subject: [Qemu-devel] [PATCH 02/10] hw/rdma: Introduce locked qlist X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" To make code more readable move handling of locked list to a generic functions. Signed-off-by: Yuval Shaia --- hw/rdma/rdma_backend.c | 20 +++++-------------- hw/rdma/rdma_backend_defs.h | 8 ++------ hw/rdma/rdma_utils.c | 39 +++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_utils.h | 9 +++++++++ 4 files changed, 55 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 5f60856d19..2f6372f8f0 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -527,9 +527,7 @@ static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, bctx->up_ctx = ctx; bctx->sge = *sge; - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + rdma_locked_list_append_int64(&backend_dev->recv_mads_list, bctx_id); return 0; } @@ -913,23 +911,19 @@ static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, static void process_incoming_mad_req(RdmaBackendDev *backend_dev, RdmaCmMuxMsg *msg) { - QObject *o_ctx_id; unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; trace_mad_message("recv", msg->umad.mad, msg->umad_len); - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - o_ctx_id = qlist_pop(backend_dev->recv_mads_list.list); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); - if (!o_ctx_id) { + cqe_ctx_id = rdma_locked_list_pop_int64(&backend_dev->recv_mads_list); + if (cqe_ctx_id == -ENOENT) { rdma_warn_report("No more free MADs buffers, waiting for a while"); sleep(THR_POLL_TO); return; } - cqe_ctx_id = qnum_get_uint(qobject_to(QNum, o_ctx_id)); bctx = rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); if (unlikely(!bctx)) { rdma_error_report("No matching ctx for req %ld", cqe_ctx_id); @@ -994,8 +988,7 @@ static int mad_init(RdmaBackendDev *backend_dev, CharBackend *mad_chr_be) return -EIO; } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); - backend_dev->recv_mads_list.list = qlist_new(); + rdma_locked_list_init(&backend_dev->recv_mads_list); enable_rdmacm_mux_async(backend_dev); @@ -1010,10 +1003,7 @@ static void mad_fini(RdmaBackendDev *backend_dev) { disable_rdmacm_mux_async(backend_dev); qemu_chr_fe_disconnect(backend_dev->rdmacm_mux.chr_be); - if (backend_dev->recv_mads_list.list) { - qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); - qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); - } + rdma_locked_list_destroy(&backend_dev->recv_mads_list); } int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 15ae8b970e..bec0457f25 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -20,6 +20,7 @@ #include "chardev/char-fe.h" #include #include "contrib/rdmacm-mux/rdmacm-mux.h" +#include "rdma_utils.h" typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -30,11 +31,6 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; -typedef struct RecvMadList { - QemuMutex lock; - QList *list; -} RecvMadList; - typedef struct RdmaCmMux { CharBackend *chr_be; int can_receive; @@ -48,7 +44,7 @@ typedef struct RdmaBackendDev { struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - RecvMadList recv_mads_list; + LockedList recv_mads_list; RdmaCmMux rdmacm_mux; } RdmaBackendDev; diff --git a/hw/rdma/rdma_utils.c b/hw/rdma/rdma_utils.c index f1c980c6be..a2a4ea2a15 100644 --- a/hw/rdma/rdma_utils.c +++ b/hw/rdma/rdma_utils.c @@ -14,6 +14,8 @@ */ #include "qemu/osdep.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" #include "trace.h" #include "rdma_utils.h" @@ -55,3 +57,40 @@ void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len) pci_dma_unmap(dev, buffer, len, DMA_DIRECTION_TO_DEVICE, 0); } } + +void rdma_locked_list_init(LockedList *list) +{ + qemu_mutex_init(&list->lock); + list->list = qlist_new(); +} + +void rdma_locked_list_destroy(LockedList *list) +{ + if (list->list) { + qlist_destroy_obj(QOBJECT(list->list)); + qemu_mutex_destroy(&list->lock); + list->list = NULL; + } +} + +void rdma_locked_list_append_int64(LockedList *list, int64_t value) +{ + qemu_mutex_lock(&list->lock); + qlist_append_int(list->list, value); + qemu_mutex_unlock(&list->lock); +} + +int64_t rdma_locked_list_pop_int64(LockedList *list) +{ + QObject *obj; + + qemu_mutex_lock(&list->lock); + obj = qlist_pop(list->list); + qemu_mutex_unlock(&list->lock); + + if (!obj) { + return -ENOENT; + } + + return qnum_get_uint(qobject_to(QNum, obj)); +} diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index acd148837f..4ba9956f81 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -29,8 +29,17 @@ #define rdma_info_report(fmt, ...) \ info_report("%s: " fmt, "rdma", ## __VA_ARGS__) +typedef struct LockedList { + QemuMutex lock; + QList *list; +} LockedList; + void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); +void rdma_locked_list_init(LockedList *list); +void rdma_locked_list_destroy(LockedList *list); +void rdma_locked_list_append_int64(LockedList *list, int64_t value); +int64_t rdma_locked_list_pop_int64(LockedList *list); static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) {