From patchwork Mon Mar 11 10:29:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuval Shaia X-Patchwork-Id: 1054343 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.b="MSFz0pnR"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 44HvmS3gnHz9s00 for ; Mon, 11 Mar 2019 21:41:42 +1100 (AEDT) Received: from localhost ([127.0.0.1]:59290 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h3INI-00080I-Ca for incoming@patchwork.ozlabs.org; Mon, 11 Mar 2019 06:41:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:36347) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1h3IBR-0007Xn-RF for qemu-devel@nongnu.org; Mon, 11 Mar 2019 06:29:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1h3IBP-0000uR-HG for qemu-devel@nongnu.org; Mon, 11 Mar 2019 06:29:25 -0400 Received: from aserp2130.oracle.com ([141.146.126.79]:59596) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1h3IBP-0000tD-3C for qemu-devel@nongnu.org; Mon, 11 Mar 2019 06:29:23 -0400 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x2BASQe2186023; Mon, 11 Mar 2019 10:29:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=pu00MwWvy8JPtuqJZe2iIrrWEOr7asqGNvEzsWjhIqI=; b=MSFz0pnR7h7jpXC0Zg+lvU2HCuDg+Vdai7Df+2AUjs+SWyr6BztbsYtVRIslAf0P+GrR 6sb19UF/Oy6K/cgHVY8szV9AET+FSaZbdhlPHycCaR2KF1gOdwxpI/u9gv3a7xDeuu0o EfA6cEeQixdAhPhsNqY5yhK8AAMn3kquvkGYiKWNEh5bwTRPEAZraXbwJ4MwEla1wZrF J9DDRZvyLBSlZuS5TGmJY6EEWiGemr0YRsV+Y9lXMFmnxiu4AULgsqRufhXQmajL3nT2 uCY6Hh+5jRglfGqEC9czeWw5H0H8ywQAZGRZ/Z4J8VIIIm6Nj1GwixiqCia1AbWaNmHg 3A== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp2130.oracle.com with ESMTP id 2r430ee095-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 11 Mar 2019 10:29:19 +0000 Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x2BATJjK028360 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 11 Mar 2019 10:29:19 GMT Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x2BATIqc021967; Mon, 11 Mar 2019 10:29:18 GMT Received: from ca-dev66.localdomain (/10.68.48.69) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 11 Mar 2019 03:29:18 -0700 From: Yuval Shaia To: dgilbert@redhat.com, yuval.shaia@oracle.com, marcel.apfelbaum@gmail.com, qemu-devel@nongnu.org, armbru@redhat.com Date: Mon, 11 Mar 2019 03:29:06 -0700 Message-Id: <1552300155-25216-3-git-send-email-yuval.shaia@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1552300155-25216-1-git-send-email-yuval.shaia@oracle.com> References: <1552300155-25216-1-git-send-email-yuval.shaia@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9191 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1903110079 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] X-Received-From: 141.146.126.79 Subject: [Qemu-devel] [PATCH v6 02/11] hw/rdma: Introduce protected qlist X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" To make code more readable move handling of protected list to a rdma_utils Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Reviewed-by: Kamal Heib --- hw/rdma/rdma_backend.c | 20 +++++--------------- hw/rdma/rdma_backend_defs.h | 8 ++------ hw/rdma/rdma_utils.c | 39 +++++++++++++++++++++++++++++++++++++++ hw/rdma/rdma_utils.h | 9 +++++++++ 4 files changed, 55 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d138591..37edf42 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -527,9 +527,7 @@ static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, bctx->up_ctx = ctx; bctx->sge = *sge; - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + rdma_protected_qlist_append_int64(&backend_dev->recv_mads_list, bctx_id); return 0; } @@ -913,23 +911,19 @@ static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, static void process_incoming_mad_req(RdmaBackendDev *backend_dev, RdmaCmMuxMsg *msg) { - QObject *o_ctx_id; unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; trace_mad_message("recv", msg->umad.mad, msg->umad_len); - qemu_mutex_lock(&backend_dev->recv_mads_list.lock); - o_ctx_id = qlist_pop(backend_dev->recv_mads_list.list); - qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); - if (!o_ctx_id) { + cqe_ctx_id = rdma_protected_qlist_pop_int64(&backend_dev->recv_mads_list); + if (cqe_ctx_id == -ENOENT) { rdma_warn_report("No more free MADs buffers, waiting for a while"); sleep(THR_POLL_TO); return; } - cqe_ctx_id = qnum_get_uint(qobject_to(QNum, o_ctx_id)); bctx = rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); if (unlikely(!bctx)) { rdma_error_report("No matching ctx for req %ld", cqe_ctx_id); @@ -994,8 +988,7 @@ static int mad_init(RdmaBackendDev *backend_dev, CharBackend *mad_chr_be) return -EIO; } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); - backend_dev->recv_mads_list.list = qlist_new(); + rdma_protected_qlist_init(&backend_dev->recv_mads_list); enable_rdmacm_mux_async(backend_dev); @@ -1010,10 +1003,7 @@ static void mad_fini(RdmaBackendDev *backend_dev) { disable_rdmacm_mux_async(backend_dev); qemu_chr_fe_disconnect(backend_dev->rdmacm_mux.chr_be); - if (backend_dev->recv_mads_list.list) { - qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); - qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); - } + rdma_protected_qlist_destroy(&backend_dev->recv_mads_list); } int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 15ae8b9..a8c15b0 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -20,6 +20,7 @@ #include "chardev/char-fe.h" #include #include "contrib/rdmacm-mux/rdmacm-mux.h" +#include "rdma_utils.h" typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -30,11 +31,6 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; -typedef struct RecvMadList { - QemuMutex lock; - QList *list; -} RecvMadList; - typedef struct RdmaCmMux { CharBackend *chr_be; int can_receive; @@ -48,7 +44,7 @@ typedef struct RdmaBackendDev { struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - RecvMadList recv_mads_list; + RdmaProtectedQList recv_mads_list; RdmaCmMux rdmacm_mux; } RdmaBackendDev; diff --git a/hw/rdma/rdma_utils.c b/hw/rdma/rdma_utils.c index b9f07fc..0a8abe5 100644 --- a/hw/rdma/rdma_utils.c +++ b/hw/rdma/rdma_utils.c @@ -14,6 +14,8 @@ */ #include "qemu/osdep.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" #include "trace.h" #include "rdma_utils.h" @@ -51,3 +53,40 @@ void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len) pci_dma_unmap(dev, buffer, len, DMA_DIRECTION_TO_DEVICE, 0); } } + +void rdma_protected_qlist_init(RdmaProtectedQList *list) +{ + qemu_mutex_init(&list->lock); + list->list = qlist_new(); +} + +void rdma_protected_qlist_destroy(RdmaProtectedQList *list) +{ + if (list->list) { + qlist_destroy_obj(QOBJECT(list->list)); + qemu_mutex_destroy(&list->lock); + list->list = NULL; + } +} + +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value) +{ + qemu_mutex_lock(&list->lock); + qlist_append_int(list->list, value); + qemu_mutex_unlock(&list->lock); +} + +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list) +{ + QObject *obj; + + qemu_mutex_lock(&list->lock); + obj = qlist_pop(list->list); + qemu_mutex_unlock(&list->lock); + + if (!obj) { + return -ENOENT; + } + + return qnum_get_uint(qobject_to(QNum, obj)); +} diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index acd1488..a8bf1d4 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -29,8 +29,17 @@ #define rdma_info_report(fmt, ...) \ info_report("%s: " fmt, "rdma", ## __VA_ARGS__) +typedef struct RdmaProtectedQList { + QemuMutex lock; + QList *list; +} RdmaProtectedQList; + void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); +void rdma_protected_qlist_init(RdmaProtectedQList *list); +void rdma_protected_qlist_destroy(RdmaProtectedQList *list); +void rdma_protected_qlist_append_int64(RdmaProtectedQList *list, int64_t value); +int64_t rdma_protected_qlist_pop_int64(RdmaProtectedQList *list); static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) {