From patchwork Sat Dec 22 09:50:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017801 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ZCpS5e17"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLP85f7wz9sDP for ; Sat, 22 Dec 2018 20:51:39 +1100 (AEDT) Received: from localhost ([::1]:36915 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwV-000694-IR for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:51:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41513) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvl-00068o-46 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvh-0006Gq-D9 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:47 -0500 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:35989) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvf-0006EK-EC for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:45 -0500 Received: by mail-wr1-x444.google.com with SMTP id u4so7504234wrp.3 for ; Sat, 22 Dec 2018 01:50:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5dVeTlzdJ2AeoFg6ycMVcl5pJNQT/zzvsAlsV3kRw0c=; b=ZCpS5e17KwabCelouqJltx2gGYSflRbmOLgUeOBton0NxaPrN/LLCPUdAQDWInLMWq Pld1scsIS+jAI2v83IjG2rNuwoODM6K/M8TNMuqr2DUDepAIOZON+31koKaF3VEOTJ+w /rqS1nV8x/O78mnNqUpGcvamTWvDQrJ4GNTuuMKLuXYK+rBACCzuCQa+CqXgfbvk31q2 iZBaNFl81FwVwvXhg+Cc6SJXkD3bmPhdbKzn+UqSaYyoGj/tUvfINroiJtye/KdnEo/b fW3gkxkDnOVEdcBOkBPAjQyjjkxw3ySVycFDTpVQBdy5OfTdscJrfMOAFy3rTAV3v58Q alUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5dVeTlzdJ2AeoFg6ycMVcl5pJNQT/zzvsAlsV3kRw0c=; b=DV3IcUd110JWQwBAfpy3Sp+1slxXA5EqRn5rg4PybJImIT57fze0WA1AcQSyGQW+Zs ByAdkV+6tRJy0JKylqgpf3+0AOQEwjNKexV6ZWIAyMoXLZXPDS++jbLV/h289eppFFv/ olDKHP8vsrWrB+QItonYNT4qCbF3zmzrlnSgZCQngMXMB/DAsJCgGSyTT+Hl9HxAmj3d OB2Vi/Qbd5cWp8DCXws7JyUVQOatC4k2eJHcaPzJ3+nKcKKs3tS3IagmRHCrpOgB16eu DNFgBmBqKRCUwYOsOak6CQSo1BtkWlXNLiygIT9WxVH4sGf3Gcgh+g0lLU6HGysznYHa VpyQ== X-Gm-Message-State: AJcUukeZY7gNJEc6w4FQTSK3y2tqrbwotKeVxD4B0xfXfnjxeVKEUJkG dLtQSHGH7XVWCbQJfTNLfbxx/Hmc X-Google-Smtp-Source: ALg8bN4IBnbop2kj2CikJeU71sdHOo/AB4VGe7NS1qNTjCBz4sghbdEQ68iXFaAxtWcxwoEnTALTKw== X-Received: by 2002:adf:f390:: with SMTP id m16mr5521470wro.71.1545472240966; Sat, 22 Dec 2018 01:50:40 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:40 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:06 +0200 Message-Id: <20181222095036.29743-2-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::444 Subject: [Qemu-devel] [PATCH PULL 01/31] hw/pvrdma: Check the correct return value X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Return value of 0 means ok, we want to free the memory only in case of error. Signed-off-by: Yuval Shaia Message-Id: <20181025061700.17050-1-yuval.shaia@oracle.com> Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 4faeb21631..57d6f41ae6 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -232,7 +232,7 @@ static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, cmd->start, cmd->length, host_virt, cmd->access_flags, &resp->mr_handle, &resp->lkey, &resp->rkey); - if (host_virt && !resp->hdr.err) { + if (resp->hdr.err && host_virt) { munmap(host_virt, cmd->length); } From patchwork Sat Dec 22 09:50:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017803 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tncX9sRJ"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLPL6BvPz9sLw for ; Sat, 22 Dec 2018 20:51:50 +1100 (AEDT) Received: from localhost ([::1]:37081 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwi-0006Gm-D3 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:51:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41557) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvr-00069r-3D for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvm-0006KG-Qu for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:37135) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvj-0006G2-9t for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:49 -0500 Received: by mail-wr1-x444.google.com with SMTP id s12so7508258wrt.4 for ; Sat, 22 Dec 2018 01:50:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sGKiH25rWK/SAQ6nWwlIY/tJTrZSKJNRqR/gpg18WXM=; b=tncX9sRJ39Ph8rrnzihTBWVH7tUNTmEuKLu+mboV2JA0RiUdB/tMujuFw5RwZkRAA/ +Llp+KITXOJTCmB9ibiMseUUMksk4ZJFxzJ2t2DeQW4alrzo+u+q0SSHhsqMJrarnPgC dLuzH4V3pwlYLGcDx280p3KgoCSdBirCVrdodB4IqqdGYPpksYUSV1/CnDNi7PoOrxmV N+QKEpHx5RuE8X+ACnyfTmIvmP/7SLWWw56Pv2hHABYvoMggu7lEWddSEIL13hKCnKS3 kLMvqsNQ1fMKIHk8I073qyHLzh8aHf1KgRTbIPVNVlmRR2UHiYirMExTXmffYXNAklrF 5aQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sGKiH25rWK/SAQ6nWwlIY/tJTrZSKJNRqR/gpg18WXM=; b=m/z5vgEUGV2eiKfMZV8B61ZNWOJjUr6221/YzjayCdU85VhQn3JyJ7BTJuWa/WMeVq byvkuFy+S4NVACOMxmhwQ5SRMd38usCghh6LTPD588SrAWo4/24ZRKxOUPltZ6fe2MCY nLaSaoWOe+gMQerkqSz6nkVupDA1Iy/fmdY4QYzGqcDA52zefZUfIuz/cqMCvkovq0rV neRl8sH0ubpAgx55Q/AFndxU3fkrwJs29A76y/RpV177L4mGaHkwwcMz6NizTdPA2J7F UF/Yf1WHDC6qx75GlTrmvazsZ7L5yfoC11o8QE+soeI7YGm+bZiM2uCWfOEBJwDhzHS1 njCQ== X-Gm-Message-State: AJcUukcL9KgHNop4sfFxUAf1WT0vjwW65xpqp8IYrsG0l+lcH0QjW2vx AVsPIo3GL3zD/8sivc2pekw2KFxy X-Google-Smtp-Source: ALg8bN5++AdPo8/z3b5taErLVQxLnAecfkvvXgJ6pHWYOjUp8Rllcbtnpp3KZ+A4PAAO5y2V8CWSJw== X-Received: by 2002:adf:f605:: with SMTP id t5mr5409299wrp.229.1545472242557; Sat, 22 Dec 2018 01:50:42 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.41 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:41 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:07 +0200 Message-Id: <20181222095036.29743-3-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::444 Subject: [Qemu-devel] [PATCH PULL 02/31] contrib/rdmacm-mux: Add implementation of RDMA User MAD multiplexer X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia RDMA MAD kernel module (ibcm) disallow more than one MAD-agent for a given MAD class. This does not go hand-by-hand with qemu pvrdma device's requirements where each VM is MAD agent. Fix it by adding implementation of RDMA MAD multiplexer service which on one hand register as a sole MAD agent with the kernel module and on the other hand gives service to more than one VM. Design Overview: Reviewed-by: Shamir Rabinovitch ---------------- A server process is registered to UMAD framework (for this to work the rdma_cm kernel module needs to be unloaded) and creates a unix socket to listen to incoming request from clients. A client process (such as QEMU) connects to this unix socket and registers with its own GID. TX: ---- When client needs to send rdma_cm MAD message it construct it the same way as without this multiplexer, i.e. creates a umad packet but this time it writes its content to the socket instead of calling umad_send(). The server, upon receiving such a message fetch local_comm_id from it so a context for this session can be maintain and relay the message to UMAD layer by calling umad_send(). RX: ---- The server creates a worker thread to process incoming rdma_cm MAD messages. When an incoming message arrived (umad_recv()) the server, depending on the message type (attr_id) looks for target client by either searching in gid->fd table or in local_comm_id->fd table. With the extracted fd the server relays to incoming message to the client. Signed-off-by: Yuval Shaia Reviewed-by: Shamir Rabinovitch Signed-off-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- MAINTAINERS | 1 + Makefile | 3 + Makefile.objs | 1 + contrib/rdmacm-mux/Makefile.objs | 4 + contrib/rdmacm-mux/main.c | 798 +++++++++++++++++++++++++++++++ contrib/rdmacm-mux/rdmacm-mux.h | 61 +++ 6 files changed, 868 insertions(+) create mode 100644 contrib/rdmacm-mux/Makefile.objs create mode 100644 contrib/rdmacm-mux/main.c create mode 100644 contrib/rdmacm-mux/rdmacm-mux.h diff --git a/MAINTAINERS b/MAINTAINERS index 3b31e07b26..856d379b0a 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2412,6 +2412,7 @@ S: Maintained F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt +F: contrib/rdmacm-mux/* Build and test automation ------------------------- diff --git a/Makefile b/Makefile index 038780c6d0..dd53965f77 100644 --- a/Makefile +++ b/Makefile @@ -362,6 +362,7 @@ dummy := $(call unnest-vars,, \ elf2dmp-obj-y \ ivshmem-client-obj-y \ ivshmem-server-obj-y \ + rdmacm-mux-obj-y \ libvhost-user-obj-y \ vhost-user-scsi-obj-y \ vhost-user-blk-obj-y \ @@ -579,6 +580,8 @@ vhost-user-scsi$(EXESUF): $(vhost-user-scsi-obj-y) libvhost-user.a $(call LINK, $^) vhost-user-blk$(EXESUF): $(vhost-user-blk-obj-y) libvhost-user.a $(call LINK, $^) +rdmacm-mux$(EXESUF): $(rdmacm-mux-obj-y) $(COMMON_LDADDS) + $(call LINK, $^) module_block.h: $(SRC_PATH)/scripts/modules/module_block.py config-host.mak $(call quiet-command,$(PYTHON) $< $@ \ diff --git a/Makefile.objs b/Makefile.objs index 56af0347d3..319f14d937 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -133,6 +133,7 @@ vhost-user-scsi.o-cflags := $(LIBISCSI_CFLAGS) vhost-user-scsi.o-libs := $(LIBISCSI_LIBS) vhost-user-scsi-obj-y = contrib/vhost-user-scsi/ vhost-user-blk-obj-y = contrib/vhost-user-blk/ +rdmacm-mux-obj-y = contrib/rdmacm-mux/ ###################################################################### trace-events-subdirs = diff --git a/contrib/rdmacm-mux/Makefile.objs b/contrib/rdmacm-mux/Makefile.objs new file mode 100644 index 0000000000..be3eacb6f7 --- /dev/null +++ b/contrib/rdmacm-mux/Makefile.objs @@ -0,0 +1,4 @@ +ifdef CONFIG_PVRDMA +CFLAGS += -libumad -Wno-format-truncation +rdmacm-mux-obj-y = main.o +endif diff --git a/contrib/rdmacm-mux/main.c b/contrib/rdmacm-mux/main.c new file mode 100644 index 0000000000..835a7f9214 --- /dev/null +++ b/contrib/rdmacm-mux/main.c @@ -0,0 +1,798 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux implementation + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include "sys/poll.h" +#include "sys/ioctl.h" +#include "pthread.h" +#include "syslog.h" + +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "infiniband/umad_types.h" +#include "infiniband/umad_sa.h" +#include "infiniband/umad_cm.h" + +#include "rdmacm-mux.h" + +#define SCALE_US 1000 +#define COMMID_TTL 2 /* How many SCALE_US a context of MAD session is saved */ +#define SLEEP_SECS 5 /* This is used both in poll() and thread */ +#define SERVER_LISTEN_BACKLOG 10 +#define MAX_CLIENTS 4096 +#define MAD_RMPP_VERSION 0 +#define MAD_METHOD_MASK0 0x8 + +#define IB_USER_MAD_LONGS_PER_METHOD_MASK (128 / (8 * sizeof(long))) + +#define CM_REQ_DGID_POS 80 +#define CM_SIDR_REQ_DGID_POS 44 + +/* The below can be override by command line parameter */ +#define UNIX_SOCKET_PATH "/var/run/rdmacm-mux" +#define RDMA_PORT_NUM 1 + +typedef struct RdmaCmServerArgs { + char unix_socket_path[PATH_MAX]; + char rdma_dev_name[NAME_MAX]; + int rdma_port_num; +} RdmaCMServerArgs; + +typedef struct CommId2FdEntry { + int fd; + int ttl; /* Initialized to 2, decrement each timeout, entry delete when 0 */ + __be64 gid_ifid; +} CommId2FdEntry; + +typedef struct RdmaCmUMadAgent { + int port_id; + int agent_id; + GHashTable *gid2fd; /* Used to find fd of a given gid */ + GHashTable *commid2fd; /* Used to find fd on of a given comm_id */ +} RdmaCmUMadAgent; + +typedef struct RdmaCmServer { + bool run; + RdmaCMServerArgs args; + struct pollfd fds[MAX_CLIENTS]; + int nfds; + RdmaCmUMadAgent umad_agent; + pthread_t umad_recv_thread; + pthread_rwlock_t lock; +} RdmaCMServer; + +static RdmaCMServer server = {0}; + +static void usage(const char *progname) +{ + printf("Usage: %s [OPTION]...\n" + "Start a RDMA-CM multiplexer\n" + "\n" + "\t-h Show this help\n" + "\t-d rdma-device-name Name of RDMA device to register with\n" + "\t-s unix-socket-path Path to unix socket to listen on (default %s)\n" + "\t-p rdma-device-port Port number of RDMA device to register with (default %d)\n", + progname, UNIX_SOCKET_PATH, RDMA_PORT_NUM); +} + +static void help(const char *progname) +{ + fprintf(stderr, "Try '%s -h' for more information.\n", progname); +} + +static void parse_args(int argc, char *argv[]) +{ + int c; + char unix_socket_path[PATH_MAX]; + + strcpy(server.args.rdma_dev_name, ""); + strcpy(unix_socket_path, UNIX_SOCKET_PATH); + server.args.rdma_port_num = RDMA_PORT_NUM; + + while ((c = getopt(argc, argv, "hs:d:p:")) != -1) { + switch (c) { + case 'h': + usage(argv[0]); + exit(0); + + case 'd': + strncpy(server.args.rdma_dev_name, optarg, NAME_MAX - 1); + break; + + case 's': + /* This is temporary, final name will build below */ + strncpy(unix_socket_path, optarg, PATH_MAX); + break; + + case 'p': + server.args.rdma_port_num = atoi(optarg); + break; + + default: + help(argv[0]); + exit(1); + } + } + + if (!strcmp(server.args.rdma_dev_name, "")) { + fprintf(stderr, "Missing RDMA device name\n"); + help(argv[0]); + exit(1); + } + + /* Build unique unix-socket file name */ + snprintf(server.args.unix_socket_path, PATH_MAX, "%s-%s-%d", + unix_socket_path, server.args.rdma_dev_name, + server.args.rdma_port_num); + + syslog(LOG_INFO, "unix_socket_path=%s", server.args.unix_socket_path); + syslog(LOG_INFO, "rdma-device-name=%s", server.args.rdma_dev_name); + syslog(LOG_INFO, "rdma-device-port=%d", server.args.rdma_port_num); +} + +static void hash_tbl_alloc(void) +{ + + server.umad_agent.gid2fd = g_hash_table_new_full(g_int64_hash, + g_int64_equal, + g_free, g_free); + server.umad_agent.commid2fd = g_hash_table_new_full(g_int_hash, + g_int_equal, + g_free, g_free); +} + +static void hash_tbl_free(void) +{ + if (server.umad_agent.commid2fd) { + g_hash_table_destroy(server.umad_agent.commid2fd); + } + if (server.umad_agent.gid2fd) { + g_hash_table_destroy(server.umad_agent.gid2fd); + } +} + + +static int _hash_tbl_search_fd_by_ifid(__be64 *gid_ifid) +{ + int *fd; + + fd = g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + if (!fd) { + /* Let's try IPv4 */ + *gid_ifid |= 0x00000000ffff0000; + fd = g_hash_table_lookup(server.umad_agent.gid2fd, gid_ifid); + } + + return fd ? *fd : 0; +} + +static int hash_tbl_search_fd_by_ifid(int *fd, __be64 *gid_ifid) +{ + pthread_rwlock_rdlock(&server.lock); + *fd = _hash_tbl_search_fd_by_ifid(gid_ifid); + pthread_rwlock_unlock(&server.lock); + + if (!fd) { + syslog(LOG_WARNING, "Can't find matching for ifid 0x%llx\n", *gid_ifid); + return -ENOENT; + } + + return 0; +} + +static int hash_tbl_search_fd_by_comm_id(uint32_t comm_id, int *fd, + __be64 *gid_idid) +{ + CommId2FdEntry *fde; + + pthread_rwlock_rdlock(&server.lock); + fde = g_hash_table_lookup(server.umad_agent.commid2fd, &comm_id); + pthread_rwlock_unlock(&server.lock); + + if (!fde) { + syslog(LOG_WARNING, "Can't find matching for comm_id 0x%x\n", comm_id); + return -ENOENT; + } + + *fd = fde->fd; + *gid_idid = fde->gid_ifid; + + return 0; +} + +static RdmaCmMuxErrCode add_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 = _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (fd1) { /* record already exist - an error */ + pthread_rwlock_unlock(&server.lock); + return fd == fd1 ? RDMACM_MUX_ERR_CODE_EEXIST : + RDMACM_MUX_ERR_CODE_EACCES; + } + + g_hash_table_insert(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid)), g_memdup(&fd, sizeof(fd))); + + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx registered on socket %d", + be64toh((uint64_t)gid_ifid), fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static RdmaCmMuxErrCode delete_fd_ifid_pair(int fd, __be64 gid_ifid) +{ + int fd1; + + pthread_rwlock_wrlock(&server.lock); + + fd1 = _hash_tbl_search_fd_by_ifid(&gid_ifid); + if (!fd1) { /* record not exist - an error */ + pthread_rwlock_unlock(&server.lock); + return RDMACM_MUX_ERR_CODE_ENOTFOUND; + } + + g_hash_table_remove(server.umad_agent.gid2fd, g_memdup(&gid_ifid, + sizeof(gid_ifid))); + pthread_rwlock_unlock(&server.lock); + + syslog(LOG_INFO, "0x%lx unregistered on socket %d", + be64toh((uint64_t)gid_ifid), fd); + + return RDMACM_MUX_ERR_CODE_OK; +} + +static void hash_tbl_save_fd_comm_id_pair(int fd, uint32_t comm_id, + uint64_t gid_ifid) +{ + CommId2FdEntry fde = {fd, COMMID_TTL, gid_ifid}; + + pthread_rwlock_wrlock(&server.lock); + g_hash_table_insert(server.umad_agent.commid2fd, + g_memdup(&comm_id, sizeof(comm_id)), + g_memdup(&fde, sizeof(fde))); + pthread_rwlock_unlock(&server.lock); +} + +static gboolean remove_old_comm_ids(gpointer key, gpointer value, + gpointer user_data) +{ + CommId2FdEntry *fde = (CommId2FdEntry *)value; + + return !fde->ttl--; +} + +static gboolean remove_entry_from_gid2fd(gpointer key, gpointer value, + gpointer user_data) +{ + if (*(int *)value == *(int *)user_data) { + syslog(LOG_INFO, "0x%lx unregistered on socket %d", + be64toh(*(uint64_t *)key), *(int *)value); + return true; + } + + return false; +} + +static void hash_tbl_remove_fd_ifid_pair(int fd) +{ + pthread_rwlock_wrlock(&server.lock); + g_hash_table_foreach_remove(server.umad_agent.gid2fd, + remove_entry_from_gid2fd, (gpointer)&fd); + pthread_rwlock_unlock(&server.lock); +} + +static int get_fd(const char *mad, int *fd, __be64 *gid_ifid) +{ + struct umad_hdr *hdr = (struct umad_hdr *)mad; + char *data = (char *)hdr + sizeof(*hdr); + int32_t comm_id = 0; + uint16_t attr_id = be16toh(hdr->attr_id); + int rc = 0; + + switch (attr_id) { + case UMAD_CM_ATTR_REQ: + memcpy(gid_ifid, data + CM_REQ_DGID_POS, sizeof(*gid_ifid)); + rc = hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_SIDR_REQ: + memcpy(gid_ifid, data + CM_SIDR_REQ_DGID_POS, sizeof(*gid_ifid)); + rc = hash_tbl_search_fd_by_ifid(fd, gid_ifid); + break; + + case UMAD_CM_ATTR_REP: + /* Fall through */ + case UMAD_CM_ATTR_REJ: + /* Fall through */ + case UMAD_CM_ATTR_DREQ: + /* Fall through */ + case UMAD_CM_ATTR_DREP: + /* Fall through */ + case UMAD_CM_ATTR_RTU: + data += sizeof(comm_id); + /* Fall through */ + case UMAD_CM_ATTR_SIDR_REP: + memcpy(&comm_id, data, sizeof(comm_id)); + if (comm_id) { + rc = hash_tbl_search_fd_by_comm_id(comm_id, fd, gid_ifid); + } + break; + + default: + rc = -EINVAL; + syslog(LOG_WARNING, "Unsupported attr_id 0x%x\n", attr_id); + } + + syslog(LOG_DEBUG, "mad_to_vm: %d 0x%x 0x%x\n", *fd, attr_id, comm_id); + + return rc; +} + +static void *umad_recv_thread_func(void *args) +{ + int rc; + RdmaCmMuxMsg msg = {0}; + int fd = -2; + + msg.hdr.msg_type = RDMACM_MUX_MSG_TYPE_REQ; + msg.hdr.op_code = RDMACM_MUX_OP_CODE_MAD; + + while (server.run) { + do { + msg.umad_len = sizeof(msg.umad.mad); + rc = umad_recv(server.umad_agent.port_id, &msg.umad, &msg.umad_len, + SLEEP_SECS * SCALE_US); + if ((rc == -EIO) || (rc == -EINVAL)) { + syslog(LOG_CRIT, "Fatal error while trying to read MAD"); + } + + if (rc == -ETIMEDOUT) { + g_hash_table_foreach_remove(server.umad_agent.commid2fd, + remove_old_comm_ids, NULL); + } + } while (rc && server.run); + + if (server.run) { + rc = get_fd(msg.umad.mad, &fd, &msg.hdr.sgid.global.interface_id); + if (rc) { + continue; + } + + send(fd, &msg, sizeof(msg), 0); + } + } + + return NULL; +} + +static int read_and_process(int fd) +{ + int rc; + RdmaCmMuxMsg msg = {0}; + struct umad_hdr *hdr; + uint32_t *comm_id = 0; + uint16_t attr_id; + + rc = recv(fd, &msg, sizeof(msg), 0); + syslog(LOG_DEBUG, "Socket %d, recv %d\n", fd, rc); + + if (rc < 0 && errno != EWOULDBLOCK) { + syslog(LOG_ERR, "Fail to read from socket %d\n", fd); + return -EIO; + } + + if (!rc) { + syslog(LOG_ERR, "Fail to read from socket %d\n", fd); + return -EPIPE; + } + + if (msg.hdr.msg_type != RDMACM_MUX_MSG_TYPE_REQ) { + syslog(LOG_WARNING, "Got non-request message (%d) from socket %d\n", + msg.hdr.msg_type, fd); + return -EPERM; + } + + switch (msg.hdr.op_code) { + case RDMACM_MUX_OP_CODE_REG: + rc = add_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_OP_CODE_UNREG: + rc = delete_fd_ifid_pair(fd, msg.hdr.sgid.global.interface_id); + break; + + case RDMACM_MUX_OP_CODE_MAD: + /* If this is REQ or REP then store the pair comm_id,fd to be later + * used for other messages where gid is unknown */ + hdr = (struct umad_hdr *)msg.umad.mad; + attr_id = be16toh(hdr->attr_id); + if ((attr_id == UMAD_CM_ATTR_REQ) || (attr_id == UMAD_CM_ATTR_DREQ) || + (attr_id == UMAD_CM_ATTR_SIDR_REQ) || + (attr_id == UMAD_CM_ATTR_REP) || (attr_id == UMAD_CM_ATTR_DREP)) { + comm_id = (uint32_t *)(msg.umad.mad + sizeof(*hdr)); + hash_tbl_save_fd_comm_id_pair(fd, *comm_id, + msg.hdr.sgid.global.interface_id); + } + + syslog(LOG_DEBUG, "vm_to_mad: %d 0x%x 0x%x\n", fd, attr_id, + comm_id ? *comm_id : 0); + rc = umad_send(server.umad_agent.port_id, server.umad_agent.agent_id, + &msg.umad, msg.umad_len, 1, 0); + if (rc) { + syslog(LOG_ERR, + "Fail to send MAD message (0x%x) from socket %d, err=%d", + attr_id, fd, rc); + } + break; + + default: + syslog(LOG_ERR, "Got invalid op_code (%d) from socket %d", + msg.hdr.msg_type, fd); + rc = RDMACM_MUX_ERR_CODE_EINVAL; + } + + msg.hdr.msg_type = RDMACM_MUX_MSG_TYPE_RESP; + msg.hdr.err_code = rc; + rc = send(fd, &msg, sizeof(msg), 0); + + return rc == sizeof(msg) ? 0 : -EPIPE; +} + +static int accept_all(void) +{ + int fd, rc = 0;; + + pthread_rwlock_wrlock(&server.lock); + + do { + if ((server.nfds + 1) > MAX_CLIENTS) { + syslog(LOG_WARNING, "Too many clients (%d)", server.nfds); + rc = -EIO; + goto out; + } + + fd = accept(server.fds[0].fd, NULL, NULL); + if (fd < 0) { + if (errno != EWOULDBLOCK) { + syslog(LOG_WARNING, "accept() failed"); + rc = -EIO; + goto out; + } + break; + } + + syslog(LOG_INFO, "Client connected on socket %d\n", fd); + server.fds[server.nfds].fd = fd; + server.fds[server.nfds].events = POLLIN; + server.nfds++; + } while (fd != -1); + +out: + pthread_rwlock_unlock(&server.lock); + return rc; +} + +static void compress_fds(void) +{ + int i, j; + int closed = 0; + + pthread_rwlock_wrlock(&server.lock); + + for (i = 1; i < server.nfds; i++) { + if (!server.fds[i].fd) { + closed++; + for (j = i; j < server.nfds - 1; j++) { + server.fds[j] = server.fds[j + 1]; + } + } + } + + server.nfds -= closed; + + pthread_rwlock_unlock(&server.lock); +} + +static void close_fd(int idx) +{ + close(server.fds[idx].fd); + syslog(LOG_INFO, "Socket %d closed\n", server.fds[idx].fd); + hash_tbl_remove_fd_ifid_pair(server.fds[idx].fd); + server.fds[idx].fd = 0; +} + +static void run(void) +{ + int rc, nfds, i; + bool compress = false; + + syslog(LOG_INFO, "Service started"); + + while (server.run) { + rc = poll(server.fds, server.nfds, SLEEP_SECS * SCALE_US); + if (rc < 0) { + if (errno != EINTR) { + syslog(LOG_WARNING, "poll() failed"); + } + continue; + } + + if (rc == 0) { + continue; + } + + nfds = server.nfds; + for (i = 0; i < nfds; i++) { + syslog(LOG_DEBUG, "pollfd[%d]: revents 0x%x, events 0x%x\n", i, + server.fds[i].revents, server.fds[i].events); + if (server.fds[i].revents == 0) { + continue; + } + + if (server.fds[i].revents != POLLIN) { + if (i == 0) { + syslog(LOG_NOTICE, "Unexpected poll() event (0x%x)\n", + server.fds[i].revents); + } else { + close_fd(i); + compress = true; + } + continue; + } + + if (i == 0) { + rc = accept_all(); + if (rc) { + continue; + } + } else { + rc = read_and_process(server.fds[i].fd); + if (rc) { + close_fd(i); + compress = true; + } + } + } + + if (compress) { + compress = false; + compress_fds(); + } + } +} + +static void fini_listener(void) +{ + int i; + + if (server.fds[0].fd <= 0) { + return; + } + + for (i = server.nfds - 1; i >= 0; i--) { + if (server.fds[i].fd) { + close(server.fds[i].fd); + } + } + + unlink(server.args.unix_socket_path); +} + +static void fini_umad(void) +{ + if (server.umad_agent.agent_id) { + umad_unregister(server.umad_agent.port_id, server.umad_agent.agent_id); + } + + if (server.umad_agent.port_id) { + umad_close_port(server.umad_agent.port_id); + } + + hash_tbl_free(); +} + +static void fini(void) +{ + if (server.umad_recv_thread) { + pthread_join(server.umad_recv_thread, NULL); + server.umad_recv_thread = 0; + } + fini_umad(); + fini_listener(); + pthread_rwlock_destroy(&server.lock); + + syslog(LOG_INFO, "Service going down"); +} + +static int init_listener(void) +{ + struct sockaddr_un sun; + int rc, on = 1; + + server.fds[0].fd = socket(AF_UNIX, SOCK_STREAM, 0); + if (server.fds[0].fd < 0) { + syslog(LOG_ALERT, "socket() failed"); + return -EIO; + } + + rc = setsockopt(server.fds[0].fd, SOL_SOCKET, SO_REUSEADDR, (char *)&on, + sizeof(on)); + if (rc < 0) { + syslog(LOG_ALERT, "setsockopt() failed"); + rc = -EIO; + goto err; + } + + rc = ioctl(server.fds[0].fd, FIONBIO, (char *)&on); + if (rc < 0) { + syslog(LOG_ALERT, "ioctl() failed"); + rc = -EIO; + goto err; + } + + if (strlen(server.args.unix_socket_path) >= sizeof(sun.sun_path)) { + syslog(LOG_ALERT, + "Invalid unix_socket_path, size must be less than %ld\n", + sizeof(sun.sun_path)); + rc = -EINVAL; + goto err; + } + + sun.sun_family = AF_UNIX; + rc = snprintf(sun.sun_path, sizeof(sun.sun_path), "%s", + server.args.unix_socket_path); + if (rc < 0 || rc >= sizeof(sun.sun_path)) { + syslog(LOG_ALERT, "Could not copy unix socket path\n"); + rc = -EINVAL; + goto err; + } + + rc = bind(server.fds[0].fd, (struct sockaddr *)&sun, sizeof(sun)); + if (rc < 0) { + syslog(LOG_ALERT, "bind() failed"); + rc = -EIO; + goto err; + } + + rc = listen(server.fds[0].fd, SERVER_LISTEN_BACKLOG); + if (rc < 0) { + syslog(LOG_ALERT, "listen() failed"); + rc = -EIO; + goto err; + } + + server.fds[0].events = POLLIN; + server.nfds = 1; + server.run = true; + + return 0; + +err: + close(server.fds[0].fd); + return rc; +} + +static int init_umad(void) +{ + long method_mask[IB_USER_MAD_LONGS_PER_METHOD_MASK]; + + server.umad_agent.port_id = umad_open_port(server.args.rdma_dev_name, + server.args.rdma_port_num); + + if (server.umad_agent.port_id < 0) { + syslog(LOG_WARNING, "umad_open_port() failed"); + return -EIO; + } + + memset(&method_mask, 0, sizeof(method_mask)); + method_mask[0] = MAD_METHOD_MASK0; + server.umad_agent.agent_id = umad_register(server.umad_agent.port_id, + UMAD_CLASS_CM, + UMAD_SA_CLASS_VERSION, + MAD_RMPP_VERSION, method_mask); + if (server.umad_agent.agent_id < 0) { + syslog(LOG_WARNING, "umad_register() failed"); + return -EIO; + } + + hash_tbl_alloc(); + + return 0; +} + +static void signal_handler(int sig, siginfo_t *siginfo, void *context) +{ + static bool warned; + + /* Prevent stop if clients are connected */ + if (server.nfds != 1) { + if (!warned) { + syslog(LOG_WARNING, + "Can't stop while active client exist, resend SIGINT to overid"); + warned = true; + return; + } + } + + if (sig == SIGINT) { + server.run = false; + fini(); + } + + exit(0); +} + +static int init(void) +{ + int rc; + struct sigaction sig = {0}; + + rc = init_listener(); + if (rc) { + return rc; + } + + rc = init_umad(); + if (rc) { + return rc; + } + + pthread_rwlock_init(&server.lock, 0); + + rc = pthread_create(&server.umad_recv_thread, NULL, umad_recv_thread_func, + NULL); + if (rc) { + syslog(LOG_ERR, "Fail to create UMAD receiver thread (%d)\n", rc); + return rc; + } + + sig.sa_sigaction = &signal_handler; + sig.sa_flags = SA_SIGINFO; + rc = sigaction(SIGINT, &sig, NULL); + if (rc < 0) { + syslog(LOG_ERR, "Fail to install SIGINT handler (%d)\n", errno); + return rc; + } + + return 0; +} + +int main(int argc, char *argv[]) +{ + int rc; + + memset(&server, 0, sizeof(server)); + + parse_args(argc, argv); + + rc = init(); + if (rc) { + syslog(LOG_ERR, "Fail to initialize server (%d)\n", rc); + rc = -EAGAIN; + goto out; + } + + run(); + +out: + fini(); + + return rc; +} diff --git a/contrib/rdmacm-mux/rdmacm-mux.h b/contrib/rdmacm-mux/rdmacm-mux.h new file mode 100644 index 0000000000..942a802c47 --- /dev/null +++ b/contrib/rdmacm-mux/rdmacm-mux.h @@ -0,0 +1,61 @@ +/* + * QEMU paravirtual RDMA - rdmacm-mux declarations + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef RDMACM_MUX_H +#define RDMACM_MUX_H + +#include "linux/if.h" +#include "infiniband/verbs.h" +#include "infiniband/umad.h" +#include "rdma/rdma_user_cm.h" + +typedef enum RdmaCmMuxMsgType { + RDMACM_MUX_MSG_TYPE_REQ = 0, + RDMACM_MUX_MSG_TYPE_RESP = 1, +} RdmaCmMuxMsgType; + +typedef enum RdmaCmMuxOpCode { + RDMACM_MUX_OP_CODE_REG = 0, + RDMACM_MUX_OP_CODE_UNREG = 1, + RDMACM_MUX_OP_CODE_MAD = 2, +} RdmaCmMuxOpCode; + +typedef enum RdmaCmMuxErrCode { + RDMACM_MUX_ERR_CODE_OK = 0, + RDMACM_MUX_ERR_CODE_EINVAL = 1, + RDMACM_MUX_ERR_CODE_EEXIST = 2, + RDMACM_MUX_ERR_CODE_EACCES = 3, + RDMACM_MUX_ERR_CODE_ENOTFOUND = 4, +} RdmaCmMuxErrCode; + +typedef struct RdmaCmMuxHdr { + RdmaCmMuxMsgType msg_type; + RdmaCmMuxOpCode op_code; + union ibv_gid sgid; + RdmaCmMuxErrCode err_code; +} RdmaCmUHdr; + +typedef struct RdmaCmUMad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +} RdmaCmUMad; + +typedef struct RdmaCmMuxMsg { + RdmaCmUHdr hdr; + int umad_len; + RdmaCmUMad umad; +} RdmaCmMuxMsg; + +#endif From patchwork Sat Dec 22 09:50:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017811 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rtCq2QmT"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLX852skz9sPS for ; Sat, 22 Dec 2018 20:57:42 +1100 (AEDT) Received: from localhost ([::1]:45004 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae2O-0002ed-3W for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:57:40 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41601) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BK-9Q for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006NK-EP for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:35987) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006Ga-5m for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wr1-x441.google.com with SMTP id u4so7504289wrp.3 for ; Sat, 22 Dec 2018 01:50:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eFP2KnZ6LP9cKtdVM0lvyx4YWm9nUuG0BkL3Nds+VSs=; b=rtCq2QmTb2pdEG+zV0BfPFW8ekXH8T2v7qIeRUsHWCk845ZGqe5uQg/EWlpx1rZLe2 2/ugBId67neE1gXSde6ftsb4D70jjMobUDb4VIGqLFNOHXr9yUnZVbJW6m9PUjWi1Rrw XI1i+g6vh6bzENohzxis4lcPg2SADgASs6uGLhiGQUMWmNQO9ubTcuvRvTvo1FEmLUgU 3clhtcpYCQoXKIGmggiSpgGrniWHlhPERHq6JrUZEV2o7VCIGNvbyLKaikzw0//slj8Q VfubmB6fD17c4EFR3OL4mU6+bU8fUNXETGjkHmjPhKf7c+1/Zj4RT+ZF2wccxPwfGkqH 27RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eFP2KnZ6LP9cKtdVM0lvyx4YWm9nUuG0BkL3Nds+VSs=; b=VDmo8eeBR/ft10bwYCdEqCs2jc+xHdbAp1zxOhBTO08FfxP7pRncXc7qSkPu4E803h VCZ9z8tBs1Yfx/Xfbq1kGX3aPykNa1p8oN2mKjV8PczTjSk6uGH8t5BrAajdWJOMqDF/ 2ukBVi42g029FBmNpN0QtbZJMygY0vf2cmvBuxboLnRAU52L12UiV9BX8YHk2POInVv8 iINaV2fI3ryi2zbo2dr2SFYoy8QHQKlsDOmzQ/eeLoAenxN1i5z4E4ugNhTo/MSXOTTo o1MKGtM7YXDyzGy6g0VykrZ2Nf9AUZrxwOTUQs+3IlYNbzWFP+fKKlA8BYQ5GePr6Zuu /QQg== X-Gm-Message-State: AJcUukco+CLfsa1nUFv6+0eMFpggrbeDUChvJRVFU6q16qZWBPWeDYQI zM8Pe12opGFx3YLxj7dLY3VaKu77 X-Google-Smtp-Source: ALg8bN7iEHF7FiVORAf8NdWrw5aQRKBzYJJgoobhFqptDYzJJ6CLJJiRZDb+7eFB07pR2c+Ihgp/QA== X-Received: by 2002:adf:a452:: with SMTP id e18mr5933901wra.190.1545472244221; Sat, 22 Dec 2018 01:50:44 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.42 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:43 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:08 +0200 Message-Id: <20181222095036.29743-4-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 03/31] hw/rdma: Add ability to force notification without re-arm X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Upon completion of incoming packet the device pushes CQE to driver's RX ring and notify the driver (msix). While for data-path incoming packets the driver needs the ability to control whether it wished to receive interrupts or not, for control-path packets such as incoming MAD the driver needs to be notified anyway, it even do not need to re-arm the notification bit. Enhance the notification field to support this. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 12 ++++++++++-- hw/rdma/rdma_rm_defs.h | 8 +++++++- hw/rdma/vmw/pvrdma_qp_ops.c | 6 ++++-- 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 8d59a42cd1..4f10fcabcc 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -263,7 +263,7 @@ int rdma_rm_alloc_cq(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, } cq->opaque = opaque; - cq->notify = false; + cq->notify = CNT_CLEAR; rc = rdma_backend_create_cq(backend_dev, &cq->backend_cq, cqe); if (rc) { @@ -291,7 +291,10 @@ void rdma_rm_req_notify_cq(RdmaDeviceResources *dev_res, uint32_t cq_handle, return; } - cq->notify = notify; + if (cq->notify != CNT_SET) { + cq->notify = notify ? CNT_ARM : CNT_CLEAR; + } + pr_dbg("notify=%d\n", cq->notify); } @@ -349,6 +352,11 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32_t pd_handle, return -EINVAL; } + if (qp_type == IBV_QPT_GSI) { + scq->notify = CNT_SET; + rcq->notify = CNT_SET; + } + qp = res_tbl_alloc(&dev_res->qp_tbl, &rm_qpn); if (!qp) { return -ENOMEM; diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7228151239..9b399063d3 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -49,10 +49,16 @@ typedef struct RdmaRmPD { uint32_t ctx_handle; } RdmaRmPD; +typedef enum CQNotificationType { + CNT_CLEAR, + CNT_ARM, + CNT_SET, +} CQNotificationType; + typedef struct RdmaRmCQ { RdmaBackendCQ backend_cq; void *opaque; - bool notify; + CQNotificationType notify; } RdmaRmCQ; /* MR (DMA region) */ diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index c668afd0ed..762700a205 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -89,8 +89,10 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, pvrdma_ring_write_inc(&dev->dsr_info.cq); pr_dbg("cq->notify=%d\n", cq->notify); - if (cq->notify) { - cq->notify = false; + if (cq->notify != CNT_CLEAR) { + if (cq->notify == CNT_ARM) { + cq->notify = CNT_CLEAR; + } post_interrupt(dev, INTR_VEC_CMD_COMPLETION_Q); } From patchwork Sat Dec 22 09:50:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017802 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ceCzB5C4"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLPD5p0yz9sDP for ; Sat, 22 Dec 2018 20:51:44 +1100 (AEDT) Received: from localhost ([::1]:36987 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwc-0006CX-8d for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:51:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41546) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvp-00069p-79 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvm-0006KC-Qp for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:53 -0500 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:45293) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvj-0006Hi-E5 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:49 -0500 Received: by mail-wr1-x444.google.com with SMTP id t6so7505658wrr.12 for ; Sat, 22 Dec 2018 01:50:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BxYSbWDd/XVr3dBZibxFvih0GIrCV1IwvMdaUb3g1/E=; b=ceCzB5C4+++Vr4mSwc3Sgd1LFYyE/aNnJ2zwjtWZ+FTYeeL3UI5vw6CpBDE8/3hofb OaBfbMMW/1JNpBdkSDi/+YB4c1bSVvqTpMF5rr1My1L/0xgjPAVYtEWDg+4FzNR6/WZZ Mx5HACtfjo82USbY1lYF8GmoCNLKSHzfMxR6MfXue54ibzFCnLd0kXOoBoe0SMxPMg+J 1LL3wd7EzaZoyhGG6T+HELQf+EIWbaa33LgjfSDkta8ykZsM6kFDQF5dzspskxrykKmj Q2DxM0LjS9JLWAgBcuDN40ZkuOHu/kxmT7lIVSLiNznpAQYPec26CZmpLm71MHclRL3C WqMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BxYSbWDd/XVr3dBZibxFvih0GIrCV1IwvMdaUb3g1/E=; b=LUeKlLdasBcTvZCaZvluBOC9hnrXQVqWPAUkXlh4ZSRVH0a4xq6SBQivlx/yZoD+Qs xC22vXvSgA3fg1a3UAyixPDZOzoKzqOhsmw/0Yx6U/qnLrE53VT3MseNYR4yhhCgGHPP kZPVd6pJv+nI+Kzj+F4Rvs74H030PYkR7flO+oJz+7fLUcS9oND6bS2/0pBzG5RS516m X3JF7OqTtBnRwSMQW5N9M/MYgDWJUDZ52794UuNdjy+9iXksGOGhNr70IT/XiRbSkMIF XYoND5GJghyyuaAsL/JINRdbxvqGzRW40ZxRalav3Q9ulIKWwcwcvCQCg+u1afJHE/o4 CP0w== X-Gm-Message-State: AJcUukeDOP2ocT5byWlYUWpPtJy3jwwiHPwCO5J1SOh/BLuzh5DBesHh yqUYYtSluHC+2gGt1ehGER8sv61V X-Google-Smtp-Source: ALg8bN7tjWVTsTEbco/3H4rGIszKaz4q4Yf0xQVqheOP348qVxUjIGSQvA2xfXXgABDxkYeovOUAuA== X-Received: by 2002:adf:ae41:: with SMTP id u1mr5429199wrd.20.1545472245648; Sat, 22 Dec 2018 01:50:45 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:45 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:09 +0200 Message-Id: <20181222095036.29743-5-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::444 Subject: [Qemu-devel] [PATCH PULL 04/31] hw/rdma: Return qpn 1 if ibqp is NULL X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Device is not supporting QP0, only QP1. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 86e8fe8ab6..3ccc9a2494 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -33,7 +33,7 @@ static inline union ibv_gid *rdma_backend_gid(RdmaBackendDev *dev) static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { - return qp->ibqp ? qp->ibqp->qp_num : 0; + return qp->ibqp ? qp->ibqp->qp_num : 1; } static inline uint32_t rdma_backend_mr_lkey(const RdmaBackendMR *mr) From patchwork Sat Dec 22 09:50:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017807 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="pc+6Bwf3"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLSf0Yc9z9sP9 for ; Sat, 22 Dec 2018 20:54:41 +1100 (AEDT) Received: from localhost ([::1]:40890 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadzT-0000Gd-Ix for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:54:39 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41606) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BN-9U for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvp-0006Lh-5r for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:43227) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvm-0006IS-Q2 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:51 -0500 Received: by mail-wr1-x442.google.com with SMTP id r10so7501221wrs.10 for ; Sat, 22 Dec 2018 01:50:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=4nvGW0Z3fwJM51D7eCHH6NpEw6DMydEHMqb7Z6Ln3ak=; b=pc+6Bwf3g0T4qBikt2WbyEq0mhZg+LR9qarICU90pzD7JsVjk98FBXe19Ma3XA1jye H7xxpbcPXn6QFAhW/0L8+iO3synimCRVnFfwPuXg00K4Zo68X8wcTZnK3fHmzr+2j3Fz 6zTUCr6T7BnOoL9ovW1xRO8s45DZkcRYP0xDx82wfDLe9RWdW/TzTcYmnRsSMIUFSqoA EHssbndbt6GQU9P4rYJ/GntnUNh23EGNKcEFrK8liPURozyy5Ih2Gr+Yn8eBzhGnpQKP kZXSKRySw4SeXUp0lhFRsE579p2hAsgAMrxC8wugGbxQjdLR6F8eOq1pYJ5fl6WQhYtf 2lig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=4nvGW0Z3fwJM51D7eCHH6NpEw6DMydEHMqb7Z6Ln3ak=; b=oAE7isSYiDc3vPsefnbev412t4YM0lUullwlEk0+EZYB0I8tv+nsub9F4o0VB6g09r 1NCUP/W4Rjj8Vc3lsngxt8jNfVEzrziyLFn//lNHfl/Y1CcDFMiTHI7w569tu1bq6D0b X/7U9gwIS3A+7vo4pnr7pUCjqV5nl4QqxgdLsCzXZXwDh/GYXDufIhiAlD3fqcfKKWrr c0nE9PiOfSdfC5CqIw1I5NXI0gc2W1O6Cj4/vCqLDtcnsiU4JaIspTI7O5Xb1XPpOLFJ +BUZxsJzf5RUUz9oTkOjKrxISyqCgVhNnbHrK5k5OXpo0PdivPmvlASmBqtfPjoYW2Ma UVmw== X-Gm-Message-State: AJcUukfls4sNfnTsiRKg7luu97EJR+hY8puthQAZ9DWPMp7+WNyPF9fi rcNkUcb0G/KXi9Z1F9DtKbkK+Lzj X-Google-Smtp-Source: ALg8bN4jVFKqH+B/rWgnJYS/2SRsPc2nfc6PAhSxx3WQ+fxDtxO8GVMfZ4GyecrJx5+17RGs2qBIcg== X-Received: by 2002:adf:a357:: with SMTP id d23mr5922374wrb.195.1545472246970; Sat, 22 Dec 2018 01:50:46 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:46 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:10 +0200 Message-Id: <20181222095036.29743-6-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 05/31] hw/rdma: Abort send-op if fail to create addr handler X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Function create_ah might return NULL, let's exit with an error. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index d7a4bbd91f..1e148398a2 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -338,6 +338,10 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, if (qp_type == IBV_QPT_UD) { wr.wr.ud.ah = create_ah(backend_dev, qp->ibpd, backend_dev->backend_gid_idx, dgid); + if (!wr.wr.ud.ah) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + goto out_dealloc_cqe_ctx; + } wr.wr.ud.remote_qpn = dqpn; wr.wr.ud.remote_qkey = dqkey; } From patchwork Sat Dec 22 09:50:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017812 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HaJqr475"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLX8562mz9sPX for ; Sat, 22 Dec 2018 20:57:44 +1100 (AEDT) Received: from localhost ([::1]:45042 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae2Q-0002fs-3T for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:57:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41619) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BX-Gp for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:59 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006NY-HV for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:46256) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006JI-5Z for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wr1-x444.google.com with SMTP id l9so7494362wrt.13 for ; Sat, 22 Dec 2018 01:50:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=voJMFy0LfLUex5PDX5hMY3TPkWpxWRXEInkRzWuaJ4o=; b=HaJqr475u9241kefbD9gLNyE2Vf8wM5Ko0dVIe7B0a3+9c5GWjlw6K0xbs+A2I9kHF pyr1VPX11d6jAxOeO8aWzVF6OuulVlL78odLKS12wpmEEIcV1zh332iAnApeJpj1/WK0 X7As5kjRs6/Q3Yw4fwsQ6nQkOeLvPYCAMtesNiRk143PmBizAzynro2vSXonk5laJ8Zs JtFwuaLkVMJvU6oaP8aXfPmn+CcQ6d0rYPaUCzbb6g0zeckGBLB9QDlI7NmmLRCwW2DK tvPrE6vqC/oZW78zQafbtPefWnx5DWbk/6xMctmGftq8f5bA5iwm6+oLYp+f6xBm1ItM 6NLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=voJMFy0LfLUex5PDX5hMY3TPkWpxWRXEInkRzWuaJ4o=; b=Mv5LNDpaJVuLLAQm9YwQYVbvYT8pqYOFupD+NphCTYWqDp/8Aqm30H4uDwm2WybOnG VTp2IhdWfryXEW8qKsLVsDgC8IiJtASEun26Iz/BMaXeSPwLhqQvD4Xa8AbZKH/RMrVf 5buNaTbqe0brpZfgA6oXV82FinlRcjYwrKWbyG4R1xyImEiB+9HFJCTClt29K/q012c0 wTEgapJvPAOO60Pk0PLi2/KrtaXBWpsE9JjHrzxai5aetYH4G5fS1KkNs2WCWHhLwKSP VDxcHKJLWGXghkO1TxMK5Q7iDPk2UkojPfCuupSHIaUYugW7m6iXpPG47pcBxzdnJ4oC aGJA== X-Gm-Message-State: AJcUukcu3yk01w/djLMX68JMkPLlD8KFg8rapK2wJFB5glYeMc67RAvA gJHGsI8st6nl1BGm8LKFu16w/WqY X-Google-Smtp-Source: ALg8bN417oj5mNfK64gdYTW6u7UtU3jbB0FTO7RvjgaapGt1aVEhIX4A1QkLoCaFU4B1lTULUBK86Q== X-Received: by 2002:a5d:454f:: with SMTP id p15mr5994731wrr.39.1545472248542; Sat, 22 Dec 2018 01:50:48 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.47 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:47 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:11 +0200 Message-Id: <20181222095036.29743-7-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::444 Subject: [Qemu-devel] [PATCH PULL 06/31] hw/rdma: Add support for MAD packets X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia MAD (Management Datagram) packets are widely used by various modules both in kernel and in user space for example the rdma_* API which is used to create and maintain "connection" layer on top of RDMA uses several types of MAD packets. For more information please refer to chapter 13.4 in Volume 1 Architecture Specification, Release 1.1 available here: https://www.infinibandta.org/ibta-specifications-download/ To support MAD packets the device uses an external utility (contrib/rdmacm-mux) to relay packets from and to the guest driver. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 250 +++++++++++++++++++++++++++++++++++- hw/rdma/rdma_backend.h | 4 +- hw/rdma/rdma_backend_defs.h | 10 +- hw/rdma/vmw/pvrdma.h | 2 + hw/rdma/vmw/pvrdma_main.c | 4 +- 5 files changed, 260 insertions(+), 10 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 1e148398a2..c6dedda555 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -16,8 +16,13 @@ #include "qemu/osdep.h" #include "qemu/error-report.h" #include "qapi/error.h" +#include "qapi/qmp/qlist.h" +#include "qapi/qmp/qnum.h" #include +#include +#include +#include #include "trace.h" #include "rdma_utils.h" @@ -33,16 +38,25 @@ #define VENDOR_ERR_MAD_SEND 0x206 #define VENDOR_ERR_INVLKEY 0x207 #define VENDOR_ERR_MR_SMALL 0x208 +#define VENDOR_ERR_INV_MAD_BUFF 0x209 +#define VENDOR_ERR_INV_NUM_SGE 0x210 #define THR_NAME_LEN 16 #define THR_POLL_TO 5000 +#define MAD_HDR_SIZE sizeof(struct ibv_grh) + typedef struct BackendCtx { - uint64_t req_id; void *up_ctx; bool is_tx_req; + struct ibv_sge sge; /* Used to save MAD recv buffer */ } BackendCtx; +struct backend_umad { + struct ib_user_mad hdr; + char mad[RDMA_MAX_PRIVATE_DATA]; +}; + static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx); static void dummy_comp_handler(int status, unsigned int vendor_err, void *ctx) @@ -286,6 +300,61 @@ static int build_host_sge_array(RdmaDeviceResources *rdma_dev_res, return 0; } +static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, + uint32_t num_sge) +{ + struct backend_umad umad = {0}; + char *hdr, *msg; + int ret; + + pr_dbg("num_sge=%d\n", num_sge); + + if (num_sge != 2) { + return -EINVAL; + } + + umad.hdr.length = sge[0].length + sge[1].length; + pr_dbg("msg_len=%d\n", umad.hdr.length); + + if (umad.hdr.length > sizeof(umad.mad)) { + return -ENOMEM; + } + + umad.hdr.addr.qpn = htobe32(1); + umad.hdr.addr.grh_present = 1; + umad.hdr.addr.gid_index = backend_dev->backend_gid_idx; + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.gid)); + umad.hdr.addr.hop_limit = 0xFF; + + hdr = rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); + if (!hdr) { + pr_dbg("Fail to map to sge[0]\n"); + return -ENOMEM; + } + msg = rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + if (!msg) { + pr_dbg("Fail to map to sge[1]\n"); + rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); + return -ENOMEM; + } + + pr_dbg_buf("mad_hdr", hdr, sge[0].length); + pr_dbg_buf("mad_data", data, sge[1].length); + + memcpy(&umad.mad[0], hdr, sge[0].length); + memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + + rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); + + ret = qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&umad, + sizeof(umad)); + + pr_dbg("qemu_chr_fe_write=%d\n", ret); + + return (ret != sizeof(umad)); +} + void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, @@ -304,9 +373,13 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type == IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc = mad_send(backend_dev, sge, num_sge); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + } else { + comp_handler(IBV_WC_SUCCESS, 0, ctx); + } } - pr_dbg("qp->ibqp is NULL for qp_type %d!!!\n", qp_type); return; } @@ -370,6 +443,48 @@ out_free_bctx: g_free(bctx); } +static unsigned int save_mad_recv_buffer(RdmaBackendDev *backend_dev, + struct ibv_sge *sge, uint32_t num_sge, + void *ctx) +{ + BackendCtx *bctx; + int rc; + uint32_t bctx_id; + + if (num_sge != 1) { + pr_dbg("Invalid num_sge (%d), expecting 1\n", num_sge); + return VENDOR_ERR_INV_NUM_SGE; + } + + if (sge[0].length < RDMA_MAX_PRIVATE_DATA + sizeof(struct ibv_grh)) { + pr_dbg("Too small buffer for MAD\n"); + return VENDOR_ERR_INV_MAD_BUFF; + } + + pr_dbg("addr=0x%" PRIx64"\n", sge[0].addr); + pr_dbg("length=%d\n", sge[0].length); + pr_dbg("lkey=%d\n", sge[0].lkey); + + bctx = g_malloc0(sizeof(*bctx)); + + rc = rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx); + if (unlikely(rc)) { + g_free(bctx); + pr_dbg("Fail to allocate cqe_ctx\n"); + return VENDOR_ERR_NOMEM; + } + + pr_dbg("bctx_id %d, bctx %p, ctx %p\n", bctx_id, bctx, ctx); + bctx->up_ctx = ctx; + bctx->sge = *sge; + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + qlist_append_int(backend_dev->recv_mads_list.list, bctx_id); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + + return 0; +} + void rdma_backend_post_recv(RdmaBackendDev *backend_dev, RdmaDeviceResources *rdma_dev_res, RdmaBackendQP *qp, uint8_t qp_type, @@ -388,7 +503,10 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, } if (qp_type == IBV_QPT_GSI) { pr_dbg("QP1\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + rc = save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); + if (rc) { + comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + } } return; } @@ -517,7 +635,6 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type, switch (qp_type) { case IBV_QPT_GSI: - pr_dbg("QP1 unsupported\n"); return 0; case IBV_QPT_RC: @@ -748,11 +865,122 @@ static int init_device_caps(RdmaBackendDev *backend_dev, return 0; } +static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, + union ibv_gid *my_gid, int paylen) +{ + grh->paylen = htons(paylen); + grh->sgid = *sgid; + grh->dgid = *my_gid; + + pr_dbg("paylen=%d (net=0x%x)\n", paylen, grh->paylen); + pr_dbg("my_gid=0x%llx\n", my_gid->global.interface_id); + pr_dbg("gid=0x%llx\n", sgid->global.interface_id); +} + +static inline int mad_can_receieve(void *opaque) +{ + return sizeof(struct backend_umad); +} + +static void mad_read(void *opaque, const uint8_t *buf, int size) +{ + RdmaBackendDev *backend_dev = (RdmaBackendDev *)opaque; + QObject *o_ctx_id; + unsigned long cqe_ctx_id; + BackendCtx *bctx; + char *mad; + struct backend_umad *umad; + + assert(size != sizeof(umad)); + umad = (struct backend_umad *)buf; + + pr_dbg("Got %d bytes\n", size); + pr_dbg("umad->hdr.length=%d\n", umad->hdr.length); + +#ifdef PVRDMA_DEBUG + struct umad_hdr *hdr = (struct umad_hdr *)&msg->umad.mad; + pr_dbg("bv %x cls %x cv %x mtd %x st %d tid %" PRIx64 " at %x atm %x\n", + hdr->base_version, hdr->mgmt_class, hdr->class_version, + hdr->method, hdr->status, be64toh(hdr->tid), + hdr->attr_id, hdr->attr_mod); +#endif + + qemu_mutex_lock(&backend_dev->recv_mads_list.lock); + o_ctx_id = qlist_pop(backend_dev->recv_mads_list.list); + qemu_mutex_unlock(&backend_dev->recv_mads_list.lock); + if (!o_ctx_id) { + pr_dbg("No more free MADs buffers, waiting for a while\n"); + sleep(THR_POLL_TO); + return; + } + + cqe_ctx_id = qnum_get_uint(qobject_to(QNum, o_ctx_id)); + bctx = rdma_rm_get_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); + if (unlikely(!bctx)) { + pr_dbg("Error: Fail to find ctx for %ld\n", cqe_ctx_id); + return; + } + + pr_dbg("id %ld, bctx %p, ctx %p\n", cqe_ctx_id, bctx, bctx->up_ctx); + + mad = rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, + bctx->sge.length); + if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); + } else { + memset(mad, 0, bctx->sge.length); + build_mad_hdr((struct ibv_grh *)mad, + (union ibv_gid *)&umad->hdr.addr.gid, + &backend_dev->gid, umad->hdr.length); + memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); + + comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + } + + g_free(bctx); + rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); +} + +static int mad_init(RdmaBackendDev *backend_dev) +{ + struct backend_umad umad = {0}; + int ret; + + if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + pr_dbg("Missing chardev for MAD multiplexer\n"); + return -EIO; + } + + qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, + mad_read, NULL, NULL, backend_dev, NULL, true); + + /* Register ourself */ + memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.gid)); + ret = qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&umad, + sizeof(umad.hdr)); + if (ret != sizeof(umad.hdr)) { + pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); + } + + qemu_mutex_init(&backend_dev->recv_mads_list.lock); + backend_dev->recv_mads_list.list = qlist_new(); + + return 0; +} + +static void mad_fini(RdmaBackendDev *backend_dev) +{ + qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); + qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr, - Error **errp) + CharBackend *mad_chr_be, Error **errp) { int i; int ret = 0; @@ -763,7 +991,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, memset(backend_dev, 0, sizeof(*backend_dev)); backend_dev->dev = pdev; - + backend_dev->mad_chr_be = mad_chr_be; backend_dev->backend_gid_idx = backend_gid_idx; backend_dev->port_num = port_num; backend_dev->rdma_dev_res = rdma_dev_res; @@ -854,6 +1082,13 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, pr_dbg("interface_id=0x%" PRIx64 "\n", be64_to_cpu(backend_dev->gid.global.interface_id)); + ret = mad_init(backend_dev); + if (ret) { + error_setg(errp, "Fail to initialize mad"); + ret = -EIO; + goto out_destroy_comm_channel; + } + backend_dev->comp_thread.run = false; backend_dev->comp_thread.is_running = false; @@ -890,6 +1125,7 @@ void rdma_backend_stop(RdmaBackendDev *backend_dev) void rdma_backend_fini(RdmaBackendDev *backend_dev) { rdma_backend_stop(backend_dev); + mad_fini(backend_dev); g_hash_table_destroy(ah_hash); ibv_destroy_comp_channel(backend_dev->channel); ibv_close_device(backend_dev->context); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 3ccc9a2494..fc83330251 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -17,6 +17,8 @@ #define RDMA_BACKEND_H #include "qapi/error.h" +#include "chardev/char-fe.h" + #include "rdma_rm_defs.h" #include "rdma_backend_defs.h" @@ -50,7 +52,7 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr, - Error **errp); + CharBackend *mad_chr_be, Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 7404f64002..2a7e667075 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -16,8 +16,9 @@ #ifndef RDMA_BACKEND_DEFS_H #define RDMA_BACKEND_DEFS_H -#include #include "qemu/thread.h" +#include "chardev/char-fe.h" +#include typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -28,6 +29,11 @@ typedef struct RdmaBackendThread { bool is_running; /* Set by the thread to report its status */ } RdmaBackendThread; +typedef struct RecvMadList { + QemuMutex lock; + QList *list; +} RecvMadList; + typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; @@ -39,6 +45,8 @@ typedef struct RdmaBackendDev { struct ibv_comp_channel *channel; uint8_t port_num; uint8_t backend_gid_idx; + RecvMadList recv_mads_list; + CharBackend *mad_chr_be; } RdmaBackendDev; typedef struct RdmaBackendPD { diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e2d9f93cdf..e3742d893a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -19,6 +19,7 @@ #include "qemu/units.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" +#include "chardev/char-fe.h" #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -83,6 +84,7 @@ typedef struct PVRDMADev { uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; + CharBackend mad_chr; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ca5fa8d981..6c8c0154fa 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -51,6 +51,7 @@ static Property pvrdma_dev_properties[] = { DEFINE_PROP_INT32("dev-caps-max-qp-init-rd-atom", PVRDMADev, dev_attr.max_qp_init_rd_atom, MAX_QP_INIT_RD_ATOM), DEFINE_PROP_INT32("dev-caps-max-ah", PVRDMADev, dev_attr.max_ah, MAX_AH), + DEFINE_PROP_CHR("mad-chardev", PVRDMADev, mad_chr), DEFINE_PROP_END_OF_LIST(), }; @@ -613,7 +614,8 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) rc = rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, errp); + dev->backend_gid_idx, &dev->dev_attr, &dev->mad_chr, + errp); if (rc) { goto out; } From patchwork Sat Dec 22 09:50:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017815 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jetU0rJL"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLbn6L1jz9sPP for ; Sat, 22 Dec 2018 21:00:53 +1100 (AEDT) Received: from localhost ([::1]:49311 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae5T-0005Hl-Dz for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:00:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41602) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BJ-9K for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006N2-97 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x443.google.com ([2a00:1450:4864:20::443]:39370) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006Jy-2y for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wr1-x443.google.com with SMTP id t27so7515649wra.6 for ; Sat, 22 Dec 2018 01:50:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yQg8b3oyr76vDuCw+AgIIzlmfFVQe+hno9oArasV4OA=; b=jetU0rJLz41fhur3wJr4m6cvwYyJDPgPp39XCpjATcUUR6FWaftLZ4Bq0ehFHrVE94 mwsg54t3HWCF6qCwR80PicFYXsUZnrH6GMqp51+dUgzoEj3SNYvajlw4n1x+eClRMW81 5ab/szHvio7Z4SMD/0fInfkC4vFcbIBtZynSkgISFfqUGWxGfwbtmt/Bhbnl7f6vNg5H UeFTHHtX+qrtVib7Ic3shfR67C1gRbUIijxJrRJQxO01+tG7UNDU1uL04E0dBEYDEYYC zq4AVQ4dD6ZVdgwUbVcRw6PUdpAYVT+5SvTb+cGQmGQvMpi4BSVK6KSSseM3KpF7OH2A ZE1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yQg8b3oyr76vDuCw+AgIIzlmfFVQe+hno9oArasV4OA=; b=a48HZpzbM4ZG1Kv8HnS5UlrMhlHcuuNMx8lwVFWlS91VdW4/5fTFQe0DiN0E5GtZ/t omRPoO8bDOKc18cQN+Y/Jfrl13l6bK2ETc51rQTXOEILjSSohSSzgo21AOq3/1SIqNsT dlQp8YTKKMCMfZEhl0cXk8+VLQchYLVOFfz+EPQxkc6WmEamSkuVhCxXlOuPlL1Op+6k OWYTI7GZktNhsHEF/7CIwitybec2dYRHhNh4rcchVemOgLayw2PTd279yQyBmDVtHQBc 1LV/hH87DbCGgAjTvCiLZ+2DNoHQhjFXElkKkwfcGO+OU3+XOe4IGlef7+H91hf3Pdow Vj9Q== X-Gm-Message-State: AJcUukczqSXF2dm/ZfEwZmiFSAwnzioLq9bOCCBcnqHo4bZIv8MyQDja +97IuTa912tIBkXKMtZW7uA1jmAI X-Google-Smtp-Source: ALg8bN7US3v/7shPCI++1G4gYdpIiGYa4UrEOEXUA4Io8GyVtQzMsYfTEEeFK44J65O6RDKWjRBhOQ== X-Received: by 2002:adf:f848:: with SMTP id d8mr6063645wrq.178.1545472249918; Sat, 22 Dec 2018 01:50:49 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:49 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:12 +0200 Message-Id: <20181222095036.29743-8-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::443 Subject: [Qemu-devel] [PATCH PULL 07/31] hw/pvrdma: Make function reset_device return void X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia This function cannot fail - fix it to return void Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_main.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 6c8c0154fa..fc2abd34af 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -369,13 +369,11 @@ static int unquiesce_device(PVRDMADev *dev) return 0; } -static int reset_device(PVRDMADev *dev) +static void reset_device(PVRDMADev *dev) { pvrdma_stop(dev); pr_dbg("Device reset complete\n"); - - return 0; } static uint64_t regs_read(void *opaque, hwaddr addr, unsigned size) From patchwork Sat Dec 22 09:50:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017809 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="SktwM26s"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLSr4D1pz9sPL for ; Sat, 22 Dec 2018 20:54:52 +1100 (AEDT) Received: from localhost ([::1]:41080 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadzd-0000NF-SK for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:54:49 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41608) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BP-9y for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006NE-DF for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x444.google.com ([2a00:1450:4864:20::444]:42940) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006L1-5A for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wr1-x444.google.com with SMTP id q18so7506661wrx.9 for ; Sat, 22 Dec 2018 01:50:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UMZbrb9gaucJzS84R+nsAQwoi5Gq22wSFceJflDnYEA=; b=SktwM26svgYTgixWdvqSpHw8FDTixS2PrqisOrBebAwHIaHj1YChDRczkOtGkpgO+2 34DYoar2tSwc0gQQM+G8Gx3HqJA18xyE42qat77qfDzzVSRaWuejCgzqXMdYPCs+/wEc +0n4W+5gVkgyO7fhYZ/e4qdzAnDcLUQUBE5fao7tBSeOckqjncGX4oVoyK7DB3kxkCeC c9yY+mrTe9LtfsolPDhxU1FCx7AeGAI4lYNKQ30Dy3vSCzev6HRb/97OUuTHQ9E4ZfkY IBR0v7Kc73F1YLIUAAg2WQnjbSBBzGoFwS4qchErUdMScej/AYmsp7ENCwewcDhiApnB DbyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UMZbrb9gaucJzS84R+nsAQwoi5Gq22wSFceJflDnYEA=; b=K/RANc5zb/5MdhPJpkOaZT7vUSeFMw+BJt0pd/M5ewPgGv20/Sex12K+FnkBvSqMeq 9quqcG/REA9XBoM7kYbtFHrpYUiryzSszuWe3V9koKO5BHtEomqTm+qd7gGcA1oyw4xj S3QZoklS3iKAGVLY8al7+vWp3LcevflJtsyn7rvvegBXjx6+xBtRXhOuijuj/qgSmGtt XPNeZrVJHd00PhkmAOKJFIftHSr204SNWUCp7tyO20IRpjU4cfi3gJDccrZ7qRurTdMo +jGYGS9Zw5UfoDTYJpbOXP8fNU3g7af8+K9Jk5UKk7o7/Vn7lAIsoOzVqlwWSX97dcV+ ayiw== X-Gm-Message-State: AJcUukcgjzhTJQKwTT4qqe4xUEYLHORaNrqtuanLIaYQU2ZrkSm9PMV1 EpP7lUsXUqnX3yB8Y/COdFr/AY1t X-Google-Smtp-Source: ALg8bN55c2StRw0+SX3RjrE97zmsLHGQ+9eiNlRyP70YfYshDvWPQ0w9tOmKihLbHr7BG9oUwK56og== X-Received: by 2002:adf:cd0e:: with SMTP id w14mr5885522wrm.218.1545472251268; Sat, 22 Dec 2018 01:50:51 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.50 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:50 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:13 +0200 Message-Id: <20181222095036.29743-9-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::444 Subject: [Qemu-devel] [PATCH PULL 08/31] hw/pvrdma: Make default pkey 0xFFFF X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Commit 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") exports default pkey as external definition but omit the change from 0x7FFF to 0xFFFF. Fixes: 6e7dba23af ("hw/pvrdma: Make default pkey 0xFFFF") Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index e3742d893a..15c3f28b86 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -52,7 +52,7 @@ #define PVRDMA_FW_VERSION 14 /* Some defaults */ -#define PVRDMA_PKEY 0x7FFF +#define PVRDMA_PKEY 0xFFFF typedef struct DSRInfo { dma_addr_t dma; From patchwork Sat Dec 22 09:50:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017808 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="hbKtUyVg"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLSf1yDmz9sPB for ; Sat, 22 Dec 2018 20:54:42 +1100 (AEDT) Received: from localhost ([::1]:40899 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadzT-0000Gv-MW for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:54:39 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41600) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BF-8z for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006NS-Gh for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:52779) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006Lr-91 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wm1-x341.google.com with SMTP id m1so7392272wml.2 for ; Sat, 22 Dec 2018 01:50:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FHZuDhDDF8ZKZQOvHvJuEQtp3p+2gysz6ZX3O62QRTE=; b=hbKtUyVgEqqzO51lZapCI8v9LbwmJeY4wfXTjJ+CbGI2hUDAtCvCFuW2zoOU6N4yrI GXr1aPPT2Tf5XTbvQbUgpil0JHePtTejl7zBrXAqcxMhzc0T5MPjLjrwpbmuF0uC59NH 5dBIA4N9J+ECcOpZL6IOGhh+wf/7TOy1CG/7GeYEoVvmvRbA15DBiKUurxZupWQVyY+Z tzcuXRpR1fqb/D4rUIRI+JkxAF5VHr420Abjyz1gmJ+q3Es/qaeOOzV5QAaNorLGT1Yo LlVFC9bB+Majy+e4gH3eJc6eUvR17w54BVz20Ji6C7RcQ2v6D7DMVdbGl07PNg7Oq0xC SrUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FHZuDhDDF8ZKZQOvHvJuEQtp3p+2gysz6ZX3O62QRTE=; b=FqiZcwuYGHogbX/rfgtff43nILkL81CDUEnemI8gSoUHegsZvv59bbdr37m4Ae6Oda HZSYgcozoOnKXHcR6LH8Zva3YjrsJO82gSXHa1X8t09JQiMOWo4X18nfCboEJvKfyyXY i15t4OMACIGJg053egNouDM/ahnfKUqyYUKT/injM3DmP+m7MuWArpOQfqUOCc1n8ahR WRMg9qxYwj2kiRg1ET3k+3YzhURJFE8wFt9Hf3jCymyrSD9Dcf2bgnXk2roet7inppcc trZgbXhW3xHrn8WL8ING+rp/HV0oNsBS49ha+FDq8/gfpYN+604Io3ey14yoRBhLxALk fATQ== X-Gm-Message-State: AJcUukeuAAFE+fteWnRt9N6AfJAR5+QJ6lluIZbL6bsQnILZEoeP5S7L jB1buxZ3DJ4KeygHNUTBVl78u/il X-Google-Smtp-Source: ALg8bN5rPFAiVlLxp43hpUYFhv262nf1lGqqPpDPN7KHuuhW+Vc1vZUHr8yR30GxDhGO0hgkhOYYGw== X-Received: by 2002:a1c:4d12:: with SMTP id o18mr6151901wmh.92.1545472252628; Sat, 22 Dec 2018 01:50:52 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.51 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:52 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:14 +0200 Message-Id: <20181222095036.29743-10-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 09/31] hw/pvrdma: Set the correct opcode for recv completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia The function pvrdma_post_cqe populates CQE entry with opcode from the given completion element. For receive operation value was not set. Fix it by setting it to IBV_WC_RECV. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_qp_ops.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 762700a205..7b0f440fda 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -196,8 +196,9 @@ int pvrdma_qp_recv(PVRDMADev *dev, uint32_t qp_handle) comp_ctx = g_malloc(sizeof(CompHandlerCtx)); comp_ctx->dev = dev; comp_ctx->cq_handle = qp->recv_cq_handle; - comp_ctx->cqe.qp = qp_handle; comp_ctx->cqe.wr_id = wqe->hdr.wr_id; + comp_ctx->cqe.qp = qp_handle; + comp_ctx->cqe.opcode = IBV_WC_RECV; rdma_backend_post_recv(&dev->backend_dev, &dev->rdma_dev_res, &qp->backend_qp, qp->qp_type, From patchwork Sat Dec 22 09:50:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017813 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JgOdqSkL"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLXH0Zydz9sPP for ; Sat, 22 Dec 2018 20:57:50 +1100 (AEDT) Received: from localhost ([::1]:45218 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae2W-0002lX-KV for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:57:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41604) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006BM-9R for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvr-0006Ne-IJ for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:46254) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvr-0006MS-Bn for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:55 -0500 Received: by mail-wr1-x441.google.com with SMTP id l9so7494483wrt.13 for ; Sat, 22 Dec 2018 01:50:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sQ4kjjEDKGZ1k2xv1BsEqR64Rbiv4B4irDbjBhCsnT8=; b=JgOdqSkLGotpDyrF+JYxLfKN3B3qfAh3l1m7DohXdlwkiNajCRUZ76SlcEJlmjne8z mji+2mKGS9qV23TXTK/EXwlPx8cuVzSbpRANn40I1XSeP+BumIhT+Z56g+jIZY9XyceG R1MPl3cRymecx5ud9bCtA6zaXNFeCH7jb87wX6aQeOf7lMWYJIolmvGT9Je9Z/uuk3L8 FPcwjREHe5xprC2o6vfreTno90Q+9R6TrzWEw/S1M7nGOPH1iOUvk+KwgChCgLVpLtbi pi4rn+gDQ20uGczL6uH6btelSicyIJBFzGjuxI3DccGJUREb+ZMba947gTlnX4vedyyo el1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sQ4kjjEDKGZ1k2xv1BsEqR64Rbiv4B4irDbjBhCsnT8=; b=MVz/RgkYAHm/TsB7pkY/zkHD+Li2guMZ56W580AfPJGV80YoJGJji1qS5cmlk45k5U b4iBaLtd/zivjinkWJ+frUnks57Pdw43Gy2Hydikn0O9Sy87tixeZ1LCWfMTY+jtP3wf TfDa4o/XPWiqm3MobLZuZgog44RsyMboLI/WsxwSrytJPVYSgugkEvcDqj1zHcIZKh7a pnJqlE2qT+TIR3gv5zR3oP51F98/D9mmIl1blbsRQ/xdE3NutINvAbVqvcxOmjaGHiAk I3Y3zPqyIg3RErhi4bShXXbHPbU+mn1BMSWoDNYf4/g3rXWq4yaeUow5hJc1Z2XXqrM1 lCTA== X-Gm-Message-State: AJcUukfxuwLSuPKpyTUwr7IYvQL3iILY2NBPLHtW7TFWFJSvNE96FQfN ZNcqQ1okxFirWeXFKjptx9Z+qXP3 X-Google-Smtp-Source: ALg8bN4ntwx2zWVQ5ec3npRtJYKZxk/0aD5iKoUJ+GeeOLfK5aoRRi/E/lwXn+dsAgObLEIUJh1qbw== X-Received: by 2002:a5d:6b81:: with SMTP id n1mr6076158wrx.149.1545472253969; Sat, 22 Dec 2018 01:50:53 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:53 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:15 +0200 Message-Id: <20181222095036.29743-11-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 10/31] hw/pvrdma: Set the correct opcode for send completion X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia opcode for WC should be set by the device and not taken from work element. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_qp_ops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 7b0f440fda..3388be1926 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -154,7 +154,7 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cq_handle = qp->send_cq_handle; comp_ctx->cqe.wr_id = wqe->hdr.wr_id; comp_ctx->cqe.qp = qp_handle; - comp_ctx->cqe.opcode = wqe->hdr.opcode; + comp_ctx->cqe.opcode = IBV_WC_SEND; rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.num_sge, From patchwork Sat Dec 22 09:50:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017810 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="p/evmPEa"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLT70bJJz9sPl for ; Sat, 22 Dec 2018 20:55:06 +1100 (AEDT) Received: from localhost ([::1]:41425 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadzs-0000Z8-H7 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:55:04 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41638) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvt-0006Br-RS for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvs-0006Oc-Q5 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:57 -0500 Received: from mail-wm1-x344.google.com ([2a00:1450:4864:20::344]:34962) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvs-0006O8-Kf for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:56 -0500 Received: by mail-wm1-x344.google.com with SMTP id c126so7865756wmh.0 for ; Sat, 22 Dec 2018 01:50:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NQqFg5YEyp8lulCHa6h8Lw0WrMYf6IBBChJyy+CJT/Y=; b=p/evmPEaZHZnZd3J0AutAfKnHQeK9QamW2voai0Op0+RV17f29VrnYfqkKgdNDSy61 E6y+PjAszvCKSJhp9M7y6Nt0TWIG6pnUKPyNcOx8BFghSWE05z23NgPs6weukNJX2FkU w84ZKgNTDGliOXug0Z6eHMKHEyDNnzzOiwnfEvPr6zyHsmDN1wGctas7pijcEvdw19H1 ucmptiBNy2Fv3Dr3685CAL8fh5KoWCOQ9ZHuA1gBZECQSm/fvZfJ2i1izBLU5Uuaf5GQ 0ZpPfVJNhcNqxoAX7xV5vZxgE+xIvHl4/iM8H4UgLdxAzZwI3pDblKQQ703oDDZN5uGQ a0ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NQqFg5YEyp8lulCHa6h8Lw0WrMYf6IBBChJyy+CJT/Y=; b=Lyq4Dg3WyFZLHd2cTpnFifzh9zHt7TpzVRboEWRiXx6p5tE6Z4NpM+/SM5DdHfLdyo cVcuyVQY/f0o9+43n3e6j8pueIVWvpVtJFRRNAtS9xOa+S4EM2ULW15nrBXSMW8he8u7 q6MAbZQvg1DY7wKRmayBtAISeq5yZ+fjdm8VoYajJNYb58x1uo51KZV7SDQXDpbZlVTx b4xzsrbNuapt+bNsWCB1ltrarpXpvlD7bgGMuf1bmpkWtWhv1laaLhqk5Z1g62oY0Jfu MMiCuUNkqnd4Tc9PAmfoeX6CJREO8zlm7p36J/zOYZwUvBoo7XorluTqZI188KybkEUj 4M8Q== X-Gm-Message-State: AJcUukeOWO5sWTC7+y6hJoKBnKVW4+CbtUlNtRvPHhM2QbmXyV7DACwf Hc+liKvIyHgqAnfexJsxl4q4CW5J X-Google-Smtp-Source: ALg8bN79UPe2TECkNhEYr9D895xKRtDyiGM0ckLi7UFeObPr05M0l9pzMaODcgFppHr6Jiebmpo6Bg== X-Received: by 2002:a1c:f916:: with SMTP id x22mr6209868wmh.87.1545472255376; Sat, 22 Dec 2018 01:50:55 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.54 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:54 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:16 +0200 Message-Id: <20181222095036.29743-12-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::344 Subject: [Qemu-devel] [PATCH PULL 11/31] qapi: Define new QMP message for pvrdma X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia pvrdma requires that the same GID attached to it will be attached to the backend device in the host. A new QMP messages is defined so pvrdma device can broadcast any change made to its GID table. This event is captured by libvirt which in turn will update the GID table in the backend device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Acked-by: Markus Armbruster Signed-off-by: Marcel Apfelbaum --- MAINTAINERS | 1 + Makefile.objs | 3 ++- qapi/qapi-schema.json | 1 + qapi/rdma.json | 38 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 qapi/rdma.json diff --git a/MAINTAINERS b/MAINTAINERS index 856d379b0a..180695f5d3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2413,6 +2413,7 @@ F: hw/rdma/* F: hw/rdma/vmw/* F: docs/pvrdma.txt F: contrib/rdmacm-mux/* +F: qapi/rdma.json Build and test automation ------------------------- diff --git a/Makefile.objs b/Makefile.objs index 319f14d937..bc5b8a8442 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -1,5 +1,6 @@ QAPI_MODULES = block-core block char common crypto introspect job migration -QAPI_MODULES += misc net rocker run-state sockets tpm trace transaction ui +QAPI_MODULES += misc net rdma rocker run-state sockets tpm trace transaction +QAPI_MODULES += ui ####################################################################### # Common libraries for tools and emulators diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json index 65b6dc2f6f..3bbdfcee84 100644 --- a/qapi/qapi-schema.json +++ b/qapi/qapi-schema.json @@ -86,6 +86,7 @@ { 'include': 'char.json' } { 'include': 'job.json' } { 'include': 'net.json' } +{ 'include': 'rdma.json' } { 'include': 'rocker.json' } { 'include': 'tpm.json' } { 'include': 'ui.json' } diff --git a/qapi/rdma.json b/qapi/rdma.json new file mode 100644 index 0000000000..b58105b1b6 --- /dev/null +++ b/qapi/rdma.json @@ -0,0 +1,38 @@ +# -*- Mode: Python -*- +# + +## +# = RDMA device +## + +## +# @RDMA_GID_STATUS_CHANGED: +# +# Emitted when guest driver adds/deletes GID to/from device +# +# @netdev: RoCE Network Device name +# +# @gid-status: Add or delete indication +# +# @subnet-prefix: Subnet Prefix +# +# @interface-id : Interface ID +# +# Since: 4.0 +# +# Example: +# +# <- {"timestamp": {"seconds": 1541579657, "microseconds": 986760}, +# "event": "RDMA_GID_STATUS_CHANGED", +# "data": +# {"netdev": "bridge0", +# "interface-id": 15880512517475447892, +# "gid-status": true, +# "subnet-prefix": 33022}} +# +## +{ 'event': 'RDMA_GID_STATUS_CHANGED', + 'data': { 'netdev' : 'str', + 'gid-status' : 'bool', + 'subnet-prefix' : 'uint64', + 'interface-id' : 'uint64' } } From patchwork Sat Dec 22 09:50:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017820 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="gu8K7cDE"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLgw297sz9sPf for ; Sat, 22 Dec 2018 21:04:28 +1100 (AEDT) Received: from localhost ([::1]:53421 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae8v-0007xH-Oy for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:04:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41685) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvy-0006GM-NZ for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:06 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvv-0006QR-51 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:02 -0500 Received: from mail-wm1-x344.google.com ([2a00:1450:4864:20::344]:55940) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvu-0006Pz-Om for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:59 -0500 Received: by mail-wm1-x344.google.com with SMTP id y139so7336541wmc.5 for ; Sat, 22 Dec 2018 01:50:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Quo4FY8Qr81bY21OfHSJ2xnGsZB6JxikRDZj7nqppMs=; b=gu8K7cDEGwDgBX5jrwMAT9tvZ5R8srt8D48pI77fwVPYLowjCA83XsyHLlaF4jpSjT ORmaslyGvPYDIjl1DgUYnP5i6mNtaA7PyfOewUwrCFKVdT+dZrt648RQoU4V2g12FOo5 kr1Fs6Ve7G8X2Y1W6kACfkBb5MjBDh3uIEdKIDds6R+ac/nUhvu03cJHKzZxXZex44Qh VQw6EAsuO1VkAvJx/RnmPe21btCOVOYguy+t2g+HH+TBytU/z8nzr+6rTcxURzYWCIMj Qr0KAS1RwSLyK/ftce2T291mZQ7smn4QCb8qvt8FUk83/weFH5Tzw3lJVEvAUlIYooHW vViw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Quo4FY8Qr81bY21OfHSJ2xnGsZB6JxikRDZj7nqppMs=; b=S528df7sL5wgOhJFgJiyXmZjpkS00B8FbjNk5zop1P4ieTzUdphfW3EBGywsTQeNkI MjQ1elnTJP0IgKXG5iBlboElOhyYgF+KEcNFQzj94vy3+NTgtgz+89pVfEYitYNLC3Wj FjmWMgR7mME2swr1SBqr0sPKd4NgxKDJw1JMpo4fhDIJ7RsRmKFOPuyJionK3xTtEyUp LHJV9C+pvburfX7USNjSVeIBb0QyMAHb4ekK3Hm9MrUaZTG3rwdP5ZHJM+5RaN5TLz/5 5WFiN+UyX2nvD3tdXbNdyJQYBfBsov9IwuVN1GfI8Ol/0fRvNXt/LmYaM/G4PuiJXJnq h2hg== X-Gm-Message-State: AJcUukfgI1pdJNXs3f8Osba20pkS67Mx93jgof75kIKMwUvmZJrvUb16 PcRscsALCJKZTm+TwXGZcnazirVs X-Google-Smtp-Source: ALg8bN4Entzt/WU0QbVdECjer+w3RPi0wgYMgp5Pf/6dlQhHpDb/DYEhKl+jXcuemsFXgie8Kx0KJg== X-Received: by 2002:a1c:888d:: with SMTP id k135mr6155437wmd.137.1545472257027; Sat, 22 Dec 2018 01:50:57 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.55 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:56 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:17 +0200 Message-Id: <20181222095036.29743-13-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::344 Subject: [Qemu-devel] [PATCH PULL 12/31] hw/pvrdma: Add support to allow guest to configure GID table X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia The control over the RDMA device's GID table is done by updating the device's Ethernet function addresses. Usually the first GID entry is determined by the MAC address, the second by the first IPv6 address and the third by the IPv4 address. Other entries can be added by adding more IP addresses. The opposite is the same, i.e. whenever an address is removed, the corresponding GID entry is removed. The process is done by the network and RDMA stacks. Whenever an address is added the ib_core driver is notified and calls the device driver add_gid function which in turn update the device. To support this in pvrdma device we need to hook into the create_bind and destroy_bind HW commands triggered by pvrdma driver in guest. Whenever a change is made to the pvrdma port's GID table a special QMP message is sent to be processed by libvirt to update the address of the backend Ethernet device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 344 +++++++++++++++++++++++++----------- hw/rdma/rdma_backend.h | 22 +-- hw/rdma/rdma_backend_defs.h | 11 +- hw/rdma/rdma_rm.c | 104 ++++++++++- hw/rdma/rdma_rm.h | 17 +- hw/rdma/rdma_rm_defs.h | 9 +- hw/rdma/rdma_utils.h | 16 ++ hw/rdma/vmw/pvrdma.h | 2 +- hw/rdma/vmw/pvrdma_cmd.c | 55 +++--- hw/rdma/vmw/pvrdma_main.c | 25 +-- hw/rdma/vmw/pvrdma_qp_ops.c | 20 +++ 11 files changed, 462 insertions(+), 163 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index c6dedda555..1d496bbd95 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -15,15 +15,18 @@ #include "qemu/osdep.h" #include "qemu/error-report.h" +#include "sysemu/sysemu.h" #include "qapi/error.h" #include "qapi/qmp/qlist.h" #include "qapi/qmp/qnum.h" +#include "qapi/qapi-events-rdma.h" #include #include #include #include +#include "contrib/rdmacm-mux/rdmacm-mux.h" #include "trace.h" #include "rdma_utils.h" #include "rdma_rm.h" @@ -160,6 +163,77 @@ static void *comp_handler_thread(void *arg) return NULL; } +static inline void disable_rdmacm_mux_async(RdmaBackendDev *backend_dev) +{ + atomic_set(&backend_dev->rdmacm_mux.can_receive, 0); +} + +static inline void enable_rdmacm_mux_async(RdmaBackendDev *backend_dev) +{ + atomic_set(&backend_dev->rdmacm_mux.can_receive, sizeof(RdmaCmMuxMsg)); +} + +static inline int rdmacm_mux_can_process_async(RdmaBackendDev *backend_dev) +{ + return atomic_read(&backend_dev->rdmacm_mux.can_receive); +} + +static int check_mux_op_status(CharBackend *mad_chr_be) +{ + RdmaCmMuxMsg msg = {0}; + int ret; + + pr_dbg("Reading response\n"); + ret = qemu_chr_fe_read_all(mad_chr_be, (uint8_t *)&msg, sizeof(msg)); + if (ret != sizeof(msg)) { + pr_dbg("Invalid message size %d, expecting %ld\n", ret, sizeof(msg)); + return -EIO; + } + + pr_dbg("msg_type=%d\n", msg.hdr.msg_type); + pr_dbg("op_code=%d\n", msg.hdr.op_code); + pr_dbg("err_code=%d\n", msg.hdr.err_code); + + if (msg.hdr.msg_type != RDMACM_MUX_MSG_TYPE_RESP) { + pr_dbg("Invalid message type %d\n", msg.hdr.msg_type); + return -EIO; + } + + if (msg.hdr.err_code != RDMACM_MUX_ERR_CODE_OK) { + pr_dbg("Operation failed in mux, error code %d\n", msg.hdr.err_code); + return -EIO; + } + + return 0; +} + +static int exec_rdmacm_mux_req(RdmaBackendDev *backend_dev, RdmaCmMuxMsg *msg) +{ + int rc = 0; + + pr_dbg("Executing request %d\n", msg->hdr.op_code); + + msg->hdr.msg_type = RDMACM_MUX_MSG_TYPE_REQ; + disable_rdmacm_mux_async(backend_dev); + rc = qemu_chr_fe_write(backend_dev->rdmacm_mux.chr_be, + (const uint8_t *)msg, sizeof(*msg)); + if (rc != sizeof(*msg)) { + enable_rdmacm_mux_async(backend_dev); + pr_dbg("Fail to send request to rdmacm_mux (rc=%d)\n", rc); + return -EIO; + } + + rc = check_mux_op_status(backend_dev->rdmacm_mux.chr_be); + if (rc) { + pr_dbg("Fail to execute rdmacm_mux request %d (rc=%d)\n", + msg->hdr.op_code, rc); + } + + enable_rdmacm_mux_async(backend_dev); + + return 0; +} + static void stop_backend_thread(RdmaBackendThread *thread) { thread->run = false; @@ -300,11 +374,11 @@ static int build_host_sge_array(RdmaDeviceResources *rdma_dev_res, return 0; } -static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, - uint32_t num_sge) +static int mad_send(RdmaBackendDev *backend_dev, uint8_t sgid_idx, + union ibv_gid *sgid, struct ibv_sge *sge, uint32_t num_sge) { - struct backend_umad umad = {0}; - char *hdr, *msg; + RdmaCmMuxMsg msg = {0}; + char *hdr, *data; int ret; pr_dbg("num_sge=%d\n", num_sge); @@ -313,26 +387,31 @@ static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, return -EINVAL; } - umad.hdr.length = sge[0].length + sge[1].length; - pr_dbg("msg_len=%d\n", umad.hdr.length); + msg.hdr.op_code = RDMACM_MUX_OP_CODE_MAD; + memcpy(msg.hdr.sgid.raw, sgid->raw, sizeof(msg.hdr.sgid)); - if (umad.hdr.length > sizeof(umad.mad)) { + msg.umad_len = sge[0].length + sge[1].length; + pr_dbg("umad_len=%d\n", msg.umad_len); + + if (msg.umad_len > sizeof(msg.umad.mad)) { return -ENOMEM; } - umad.hdr.addr.qpn = htobe32(1); - umad.hdr.addr.grh_present = 1; - umad.hdr.addr.gid_index = backend_dev->backend_gid_idx; - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.gid)); - umad.hdr.addr.hop_limit = 0xFF; + msg.umad.hdr.addr.qpn = htobe32(1); + msg.umad.hdr.addr.grh_present = 1; + pr_dbg("sgid_idx=%d\n", sgid_idx); + pr_dbg("sgid=0x%llx\n", sgid->global.interface_id); + msg.umad.hdr.addr.gid_index = sgid_idx; + memcpy(msg.umad.hdr.addr.gid, sgid->raw, sizeof(msg.umad.hdr.addr.gid)); + msg.umad.hdr.addr.hop_limit = 0xFF; hdr = rdma_pci_dma_map(backend_dev->dev, sge[0].addr, sge[0].length); if (!hdr) { pr_dbg("Fail to map to sge[0]\n"); return -ENOMEM; } - msg = rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); - if (!msg) { + data = rdma_pci_dma_map(backend_dev->dev, sge[1].addr, sge[1].length); + if (!data) { pr_dbg("Fail to map to sge[1]\n"); rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); return -ENOMEM; @@ -341,25 +420,27 @@ static int mad_send(RdmaBackendDev *backend_dev, struct ibv_sge *sge, pr_dbg_buf("mad_hdr", hdr, sge[0].length); pr_dbg_buf("mad_data", data, sge[1].length); - memcpy(&umad.mad[0], hdr, sge[0].length); - memcpy(&umad.mad[sge[0].length], msg, sge[1].length); + memcpy(&msg.umad.mad[0], hdr, sge[0].length); + memcpy(&msg.umad.mad[sge[0].length], data, sge[1].length); - rdma_pci_dma_unmap(backend_dev->dev, msg, sge[1].length); + rdma_pci_dma_unmap(backend_dev->dev, data, sge[1].length); rdma_pci_dma_unmap(backend_dev->dev, hdr, sge[0].length); - ret = qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&umad, - sizeof(umad)); + ret = exec_rdmacm_mux_req(backend_dev, &msg); + if (ret) { + pr_dbg("Fail to send MAD to rdma_umadmux (%d)\n", ret); + return -EIO; + } - pr_dbg("qemu_chr_fe_write=%d\n", ret); - - return (ret != sizeof(umad)); + return 0; } void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, - union ibv_gid *dgid, uint32_t dqpn, - uint32_t dqkey, void *ctx) + uint8_t sgid_idx, union ibv_gid *sgid, + union ibv_gid *dgid, uint32_t dqpn, uint32_t dqkey, + void *ctx) { BackendCtx *bctx; struct ibv_sge new_sge[MAX_SGE]; @@ -373,7 +454,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type == IBV_QPT_GSI) { pr_dbg("QP1\n"); - rc = mad_send(backend_dev, sge, num_sge); + rc = mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); } else { @@ -409,8 +490,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, } if (qp_type == IBV_QPT_UD) { - wr.wr.ud.ah = create_ah(backend_dev, qp->ibpd, - backend_dev->backend_gid_idx, dgid); + wr.wr.ud.ah = create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; @@ -715,9 +795,9 @@ int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, } int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qkey, - bool use_qkey) + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qkey) { struct ibv_qp_attr attr = {0}; union ibv_gid ibv_gid = { @@ -729,13 +809,15 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, attr.qp_state = IBV_QPS_RTR; attr_mask = IBV_QP_STATE; + qp->sgid_idx = sgid_idx; + switch (qp_type) { case IBV_QPT_RC: pr_dbg("dgid=0x%" PRIx64 ",%" PRIx64 "\n", be64_to_cpu(ibv_gid.global.subnet_prefix), be64_to_cpu(ibv_gid.global.interface_id)); pr_dbg("dqpn=0x%x\n", dqpn); - pr_dbg("sgid_idx=%d\n", backend_dev->backend_gid_idx); + pr_dbg("sgid_idx=%d\n", qp->sgid_idx); pr_dbg("sport_num=%d\n", backend_dev->port_num); pr_dbg("rq_psn=0x%x\n", rq_psn); @@ -747,7 +829,7 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, attr.ah_attr.is_global = 1; attr.ah_attr.grh.hop_limit = 1; attr.ah_attr.grh.dgid = ibv_gid; - attr.ah_attr.grh.sgid_index = backend_dev->backend_gid_idx; + attr.ah_attr.grh.sgid_index = qp->sgid_idx; attr.rq_psn = rq_psn; attr_mask |= IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN | @@ -756,8 +838,8 @@ int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, break; case IBV_QPT_UD: + pr_dbg("qkey=0x%x\n", qkey); if (use_qkey) { - pr_dbg("qkey=0x%x\n", qkey); attr.qkey = qkey; attr_mask |= IBV_QP_QKEY; } @@ -873,29 +955,19 @@ static inline void build_mad_hdr(struct ibv_grh *grh, union ibv_gid *sgid, grh->dgid = *my_gid; pr_dbg("paylen=%d (net=0x%x)\n", paylen, grh->paylen); - pr_dbg("my_gid=0x%llx\n", my_gid->global.interface_id); - pr_dbg("gid=0x%llx\n", sgid->global.interface_id); + pr_dbg("dgid=0x%llx\n", my_gid->global.interface_id); + pr_dbg("sgid=0x%llx\n", sgid->global.interface_id); } -static inline int mad_can_receieve(void *opaque) +static void process_incoming_mad_req(RdmaBackendDev *backend_dev, + RdmaCmMuxMsg *msg) { - return sizeof(struct backend_umad); -} - -static void mad_read(void *opaque, const uint8_t *buf, int size) -{ - RdmaBackendDev *backend_dev = (RdmaBackendDev *)opaque; QObject *o_ctx_id; unsigned long cqe_ctx_id; BackendCtx *bctx; char *mad; - struct backend_umad *umad; - assert(size != sizeof(umad)); - umad = (struct backend_umad *)buf; - - pr_dbg("Got %d bytes\n", size); - pr_dbg("umad->hdr.length=%d\n", umad->hdr.length); + pr_dbg("umad_len=%d\n", msg->umad_len); #ifdef PVRDMA_DEBUG struct umad_hdr *hdr = (struct umad_hdr *)&msg->umad.mad; @@ -925,15 +997,16 @@ static void mad_read(void *opaque, const uint8_t *buf, int size) mad = rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); - if (!mad || bctx->sge.length < umad->hdr.length + MAD_HDR_SIZE) { + if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, bctx->up_ctx); } else { + pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, - (union ibv_gid *)&umad->hdr.addr.gid, - &backend_dev->gid, umad->hdr.length); - memcpy(&mad[MAD_HDR_SIZE], umad->mad, umad->hdr.length); + (union ibv_gid *)&msg->umad.hdr.addr.gid, &msg->hdr.sgid, + msg->umad_len); + memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); @@ -943,56 +1016,151 @@ static void mad_read(void *opaque, const uint8_t *buf, int size) rdma_rm_dealloc_cqe_ctx(backend_dev->rdma_dev_res, cqe_ctx_id); } -static int mad_init(RdmaBackendDev *backend_dev) +static inline int rdmacm_mux_can_receive(void *opaque) +{ + RdmaBackendDev *backend_dev = (RdmaBackendDev *)opaque; + + return rdmacm_mux_can_process_async(backend_dev); +} + +static void rdmacm_mux_read(void *opaque, const uint8_t *buf, int size) +{ + RdmaBackendDev *backend_dev = (RdmaBackendDev *)opaque; + RdmaCmMuxMsg *msg = (RdmaCmMuxMsg *)buf; + + pr_dbg("Got %d bytes\n", size); + pr_dbg("msg_type=%d\n", msg->hdr.msg_type); + pr_dbg("op_code=%d\n", msg->hdr.op_code); + + if (msg->hdr.msg_type != RDMACM_MUX_MSG_TYPE_REQ && + msg->hdr.op_code != RDMACM_MUX_OP_CODE_MAD) { + pr_dbg("Error: Not a MAD request, skipping\n"); + return; + } + process_incoming_mad_req(backend_dev, msg); +} + +static int mad_init(RdmaBackendDev *backend_dev, CharBackend *mad_chr_be) { - struct backend_umad umad = {0}; int ret; - if (!qemu_chr_fe_backend_connected(backend_dev->mad_chr_be)) { + backend_dev->rdmacm_mux.chr_be = mad_chr_be; + + ret = qemu_chr_fe_backend_connected(backend_dev->rdmacm_mux.chr_be); + if (!ret) { pr_dbg("Missing chardev for MAD multiplexer\n"); return -EIO; } - qemu_chr_fe_set_handlers(backend_dev->mad_chr_be, mad_can_receieve, - mad_read, NULL, NULL, backend_dev, NULL, true); - - /* Register ourself */ - memcpy(umad.hdr.addr.gid, backend_dev->gid.raw, sizeof(umad.hdr.addr.gid)); - ret = qemu_chr_fe_write(backend_dev->mad_chr_be, (const uint8_t *)&umad, - sizeof(umad.hdr)); - if (ret != sizeof(umad.hdr)) { - pr_dbg("Fail to register to rdma_umadmux (%d)\n", ret); - } - qemu_mutex_init(&backend_dev->recv_mads_list.lock); backend_dev->recv_mads_list.list = qlist_new(); + enable_rdmacm_mux_async(backend_dev); + + qemu_chr_fe_set_handlers(backend_dev->rdmacm_mux.chr_be, + rdmacm_mux_can_receive, rdmacm_mux_read, NULL, + NULL, backend_dev, NULL, true); + return 0; } static void mad_fini(RdmaBackendDev *backend_dev) { + pr_dbg("Stopping MAD\n"); + disable_rdmacm_mux_async(backend_dev); + qemu_chr_fe_disconnect(backend_dev->rdmacm_mux.chr_be); qlist_destroy_obj(QOBJECT(backend_dev->recv_mads_list.list)); qemu_mutex_destroy(&backend_dev->recv_mads_list.lock); } +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid) +{ + union ibv_gid sgid; + int ret; + int i = 0; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + do { + ret = ibv_query_gid(backend_dev->context, backend_dev->port_num, i, + &sgid); + i++; + } while (!ret && (memcmp(&sgid, gid, sizeof(*gid)))); + + pr_dbg("gid_index=%d\n", i - 1); + + return ret ? ret : i - 1; +} + +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg = {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.op_code = RDMACM_MUX_OP_CODE_REG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + + ret = exec_rdmacm_mux_req(backend_dev, &msg); + if (ret) { + pr_dbg("Fail to register GID to rdma_umadmux (%d)\n", ret); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, true, + gid->global.subnet_prefix, + gid->global.interface_id); + + return ret; +} + +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid) +{ + RdmaCmMuxMsg msg = {0}; + int ret; + + pr_dbg("0x%llx, 0x%llx\n", + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); + + msg.hdr.op_code = RDMACM_MUX_OP_CODE_UNREG; + memcpy(msg.hdr.sgid.raw, gid->raw, sizeof(msg.hdr.sgid)); + + ret = exec_rdmacm_mux_req(backend_dev, &msg); + if (ret) { + pr_dbg("Fail to unregister GID from rdma_umadmux (%d)\n", ret); + return -EIO; + } + + qapi_event_send_rdma_gid_status_changed(ifname, false, + gid->global.subnet_prefix, + gid->global.interface_id); + + return 0; +} + int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr, - CharBackend *mad_chr_be, Error **errp) + struct ibv_device_attr *dev_attr, CharBackend *mad_chr_be, + Error **errp) { int i; int ret = 0; int num_ibv_devices; struct ibv_device **dev_list; - struct ibv_port_attr port_attr; memset(backend_dev, 0, sizeof(*backend_dev)); backend_dev->dev = pdev; - backend_dev->mad_chr_be = mad_chr_be; - backend_dev->backend_gid_idx = backend_gid_idx; backend_dev->port_num = port_num; backend_dev->rdma_dev_res = rdma_dev_res; @@ -1029,9 +1197,9 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, backend_dev->ib_dev = *dev_list; } - pr_dbg("Using backend device %s, port %d, gid_idx %d\n", - ibv_get_device_name(backend_dev->ib_dev), - backend_dev->port_num, backend_dev->backend_gid_idx); + pr_dbg("Using backend device %s, port %d\n", + ibv_get_device_name(backend_dev->ib_dev), backend_dev->port_num); + pr_dbg("uverb device %s\n", backend_dev->ib_dev->dev_name); backend_dev->context = ibv_open_device(backend_dev->ib_dev); if (!backend_dev->context) { @@ -1048,20 +1216,6 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, } pr_dbg("dev->backend_dev.channel=%p\n", backend_dev->channel); - ret = ibv_query_port(backend_dev->context, backend_dev->port_num, - &port_attr); - if (ret) { - error_setg(errp, "Error %d from ibv_query_port", ret); - ret = -EIO; - goto out_destroy_comm_channel; - } - - if (backend_dev->backend_gid_idx >= port_attr.gid_tbl_len) { - error_setg(errp, "Invalid backend_gid_idx, should be less than %d", - port_attr.gid_tbl_len); - goto out_destroy_comm_channel; - } - ret = init_device_caps(backend_dev, dev_attr); if (ret) { error_setg(errp, "Failed to initialize device capabilities"); @@ -1069,20 +1223,8 @@ int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, goto out_destroy_comm_channel; } - ret = ibv_query_gid(backend_dev->context, backend_dev->port_num, - backend_dev->backend_gid_idx, &backend_dev->gid); - if (ret) { - error_setg(errp, "Failed to query gid %d", - backend_dev->backend_gid_idx); - ret = -EIO; - goto out_destroy_comm_channel; - } - pr_dbg("subnet_prefix=0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.subnet_prefix)); - pr_dbg("interface_id=0x%" PRIx64 "\n", - be64_to_cpu(backend_dev->gid.global.interface_id)); - ret = mad_init(backend_dev); + ret = mad_init(backend_dev, mad_chr_be); if (ret) { error_setg(errp, "Fail to initialize mad"); ret = -EIO; diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index fc83330251..59ad2b874b 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -28,11 +28,6 @@ enum ibv_special_qp_type { IBV_QPT_GSI = 1, }; -static inline union ibv_gid *rdma_backend_gid(RdmaBackendDev *dev) -{ - return &dev->gid; -} - static inline uint32_t rdma_backend_qpn(const RdmaBackendQP *qp) { return qp->ibqp ? qp->ibqp->qp_num : 1; @@ -51,9 +46,15 @@ static inline uint32_t rdma_backend_mr_rkey(const RdmaBackendMR *mr) int rdma_backend_init(RdmaBackendDev *backend_dev, PCIDevice *pdev, RdmaDeviceResources *rdma_dev_res, const char *backend_device_name, uint8_t port_num, - uint8_t backend_gid_idx, struct ibv_device_attr *dev_attr, - CharBackend *mad_chr_be, Error **errp); + struct ibv_device_attr *dev_attr, CharBackend *mad_chr_be, + Error **errp); void rdma_backend_fini(RdmaBackendDev *backend_dev); +int rdma_backend_add_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_del_gid(RdmaBackendDev *backend_dev, const char *ifname, + union ibv_gid *gid); +int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, + union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); void rdma_backend_register_comp_handler(void (*handler)(int status, @@ -82,9 +83,9 @@ int rdma_backend_create_qp(RdmaBackendQP *qp, uint8_t qp_type, int rdma_backend_qp_state_init(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, uint32_t qkey); int rdma_backend_qp_state_rtr(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, - uint8_t qp_type, union ibv_gid *dgid, - uint32_t dqpn, uint32_t rq_psn, uint32_t qkey, - bool use_qkey); + uint8_t qp_type, uint8_t sgid_idx, + union ibv_gid *dgid, uint32_t dqpn, + uint32_t rq_psn, uint32_t qkey, bool use_qkey); int rdma_backend_qp_state_rts(RdmaBackendQP *qp, uint8_t qp_type, uint32_t sq_psn, uint32_t qkey, bool use_qkey); int rdma_backend_query_qp(RdmaBackendQP *qp, struct ibv_qp_attr *attr, @@ -94,6 +95,7 @@ void rdma_backend_destroy_qp(RdmaBackendQP *qp); void rdma_backend_post_send(RdmaBackendDev *backend_dev, RdmaBackendQP *qp, uint8_t qp_type, struct ibv_sge *sge, uint32_t num_sge, + uint8_t sgid_idx, union ibv_gid *sgid, union ibv_gid *dgid, uint32_t dqpn, uint32_t dqkey, void *ctx); void rdma_backend_post_recv(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/rdma_backend_defs.h b/hw/rdma/rdma_backend_defs.h index 2a7e667075..1e5c3dd3bf 100644 --- a/hw/rdma/rdma_backend_defs.h +++ b/hw/rdma/rdma_backend_defs.h @@ -19,6 +19,7 @@ #include "qemu/thread.h" #include "chardev/char-fe.h" #include +#include "contrib/rdmacm-mux/rdmacm-mux.h" typedef struct RdmaDeviceResources RdmaDeviceResources; @@ -34,19 +35,22 @@ typedef struct RecvMadList { QList *list; } RecvMadList; +typedef struct RdmaCmMux { + CharBackend *chr_be; + int can_receive; +} RdmaCmMux; + typedef struct RdmaBackendDev { struct ibv_device_attr dev_attr; RdmaBackendThread comp_thread; - union ibv_gid gid; PCIDevice *dev; RdmaDeviceResources *rdma_dev_res; struct ibv_device *ib_dev; struct ibv_context *context; struct ibv_comp_channel *channel; uint8_t port_num; - uint8_t backend_gid_idx; RecvMadList recv_mads_list; - CharBackend *mad_chr_be; + RdmaCmMux rdmacm_mux; } RdmaBackendDev; typedef struct RdmaBackendPD { @@ -66,6 +70,7 @@ typedef struct RdmaBackendCQ { typedef struct RdmaBackendQP { struct ibv_pd *ibpd; struct ibv_qp *ibqp; + uint8_t sgid_idx; } RdmaBackendQP; #endif diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 4f10fcabcc..250254561c 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -391,7 +391,7 @@ out_dealloc_qp: } int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid_idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn) @@ -400,6 +400,7 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, int ret; pr_dbg("qpn=0x%x\n", qp_handle); + pr_dbg("qkey=0x%x\n", qkey); qp = rdma_rm_get_qp(dev_res, qp_handle); if (!qp) { @@ -430,9 +431,19 @@ int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, } if (qp->qp_state == IBV_QPS_RTR) { + /* Get backend gid index */ + pr_dbg("Guest sgid_idx=%d\n", sgid_idx); + sgid_idx = rdma_rm_get_backend_gid_index(dev_res, backend_dev, + sgid_idx); + if (sgid_idx <= 0) { /* TODO check also less than bk.max_sgid */ + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", sgid_idx); + return -EIO; + } + ret = rdma_backend_qp_state_rtr(backend_dev, &qp->backend_qp, - qp->qp_type, dgid, dqpn, rq_psn, - qkey, attr_mask & IBV_QP_QKEY); + qp->qp_type, sgid_idx, dgid, dqpn, + rq_psn, qkey, + attr_mask & IBV_QP_QKEY); if (ret) { return -EIO; } @@ -523,11 +534,91 @@ void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id) res_tbl_dealloc(&dev_res->cqe_ctx_tbl, cqe_ctx_id); } +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname, union ibv_gid *gid, int gid_idx) +{ + int rc; + + rc = rdma_backend_add_gid(backend_dev, ifname, gid); + if (rc) { + pr_dbg("Fail to add gid\n"); + return -EINVAL; + } + + memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + + return 0; +} + +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname, int gid_idx) +{ + int rc; + + rc = rdma_backend_del_gid(backend_dev, ifname, + &dev_res->ports[0].gid_tbl[gid_idx].gid); + if (rc) { + pr_dbg("Fail to delete gid\n"); + return -EINVAL; + } + + memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); + dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index = -1; + + return 0; +} + +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_idx) +{ + if (unlikely(sgid_idx < 0 || sgid_idx > MAX_PORT_GIDS)) { + pr_dbg("Got invalid sgid_idx %d\n", sgid_idx); + return -EINVAL; + } + + if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index == -1)) { + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index = + rdma_backend_get_gid_index(backend_dev, + &dev_res->ports[0].gid_tbl[sgid_idx].gid); + } + + pr_dbg("backend_gid_index=%d\n", + dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + + return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; +} + static void destroy_qp_hash_key(gpointer data) { g_bytes_unref(data); } +static void init_ports(RdmaDeviceResources *dev_res) +{ + int i, j; + + memset(dev_res->ports, 0, sizeof(dev_res->ports)); + + for (i = 0; i < MAX_PORTS; i++) { + dev_res->ports[i].state = IBV_PORT_DOWN; + for (j = 0; j < MAX_PORT_GIDS; j++) { + dev_res->ports[i].gid_tbl[j].backend_gid_index = -1; + } + } +} + +static void fini_ports(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, const char *ifname) +{ + int i; + + dev_res->ports[0].state = IBV_PORT_DOWN; + for (i = 0; i < MAX_PORT_GIDS; i++) { + rdma_rm_del_gid(dev_res, backend_dev, ifname, i); + } +} + int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr, Error **errp) { @@ -545,11 +636,16 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr, dev_attr->max_qp_wr, sizeof(void *)); res_tbl_init("UC", &dev_res->uc_tbl, MAX_UCS, sizeof(RdmaRmUC)); + init_ports(dev_res); + return 0; } -void rdma_rm_fini(RdmaDeviceResources *dev_res) +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname) { + fini_ports(dev_res, backend_dev, ifname); + res_tbl_free(&dev_res->uc_tbl); res_tbl_free(&dev_res->cqe_ctx_tbl); res_tbl_free(&dev_res->qp_tbl); diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index b4e04cc7b4..a7169b4e89 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -22,7 +22,8 @@ int rdma_rm_init(RdmaDeviceResources *dev_res, struct ibv_device_attr *dev_attr, Error **errp); -void rdma_rm_fini(RdmaDeviceResources *dev_res); +void rdma_rm_fini(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname); int rdma_rm_alloc_pd(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, uint32_t *pd_handle, uint32_t ctx_handle); @@ -55,7 +56,7 @@ int rdma_rm_alloc_qp(RdmaDeviceResources *dev_res, uint32_t pd_handle, uint32_t recv_cq_handle, void *opaque, uint32_t *qpn); RdmaRmQP *rdma_rm_get_qp(RdmaDeviceResources *dev_res, uint32_t qpn); int rdma_rm_modify_qp(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, - uint32_t qp_handle, uint32_t attr_mask, + uint32_t qp_handle, uint32_t attr_mask, uint8_t sgid_idx, union ibv_gid *dgid, uint32_t dqpn, enum ibv_qp_state qp_state, uint32_t qkey, uint32_t rq_psn, uint32_t sq_psn); @@ -69,4 +70,16 @@ int rdma_rm_alloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t *cqe_ctx_id, void *rdma_rm_get_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id); void rdma_rm_dealloc_cqe_ctx(RdmaDeviceResources *dev_res, uint32_t cqe_ctx_id); +int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname, union ibv_gid *gid, int gid_idx); +int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, + const char *ifname, int gid_idx); +int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, + RdmaBackendDev *backend_dev, int sgid_idx); +static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, + int sgid_idx) +{ + return &dev_res->ports[0].gid_tbl[sgid_idx].gid; +} + #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 9b399063d3..7b3435f991 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -19,7 +19,7 @@ #include "rdma_backend_defs.h" #define MAX_PORTS 1 -#define MAX_PORT_GIDS 1 +#define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 #define MAX_PKEYS MAX_PORT_PKEYS @@ -86,8 +86,13 @@ typedef struct RdmaRmQP { enum ibv_qp_state qp_state; } RdmaRmQP; +typedef struct RdmaRmGid { + union ibv_gid gid; + int backend_gid_index; +} RdmaRmGid; + typedef struct RdmaRmPort { - union ibv_gid gid_tbl[MAX_PORT_GIDS]; + RdmaRmGid gid_tbl[MAX_PORT_GIDS]; enum ibv_port_state state; } RdmaRmPort; diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index c4f96c4f2a..062e2cd688 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -19,6 +19,7 @@ #include "hw/pci/pci.h" #include "sysemu/dma.h" +#include "stdio.h" #define pr_info(fmt, ...) \ fprintf(stdout, "%s: %-20s (%3d): " fmt, "rdma", __func__, __LINE__,\ @@ -39,9 +40,24 @@ extern unsigned long pr_dbg_cnt; #define pr_dbg(fmt, ...) \ fprintf(stdout, "%lx %ld: %-20s (%3d): " fmt, pthread_self(), pr_dbg_cnt++, \ __func__, __LINE__, ## __VA_ARGS__) + +#define pr_dbg_buf(title, buf, len) \ +{ \ + int i; \ + char *b = g_malloc0(len * 3 + 1); \ + char b1[4]; \ + for (i = 0; i < len; i++) { \ + sprintf(b1, "%.2X ", buf[i] & 0x000000FF); \ + strcat(b, b1); \ + } \ + pr_dbg("%s (%d): %s\n", title, len, b); \ + g_free(b); \ +} + #else #define init_pr_dbg(void) #define pr_dbg(fmt, ...) +#define pr_dbg_buf(title, buf, len) #endif void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 15c3f28b86..b019cb843a 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -79,8 +79,8 @@ typedef struct PVRDMADev { int interrupt_mask; struct ibv_device_attr dev_attr; uint64_t node_guid; + char *backend_eth_device_name; char *backend_device_name; - uint8_t backend_gid_idx; uint8_t backend_port_num; RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 57d6f41ae6..a334f6205e 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -504,13 +504,16 @@ static int modify_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, rsp->hdr.response = cmd->hdr.response; rsp->hdr.ack = PVRDMA_CMD_MODIFY_QP_RESP; - rsp->hdr.err = rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, - cmd->qp_handle, cmd->attr_mask, - (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, - cmd->attrs.dest_qp_num, - (enum ibv_qp_state)cmd->attrs.qp_state, - cmd->attrs.qkey, cmd->attrs.rq_psn, - cmd->attrs.sq_psn); + /* No need to verify sgid_index since it is u8 */ + + rsp->hdr.err = + rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, cmd->qp_handle, + cmd->attr_mask, cmd->attrs.ah_attr.grh.sgid_index, + (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, + cmd->attrs.dest_qp_num, + (enum ibv_qp_state)cmd->attrs.qp_state, + cmd->attrs.qkey, cmd->attrs.rq_psn, + cmd->attrs.sq_psn); pr_dbg("ret=%d\n", rsp->hdr.err); return rsp->hdr.err; @@ -570,10 +573,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { struct pvrdma_cmd_create_bind *cmd = &req->create_bind; -#ifdef PVRDMA_DEBUG - __be64 *subnet = (__be64 *)&cmd->new_gid[0]; - __be64 *if_id = (__be64 *)&cmd->new_gid[8]; -#endif + int rc; + union ibv_gid *gid = (union ibv_gid *)&cmd->new_gid; pr_dbg("index=%d\n", cmd->index); @@ -582,19 +583,24 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, } pr_dbg("gid[%d]=0x%llx,0x%llx\n", cmd->index, - (long long unsigned int)be64_to_cpu(*subnet), - (long long unsigned int)be64_to_cpu(*if_id)); + (long long unsigned int)be64_to_cpu(gid->global.subnet_prefix), + (long long unsigned int)be64_to_cpu(gid->global.interface_id)); - /* Driver forces to one port only */ - memcpy(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, &cmd->new_gid, - sizeof(cmd->new_gid)); + rc = rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, gid, cmd->index); + if (rc < 0) { + return -EINVAL; + } /* TODO: Since drivers stores node_guid at load_dsr phase then this * assignment is not relevant, i need to figure out a way how to * retrieve MAC of our netdev */ - dev->node_guid = dev->rdma_dev_res.ports[0].gid_tbl[0].global.interface_id; - pr_dbg("dev->node_guid=0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); + if (!cmd->index) { + dev->node_guid = + dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; + pr_dbg("dev->node_guid=0x%llx\n", + (long long unsigned int)be64_to_cpu(dev->node_guid)); + } return 0; } @@ -602,6 +608,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { + int rc; + struct pvrdma_cmd_destroy_bind *cmd = &req->destroy_bind; pr_dbg("index=%d\n", cmd->index); @@ -610,8 +618,13 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, return -EINVAL; } - memset(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw, 0, - sizeof(dev->rdma_dev_res.ports[0].gid_tbl[cmd->index].raw)); + rc = rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name, cmd->index); + + if (rc < 0) { + rsp->hdr.err = rc; + goto out; + } return 0; } diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index fc2abd34af..ac8c092db0 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -36,9 +36,9 @@ #include "pvrdma_qp_ops.h" static Property pvrdma_dev_properties[] = { - DEFINE_PROP_STRING("backend-dev", PVRDMADev, backend_device_name), - DEFINE_PROP_UINT8("backend-port", PVRDMADev, backend_port_num, 1), - DEFINE_PROP_UINT8("backend-gid-idx", PVRDMADev, backend_gid_idx, 0), + DEFINE_PROP_STRING("netdev", PVRDMADev, backend_eth_device_name), + DEFINE_PROP_STRING("ibdev", PVRDMADev, backend_device_name), + DEFINE_PROP_UINT8("ibport", PVRDMADev, backend_port_num, 1), DEFINE_PROP_UINT64("dev-caps-max-mr-size", PVRDMADev, dev_attr.max_mr_size, MAX_MR_SIZE), DEFINE_PROP_INT32("dev-caps-max-qp", PVRDMADev, dev_attr.max_qp, MAX_QP), @@ -276,17 +276,6 @@ static void init_dsr_dev_caps(PVRDMADev *dev) pr_dbg("Initialized\n"); } -static void init_ports(PVRDMADev *dev, Error **errp) -{ - int i; - - memset(dev->rdma_dev_res.ports, 0, sizeof(dev->rdma_dev_res.ports)); - - for (i = 0; i < MAX_PORTS; i++) { - dev->rdma_dev_res.ports[i].state = IBV_PORT_DOWN; - } -} - static void uninit_msix(PCIDevice *pdev, int used_vectors) { PVRDMADev *dev = PVRDMA_DEV(pdev); @@ -335,7 +324,8 @@ static void pvrdma_fini(PCIDevice *pdev) pvrdma_qp_ops_fini(); - rdma_rm_fini(&dev->rdma_dev_res); + rdma_rm_fini(&dev->rdma_dev_res, &dev->backend_dev, + dev->backend_eth_device_name); rdma_backend_fini(&dev->backend_dev); @@ -612,8 +602,7 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) rc = rdma_backend_init(&dev->backend_dev, pdev, &dev->rdma_dev_res, dev->backend_device_name, dev->backend_port_num, - dev->backend_gid_idx, &dev->dev_attr, &dev->mad_chr, - errp); + &dev->dev_attr, &dev->mad_chr, errp); if (rc) { goto out; } @@ -623,8 +612,6 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) goto out; } - init_ports(dev, errp); - rc = pvrdma_qp_ops_init(); if (rc) { goto out; diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 3388be1926..2130824098 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -131,6 +131,8 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) RdmaRmQP *qp; PvrdmaSqWqe *wqe; PvrdmaRing *ring; + int sgid_idx; + union ibv_gid *sgid; pr_dbg("qp_handle=0x%x\n", qp_handle); @@ -156,8 +158,26 @@ int pvrdma_qp_send(PVRDMADev *dev, uint32_t qp_handle) comp_ctx->cqe.qp = qp_handle; comp_ctx->cqe.opcode = IBV_WC_SEND; + sgid = rdma_rm_get_gid(&dev->rdma_dev_res, wqe->hdr.wr.ud.av.gid_index); + if (!sgid) { + pr_dbg("Fail to get gid for idx %d\n", wqe->hdr.wr.ud.av.gid_index); + return -EIO; + } + pr_dbg("sgid_id=%d, sgid=0x%llx\n", wqe->hdr.wr.ud.av.gid_index, + sgid->global.interface_id); + + sgid_idx = rdma_rm_get_backend_gid_index(&dev->rdma_dev_res, + &dev->backend_dev, + wqe->hdr.wr.ud.av.gid_index); + if (sgid_idx <= 0) { + pr_dbg("Fail to get bk sgid_idx for sgid_idx %d\n", + wqe->hdr.wr.ud.av.gid_index); + return -EIO; + } + rdma_backend_post_send(&dev->backend_dev, &qp->backend_qp, qp->qp_type, (struct ibv_sge *)&wqe->sge[0], wqe->hdr.num_sge, + sgid_idx, sgid, (union ibv_gid *)wqe->hdr.wr.ud.av.dgid, wqe->hdr.wr.ud.remote_qpn, wqe->hdr.wr.ud.remote_qkey, comp_ctx); From patchwork Sat Dec 22 09:50:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="T3S4QRAi"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLXc4vSyz9sPP for ; Sat, 22 Dec 2018 20:58:08 +1100 (AEDT) Received: from localhost ([::1]:45588 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae2o-0002xZ-Bk for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 04:58:06 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41663) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvx-0006Em-9a for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvv-0006Qq-TW for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:01 -0500 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]:38516) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvv-0006QS-Ng for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:50:59 -0500 Received: by mail-wm1-x343.google.com with SMTP id m22so7904224wml.3 for ; Sat, 22 Dec 2018 01:50:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YjDDlLJo+u56a7rKZS/C/FvZaZ86BdUwrm4YFa7Ayl4=; b=T3S4QRAiBG3SLyb6oKbP3xLxyAf1JgBqr/yVqV05kZ6fRIdp17ZjOC5AUEVeYDSCRT cLetfrS77gllGquDF5umrVi6KT5qkSCnBkFrToSosXCUztV6Kb7Ijz++0JpU1ruP7rAo vG7xX6oBkXvXGnPY/aonVXH8cyrJaQ2TVU1++acZbirlKJmu1VX1cUiXo1neZlF0EclQ Pqb0q4Rncanh7zsJ5yXFMQvFaWvx4OcEvy0cx1sug8disZIn4nkdU//qd0V+Qp6cuaAR dkVmejivx7cthnVk3uKnxohdVxjaibUfEN+8d7TGy7bbsFlJh0UZ0nKLKCDhc/7Sru3x cyqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YjDDlLJo+u56a7rKZS/C/FvZaZ86BdUwrm4YFa7Ayl4=; b=DeCXMEy/hJCBye/qbrXeRkJDVAsuIBOPG/YfplnccaRO4Vsih/6n8nZRm8z10Rqid1 +CeuW+HiD0zS9wtyUmLQWlacLrgcR5CQV2hnPj2WvGHPwrubrGNQyhDXu+DSfl0Ro+UE p5fzj4hwd9lEVl6ndhxNFszAGsYGVy0bK6RHmV29sIRQP13wATejUX13ioX8a6cx5XmR lL+A7yhZ54T05rUuITSxkxRC57zMNPQa1GWPpx7dCLGS03LRcAgyLNOzwDucABGERMRR J79UCVpdAgjtWC6nZEvAW9j1rqHoJP0wqXSLXS3IrGbAYGQsqvH6Hf2anAudCQCiCmQP Pwrw== X-Gm-Message-State: AA+aEWYIO4xu8cd7s00jiW0KKXethER6JIYQQHacK7Q0UpTAsI+WvkX9 v1npn5MN2zu4IgePKxJGLX7oPWvU X-Google-Smtp-Source: AFSGD/VsoFnfSzmi8UOkzRkrxy0mB1OPZvNqUwAWqap15dHmVVkFUx7sLCRd71ec5KtpynxCB2ORjw== X-Received: by 2002:a1c:e1d5:: with SMTP id y204mr5862921wmg.65.1545472258478; Sat, 22 Dec 2018 01:50:58 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.57 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:57 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:18 +0200 Message-Id: <20181222095036.29743-14-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::343 Subject: [Qemu-devel] [PATCH PULL 13/31] vmxnet3: Move some definitions to header file X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia pvrdma setup requires vmxnet3 device on PCI function 0 and PVRDMA device on PCI function 1. pvrdma device needs to access vmxnet3 device object for several reasons: 1. Make sure PCI function 0 is vmxnet3. 2. To monitor vmxnet3 device state. 3. To configure node_guid accoring to vmxnet3 device's MAC address. To be able to access vmxnet3 device the definition of VMXNET3State is moved to a new header file. Signed-off-by: Yuval Shaia Reviewed-by: Dmitry Fleytman Signed-off-by: Marcel Apfelbaum --- hw/net/vmxnet3.c | 116 +----------------------------------- hw/net/vmxnet3_defs.h | 133 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 134 insertions(+), 115 deletions(-) create mode 100644 hw/net/vmxnet3_defs.h diff --git a/hw/net/vmxnet3.c b/hw/net/vmxnet3.c index 3648630386..54746a4030 100644 --- a/hw/net/vmxnet3.c +++ b/hw/net/vmxnet3.c @@ -18,7 +18,6 @@ #include "qemu/osdep.h" #include "hw/hw.h" #include "hw/pci/pci.h" -#include "net/net.h" #include "net/tap.h" #include "net/checksum.h" #include "sysemu/sysemu.h" @@ -29,6 +28,7 @@ #include "migration/register.h" #include "vmxnet3.h" +#include "vmxnet3_defs.h" #include "vmxnet_debug.h" #include "vmware_utils.h" #include "net_tx_pkt.h" @@ -131,23 +131,11 @@ typedef struct VMXNET3Class { DeviceRealize parent_dc_realize; } VMXNET3Class; -#define TYPE_VMXNET3 "vmxnet3" -#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) - #define VMXNET3_DEVICE_CLASS(klass) \ OBJECT_CLASS_CHECK(VMXNET3Class, (klass), TYPE_VMXNET3) #define VMXNET3_DEVICE_GET_CLASS(obj) \ OBJECT_GET_CLASS(VMXNET3Class, (obj), TYPE_VMXNET3) -/* Cyclic ring abstraction */ -typedef struct { - hwaddr pa; - uint32_t size; - uint32_t cell_size; - uint32_t next; - uint8_t gen; -} Vmxnet3Ring; - static inline void vmxnet3_ring_init(PCIDevice *d, Vmxnet3Ring *ring, hwaddr pa, @@ -245,108 +233,6 @@ vmxnet3_dump_rx_descr(struct Vmxnet3_RxDesc *descr) descr->rsvd, descr->dtype, descr->ext1, descr->btype); } -/* Device state and helper functions */ -#define VMXNET3_RX_RINGS_PER_QUEUE (2) - -typedef struct { - Vmxnet3Ring tx_ring; - Vmxnet3Ring comp_ring; - - uint8_t intr_idx; - hwaddr tx_stats_pa; - struct UPT1_TxStats txq_stats; -} Vmxnet3TxqDescr; - -typedef struct { - Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; - Vmxnet3Ring comp_ring; - uint8_t intr_idx; - hwaddr rx_stats_pa; - struct UPT1_RxStats rxq_stats; -} Vmxnet3RxqDescr; - -typedef struct { - bool is_masked; - bool is_pending; - bool is_asserted; -} Vmxnet3IntState; - -typedef struct { - PCIDevice parent_obj; - NICState *nic; - NICConf conf; - MemoryRegion bar0; - MemoryRegion bar1; - MemoryRegion msix_bar; - - Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; - Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; - - /* Whether MSI-X support was installed successfully */ - bool msix_used; - hwaddr drv_shmem; - hwaddr temp_shared_guest_driver_memory; - - uint8_t txq_num; - - /* This boolean tells whether RX packet being indicated has to */ - /* be split into head and body chunks from different RX rings */ - bool rx_packets_compound; - - bool rx_vlan_stripping; - bool lro_supported; - - uint8_t rxq_num; - - /* Network MTU */ - uint32_t mtu; - - /* Maximum number of fragments for indicated TX packets */ - uint32_t max_tx_frags; - - /* Maximum number of fragments for indicated RX packets */ - uint16_t max_rx_frags; - - /* Index for events interrupt */ - uint8_t event_int_idx; - - /* Whether automatic interrupts masking enabled */ - bool auto_int_masking; - - bool peer_has_vhdr; - - /* TX packets to QEMU interface */ - struct NetTxPkt *tx_pkt; - uint32_t offload_mode; - uint32_t cso_or_gso_size; - uint16_t tci; - bool needs_vlan; - - struct NetRxPkt *rx_pkt; - - bool tx_sop; - bool skip_current_tx_pkt; - - uint32_t device_active; - uint32_t last_command; - - uint32_t link_status_and_speed; - - Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; - - uint32_t temp_mac; /* To store the low part first */ - - MACAddr perm_mac; - uint32_t vlan_table[VMXNET3_VFT_SIZE]; - uint32_t rx_mode; - MACAddr *mcast_list; - uint32_t mcast_list_len; - uint32_t mcast_list_buff_size; /* needed for live migration. */ - - /* Compatibility flags for migration */ - uint32_t compat_flags; -} VMXNET3State; - /* Interrupt management */ /* diff --git a/hw/net/vmxnet3_defs.h b/hw/net/vmxnet3_defs.h new file mode 100644 index 0000000000..6c19d29b12 --- /dev/null +++ b/hw/net/vmxnet3_defs.h @@ -0,0 +1,133 @@ +/* + * QEMU VMWARE VMXNET3 paravirtual NIC + * + * Copyright (c) 2012 Ravello Systems LTD (http://ravellosystems.com) + * + * Developed by Daynix Computing LTD (http://www.daynix.com) + * + * Authors: + * Dmitry Fleytman + * Tamir Shomer + * Yan Vugenfirer + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + * + */ + +#include "net/net.h" +#include "hw/net/vmxnet3.h" + +#define TYPE_VMXNET3 "vmxnet3" +#define VMXNET3(obj) OBJECT_CHECK(VMXNET3State, (obj), TYPE_VMXNET3) + +/* Device state and helper functions */ +#define VMXNET3_RX_RINGS_PER_QUEUE (2) + +/* Cyclic ring abstraction */ +typedef struct { + hwaddr pa; + uint32_t size; + uint32_t cell_size; + uint32_t next; + uint8_t gen; +} Vmxnet3Ring; + +typedef struct { + Vmxnet3Ring tx_ring; + Vmxnet3Ring comp_ring; + + uint8_t intr_idx; + hwaddr tx_stats_pa; + struct UPT1_TxStats txq_stats; +} Vmxnet3TxqDescr; + +typedef struct { + Vmxnet3Ring rx_ring[VMXNET3_RX_RINGS_PER_QUEUE]; + Vmxnet3Ring comp_ring; + uint8_t intr_idx; + hwaddr rx_stats_pa; + struct UPT1_RxStats rxq_stats; +} Vmxnet3RxqDescr; + +typedef struct { + bool is_masked; + bool is_pending; + bool is_asserted; +} Vmxnet3IntState; + +typedef struct { + PCIDevice parent_obj; + NICState *nic; + NICConf conf; + MemoryRegion bar0; + MemoryRegion bar1; + MemoryRegion msix_bar; + + Vmxnet3RxqDescr rxq_descr[VMXNET3_DEVICE_MAX_RX_QUEUES]; + Vmxnet3TxqDescr txq_descr[VMXNET3_DEVICE_MAX_TX_QUEUES]; + + /* Whether MSI-X support was installed successfully */ + bool msix_used; + hwaddr drv_shmem; + hwaddr temp_shared_guest_driver_memory; + + uint8_t txq_num; + + /* This boolean tells whether RX packet being indicated has to */ + /* be split into head and body chunks from different RX rings */ + bool rx_packets_compound; + + bool rx_vlan_stripping; + bool lro_supported; + + uint8_t rxq_num; + + /* Network MTU */ + uint32_t mtu; + + /* Maximum number of fragments for indicated TX packets */ + uint32_t max_tx_frags; + + /* Maximum number of fragments for indicated RX packets */ + uint16_t max_rx_frags; + + /* Index for events interrupt */ + uint8_t event_int_idx; + + /* Whether automatic interrupts masking enabled */ + bool auto_int_masking; + + bool peer_has_vhdr; + + /* TX packets to QEMU interface */ + struct NetTxPkt *tx_pkt; + uint32_t offload_mode; + uint32_t cso_or_gso_size; + uint16_t tci; + bool needs_vlan; + + struct NetRxPkt *rx_pkt; + + bool tx_sop; + bool skip_current_tx_pkt; + + uint32_t device_active; + uint32_t last_command; + + uint32_t link_status_and_speed; + + Vmxnet3IntState interrupt_states[VMXNET3_MAX_INTRS]; + + uint32_t temp_mac; /* To store the low part first */ + + MACAddr perm_mac; + uint32_t vlan_table[VMXNET3_VFT_SIZE]; + uint32_t rx_mode; + MACAddr *mcast_list; + uint32_t mcast_list_len; + uint32_t mcast_list_buff_size; /* needed for live migration. */ + + /* Compatibility flags for migration */ + uint32_t compat_flags; +} VMXNET3State; From patchwork Sat Dec 22 09:50:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017817 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="tR8TF+AV"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLcB2jdyz9sPd for ; Sat, 22 Dec 2018 21:01:14 +1100 (AEDT) Received: from localhost ([::1]:49749 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae5n-0005Zq-Qg for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:01:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41674) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvy-0006G4-9a for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvx-0006Rq-GM for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:02 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:40012) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvx-0006RT-Ad for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:01 -0500 Received: by mail-wm1-x342.google.com with SMTP id f188so7860807wmf.5 for ; Sat, 22 Dec 2018 01:51:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+lbu8Whjdwd0P/q1lJ6uz6K3myvpLKWj70uidx29BqE=; b=tR8TF+AVK4fT4ihTB6xs1ZvIIGQsMDDvTQOCHJnrkW0UwOxdf46WzjbdgMGVkj+2jX /oQabLB9WkJSIo1HZQuGdr6IBAjcsSBsrAG26NKBs2lRrcoFJvn3BqvkfD/FGeXOYJBg 25AbbWBYYkffqX7WqdOsReP8yl4vwI2n/6nzu/5dlDnuQaUE4udu3xbBj7WLCiTi/HM6 DI1YD9aO5Y5AiLOuaEPAYLyMjZMRLb2kur3uh+ccICilQDfG3uXD9nyUYQhWwEBMDLe2 hIYdlEYphGcYQScpuK+86gno1VBeycOLJPI0Caywl2dj7p209E27340S5VEtaQSOouH4 lAMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+lbu8Whjdwd0P/q1lJ6uz6K3myvpLKWj70uidx29BqE=; b=f58jELHpHRFZA7b6kaEdNvC45+lnBnDfSYCyF1LbOUWC4NHm/X9n2atoWYaCZ9egNT xUs63aaw+c44uJIfYZz3t0TA8khGecUqWHTysmNDgHtOPiQxhpD3AmBgwtojh0SVz8iG scZTGOSrKemw2Qt/7ohl1tk5QUNe9liZ7/M5V/q3Y7q41Buikeg8jmbfBAROaCrJbUZK za6Yf/gHqmzRDzSkRzBU/jtfLjEvlExmpI0RpveoiqQL5ymdrom6LgbneysoOk4bJDzu Tc5tVUF76ismHXs9c/S0xL3hRIzz1+gZn7GjCOWDXqIcT7PR8wk6SId8+SSVn6GP7L0w 7EBw== X-Gm-Message-State: AJcUukehnFdexHXxJPHeTFneKzB20TdoMfFk1s0wlC770Yfr6lOg+Rsn nGlpv7L2GdkSH52LXJ6929Jnrp1n X-Google-Smtp-Source: ALg8bN4zhQZBCHz3pL4gJCh/Xruzar/YaqSeVe7za0Wes0hvzKiHZstE6zKfM3dijLu9AeGNZqxQGQ== X-Received: by 2002:a1c:8c05:: with SMTP id o5mr5629708wmd.29.1545472260182; Sat, 22 Dec 2018 01:51:00 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.50.58 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:50:59 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:19 +0200 Message-Id: <20181222095036.29743-15-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH PULL 14/31] hw/pvrdma: Make sure PCI function 0 is vmxnet3 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Guest driver enforces it, we should also. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 12 ++++++++++++ 2 files changed, 14 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index b019cb843a..10a3c4fb7c 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -20,6 +20,7 @@ #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" +#include "hw/net/vmxnet3_defs.h" #include "../rdma_backend_defs.h" #include "../rdma_rm_defs.h" @@ -85,6 +86,7 @@ typedef struct PVRDMADev { RdmaBackendDev backend_dev; RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; + VMXNET3State *func0; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index ac8c092db0..b35b5dc5f0 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -565,6 +565,7 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) PVRDMADev *dev = PVRDMA_DEV(pdev); Object *memdev_root; bool ram_shared = false; + PCIDevice *func0; init_pr_dbg(); @@ -576,6 +577,17 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) return; } + func0 = pci_get_function_0(pdev); + /* Break if not vmxnet3 device in slot 0 */ + if (strcmp(object_get_typename(&func0->qdev.parent_obj), TYPE_VMXNET3)) { + pr_dbg("func0 type is %s\n", + object_get_typename(&func0->qdev.parent_obj)); + error_setg(errp, "Device on %x.0 must be %s", PCI_SLOT(pdev->devfn), + TYPE_VMXNET3); + return; + } + dev->func0 = VMXNET3(func0); + memdev_root = object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_shared); From patchwork Sat Dec 22 09:50:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017818 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OYj3Q4Fp"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLd43MzRz9sPd for ; Sat, 22 Dec 2018 21:02:00 +1100 (AEDT) Received: from localhost ([::1]:50604 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae6Y-00065s-3e for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:01:58 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41697) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadvz-0006HP-QG for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadvy-0006T3-Tp for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:03 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:35991) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadvy-0006SG-Ns for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:02 -0500 Received: by mail-wr1-x441.google.com with SMTP id u4so7504689wrp.3 for ; Sat, 22 Dec 2018 01:51:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BikPm4y4AmQ6P/GRrvceGnIfEEOHD3twxfku3ohsQcI=; b=OYj3Q4Fphmy/rdAfoBJwVzpNKa9OUjbFue59PYw5QfvbReetMgeY89RZ6yyMZ+akLw p+r+tfYDE6/UqIsBNAhp8juejceHdfj/F8al0wBqfTcjG4XJBhLaOpce3XJ/AYIctWhy kKm4Lhh8XMT+AtC61jZsKWcnz2anfEXboZXX7CxROMR0tYxCixv5L/P/NpjWh+EQaYJN Qjd7ZREt8K96cOnUYRmXl0SphSAvOKCArykGRlie8Eki7iVlNh+6OoCbADagjD+PcQXM hQgwV/s2TCMTQElpBgJ8w/FgRI06J7r2Jd/nGOL6ccTEGiZ1jFG0Sa7H7nGTWok3cTgu JwvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BikPm4y4AmQ6P/GRrvceGnIfEEOHD3twxfku3ohsQcI=; b=suiyD3hvFF+TRIPJJP3BiVT1D9GzRpYhW2eY3zLh6C3QWb+zCXo5DjpZ9yY8Cyiy2B CzPc+tsCcorXzVjhOIM8wXg5c0GiHLgus4AAn9fhr5+1t/GiVEMpWU1ssBUFr0ODSbBH 62PtlB22kX6Fk4eaBvwWa0fyhC8n/wO+yblfGk3zN111Ld1oJ3tzSYCws1FoqVq2bWDe 2QcvWwpWN0QUGVCQBI60O9LsY8jIR9BRTmQ/+QGIFp2eKVUAfQyaqE1Cf/L/DpW3Iksd MLJ7G8sb1pdW/Hm38Ylk2yRRwrxQMCqRZDrVOcQ+dulHS/15xuBpNkuY9nGdKXMBDXLg htdg== X-Gm-Message-State: AJcUukcnxb6KkzT8y7bFeZsXmXOHVOBdSUx7SAh9qatJ39re1DsBu9sc F5c5lfnc6r8kQaXYit16pHmy4OVl X-Google-Smtp-Source: ALg8bN53oPl4vdGjumvYWHh9lVrBw49ZIj7/tARuNNV9Va3f4x6f6yV+KN7Cg/61kWJ7utOoMg+9iQ== X-Received: by 2002:adf:93e2:: with SMTP id 89mr5930590wrp.129.1545472261563; Sat, 22 Dec 2018 01:51:01 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:01 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:20 +0200 Message-Id: <20181222095036.29743-16-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 15/31] hw/rdma: Initialize node_guid from vmxnet3 mac address X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia node_guid should be set once device is load. Make node_guid be GID format (32 bit) of PCI function 0 vmxnet3 device's MAC. A new function was added to do the conversion. So for example the MAC 56:b6:44:e9:62:dc will be converted to GID 54b6:44ff:fee9:62dc. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_utils.h | 9 +++++++++ hw/rdma/vmw/pvrdma_cmd.c | 10 ---------- hw/rdma/vmw/pvrdma_main.c | 5 ++++- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/hw/rdma/rdma_utils.h b/hw/rdma/rdma_utils.h index 062e2cd688..4490ea0b94 100644 --- a/hw/rdma/rdma_utils.h +++ b/hw/rdma/rdma_utils.h @@ -63,4 +63,13 @@ extern unsigned long pr_dbg_cnt; void *rdma_pci_dma_map(PCIDevice *dev, dma_addr_t addr, dma_addr_t plen); void rdma_pci_dma_unmap(PCIDevice *dev, void *buffer, dma_addr_t len); +static inline void addrconf_addr_eui48(uint8_t *eui, const char *addr) +{ + memcpy(eui, addr, 3); + eui[3] = 0xFF; + eui[4] = 0xFE; + memcpy(eui + 5, addr + 3, 3); + eui[0] ^= 2; +} + #endif diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index a334f6205e..2979582fac 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -592,16 +592,6 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, return -EINVAL; } - /* TODO: Since drivers stores node_guid at load_dsr phase then this - * assignment is not relevant, i need to figure out a way how to - * retrieve MAC of our netdev */ - if (!cmd->index) { - dev->node_guid = - dev->rdma_dev_res.ports[0].gid_tbl[0].gid.global.interface_id; - pr_dbg("dev->node_guid=0x%llx\n", - (long long unsigned int)be64_to_cpu(dev->node_guid)); - } - return 0; } diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index b35b5dc5f0..150404dfa6 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -264,7 +264,7 @@ static void init_dsr_dev_caps(PVRDMADev *dev) dsr->caps.sys_image_guid = 0; pr_dbg("sys_image_guid=%" PRIx64 "\n", dsr->caps.sys_image_guid); - dsr->caps.node_guid = cpu_to_be64(dev->node_guid); + dsr->caps.node_guid = dev->node_guid; pr_dbg("node_guid=%" PRIx64 "\n", be64_to_cpu(dsr->caps.node_guid)); dsr->caps.phys_port_cnt = MAX_PORTS; @@ -588,6 +588,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) } dev->func0 = VMXNET3(func0); + addrconf_addr_eui48((unsigned char *)&dev->node_guid, + (const char *)&dev->func0->conf.macaddr.a); + memdev_root = object_resolve_path("/objects", NULL); if (memdev_root) { object_child_foreach(memdev_root, pvrdma_check_ram_shared, &ram_shared); From patchwork Sat Dec 22 09:50:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017821 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Fy5iVsPK"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLhM2nzmz9sPd for ; Sat, 22 Dec 2018 21:04:51 +1100 (AEDT) Received: from localhost ([::1]:53887 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae9I-0008EZ-UD for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:04:48 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41718) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw3-0006Ls-Qp for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw0-0006UA-94 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:07 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:50583) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw0-0006TI-3H for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:04 -0500 Received: by mail-wm1-x341.google.com with SMTP id n190so7399864wmd.0 for ; Sat, 22 Dec 2018 01:51:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1jH+qlmzkoR61VpTtmJ6wpMY2OAsBVpMtrW4d73nlow=; b=Fy5iVsPKTZKvoWyy/ZBy2gckILVJkelB8FgrHhaFSYanepYWBrPOcQChDZlPHmWiYo Z36JZhqBN7N4PYcWEO/XVZ00jz7utj8ItOBPwwslDz9zI8Gt7XEY15B9VAQw8lIfE1lC UU6V0yRWRK0E5WYPn6tGUhaacfiXdyXOGvKL8maUTAkzAuQLFDjs/JrJM9z+O2u5yf40 6xGSgqFv/qxhogPn5x3kXg0lr/SAAqEmYE7xDiao37MYUuhRtY4zc8SdvxSFfdWUDMa4 WvWP5wMlSnfbRvDsmy8YsN4FzJUw0Tb5pbv9jwL1maMd8XhagnPbkOaWu/m4yf1y2tcD 9lpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1jH+qlmzkoR61VpTtmJ6wpMY2OAsBVpMtrW4d73nlow=; b=azBlQgNMm1ek/PDX6rStJ9vJMa6EFkQFEJHo02QPyr1w81F29mCZdfRnOL1UBK2o2x bfn1JwXU+7KRHf2zALfJUzmhX32HVxIyFglmLl7uG1C4BzkdIKfj/A+5K6oAfSaR5yDV OzH3CPvJR/jEV/u9VYrgs4ViFyROqoFHR5C/D1yKTrQzIr8Ylscn6HD/rsC4lehEnxLT 7aX7qtd2OPp2Je9RriMztZ5DOED0L8yZ7wHUgWD43EptGrEv5wz2eZAmQhoeT59rQT3i 0OU1IHzimstZWMO8JYUcpSsi41KAUDLax0ktdHWdhpsom4Exbbg49PvTMnikRbck20bK 62OQ== X-Gm-Message-State: AJcUukc45uBQItfzXEBcajIRtaz7hjMEjx7MvPZ3lNxhXwrYOjPTCMLk 7MS40weILdud4JCoEy8jkV9Dn205 X-Google-Smtp-Source: ALg8bN53BWifALXEB9aI7TAdIKZT5d1xPPLw7ZuM7cB1nUktcnTO4NTmtyox/nYDz1TC+oWnk5M6Mw== X-Received: by 2002:a1c:8cd0:: with SMTP id o199mr5696490wmd.56.1545472262953; Sat, 22 Dec 2018 01:51:02 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.01 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:02 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:21 +0200 Message-Id: <20181222095036.29743-17-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 16/31] hw/pvrdma: Make device state depend on Ethernet function state X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia User should be able to control the device by changing Ethernet function state so if user runs 'ifconfig ens3 down' the PVRDMA function should be down as well. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 2979582fac..0d3c818c20 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -139,7 +139,8 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_req *req, resp->hdr.ack = PVRDMA_CMD_QUERY_PORT_RESP; resp->hdr.err = 0; - resp->attrs.state = attrs.state; + resp->attrs.state = dev->func0->device_active ? attrs.state : + PVRDMA_PORT_DOWN; resp->attrs.max_mtu = attrs.max_mtu; resp->attrs.active_mtu = attrs.active_mtu; resp->attrs.phys_state = attrs.phys_state; From patchwork Sat Dec 22 09:50:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017816 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="gxxGVf2H"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLbp337tz9sPS for ; Sat, 22 Dec 2018 21:00:54 +1100 (AEDT) Received: from localhost ([::1]:49294 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae5T-0005HI-VP for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:00:52 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41719) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw3-0006Lt-RC for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw1-0006VC-RR for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:07 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:42941) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw1-0006Ug-IQ for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:05 -0500 Received: by mail-wr1-x441.google.com with SMTP id q18so7506960wrx.9 for ; Sat, 22 Dec 2018 01:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qOvOhJwcG4T39J5QUh010McIhaKKfOSPLbwVwA7smdw=; b=gxxGVf2HmwGEYUhwisvbW0gEVLQt8FiHPYnFr0qWXG1RMSI8asB3LQDQLIC+vn/Iq7 ++76zBiOPBOSgODWBvqLhxX94K8tWhbM6+hvDQ+DuPZOAWm0nkYvhJNE7bvZFuj8j/ff ueonaDNPY60aOZ4XHwHFrd+p4xDOeNMDebt+NchJKw4RzBqrEOSjLBEymglxgixIhMY/ chJvknG9P82zt3FygZTBcrb6hVol45/qDSMvC5BiT6/h5/vo2cqPOWyjkppUmSTTdlZ+ BpDN1ydahmGU6ybPYI3gS61xr9bz991woyxSsh7OA29mZoezCFyDp2uiErV70ctRF3KC 4diQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qOvOhJwcG4T39J5QUh010McIhaKKfOSPLbwVwA7smdw=; b=Kx44fAmCPlD7E0akGyBXVTjYj2D7Nf16PVbFQy4tAhi1dPb4jBK8tDmeoXLwKHLWrT EWmdpb+h9c7SS8TB4c5ysmlG5xsZaXEZKDmWg+EN+LTVkAaVzU8yTYg6ecqqJ+UkzF5B AByilfdR1vW494c+AWqasr13Xv8ktySLy0IV22U8/V072Q0+VXDvsScWCozYW2LTG8mB NsqurpNtnJ2Spwxylg53GuvieeSG4naAwgjmVNK/dDjT+mj9lBFJAnWVGmh1y6/lppYh 8kEvfGF7MfxB5GT0/IaNxUZr0gEGQ7ta6sOpm14+aWSVO37qWL8fSkZ2JP6Rare0aKPM QuBg== X-Gm-Message-State: AJcUuke7MKLNPBrdcKtD1RVK3IyITF1dKBN/pQmasg4CV8F9/nqqHA7k 37nAq02zxmXDwOVFMpDgzhfdnKr3 X-Google-Smtp-Source: ALg8bN4s7EARgBf4ipkqSsdw94GIY0Xl/M3tNvgS0TcIKkJSo8tbFXQJGXAdVEwCM8kXbuT4kx6efA== X-Received: by 2002:adf:fd87:: with SMTP id d7mr5501701wrr.74.1545472264391; Sat, 22 Dec 2018 01:51:04 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.03 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:03 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:22 +0200 Message-Id: <20181222095036.29743-18-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 17/31] hw/pvrdma: Fill all CQE fields X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Add ability to pass specific WC attributes to CQE such as GRH_BIT flag. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 59 +++++++++++++++++++++++-------------- hw/rdma/rdma_backend.h | 4 +-- hw/rdma/vmw/pvrdma_qp_ops.c | 31 +++++++++++-------- 3 files changed, 58 insertions(+), 36 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index 1d496bbd95..ae1e4dcb29 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -60,13 +60,24 @@ struct backend_umad { char mad[RDMA_MAX_PRIVATE_DATA]; }; -static void (*comp_handler)(int status, unsigned int vendor_err, void *ctx); +static void (*comp_handler)(void *ctx, struct ibv_wc *wc); -static void dummy_comp_handler(int status, unsigned int vendor_err, void *ctx) +static void dummy_comp_handler(void *ctx, struct ibv_wc *wc) { pr_err("No completion handler is registered\n"); } +static inline void complete_work(enum ibv_wc_status status, uint32_t vendor_err, + void *ctx) +{ + struct ibv_wc wc = {0}; + + wc.status = status; + wc.vendor_err = vendor_err; + + comp_handler(ctx, &wc); +} + static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq) { int i, ne; @@ -91,7 +102,7 @@ static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq) } pr_dbg("Processing %s CQE\n", bctx->is_tx_req ? "send" : "recv"); - comp_handler(wc[i].status, wc[i].vendor_err, bctx->up_ctx); + comp_handler(bctx->up_ctx, &wc[i]); rdma_rm_dealloc_cqe_ctx(rdma_dev_res, wc[i].wr_id); g_free(bctx); @@ -256,8 +267,8 @@ static void start_comp_thread(RdmaBackendDev *backend_dev) comp_handler_thread, backend_dev, QEMU_THREAD_DETACHED); } -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx)) +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc)) { comp_handler = handler; } @@ -451,14 +462,14 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP1 */ if (qp_type == IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } else if (qp_type == IBV_QPT_GSI) { pr_dbg("QP1\n"); rc = mad_send(backend_dev, sgid_idx, sgid, sge, num_sge); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_MAD_SEND, ctx); } else { - comp_handler(IBV_WC_SUCCESS, 0, ctx); + complete_work(IBV_WC_SUCCESS, 0, ctx); } } return; @@ -467,7 +478,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, pr_dbg("num_sge=%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } @@ -478,21 +489,21 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, rc = rdma_rm_alloc_cqe_ctx(backend_dev->rdma_dev_res, &bctx_id, bctx); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } rc = build_host_sge_array(backend_dev->rdma_dev_res, new_sge, sge, num_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } if (qp_type == IBV_QPT_UD) { wr.wr.ud.ah = create_ah(backend_dev, qp->ibpd, sgid_idx, dgid); if (!wr.wr.ud.ah) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } wr.wr.ud.remote_qpn = dqpn; @@ -510,7 +521,7 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post send WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } @@ -579,13 +590,13 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, if (!qp->ibqp) { /* This field does not get initialized for QP0 and QP1 */ if (qp_type == IBV_QPT_SMI) { pr_dbg("QP0 unsupported\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_QP0, ctx); } if (qp_type == IBV_QPT_GSI) { pr_dbg("QP1\n"); rc = save_mad_recv_buffer(backend_dev, sge, num_sge, ctx); if (rc) { - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); } } return; @@ -594,7 +605,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, pr_dbg("num_sge=%d\n", num_sge); if (!num_sge) { pr_dbg("num_sge=0\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); return; } @@ -605,14 +616,14 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, rc = rdma_rm_alloc_cqe_ctx(rdma_dev_res, &bctx_id, bctx); if (unlikely(rc)) { pr_dbg("Failed to allocate cqe_ctx\n"); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NOMEM, ctx); goto out_free_bctx; } rc = build_host_sge_array(rdma_dev_res, new_sge, sge, num_sge); if (rc) { pr_dbg("Error: Failed to build host SGE array\n"); - comp_handler(IBV_WC_GENERAL_ERR, rc, ctx); + complete_work(IBV_WC_GENERAL_ERR, rc, ctx); goto out_dealloc_cqe_ctx; } @@ -624,7 +635,7 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, if (rc) { pr_dbg("Fail (%d, %d) to post recv WQE to qpn %d\n", rc, errno, qp->ibqp->qp_num); - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_FAIL_BACKEND, ctx); goto out_dealloc_cqe_ctx; } @@ -998,9 +1009,10 @@ static void process_incoming_mad_req(RdmaBackendDev *backend_dev, mad = rdma_pci_dma_map(backend_dev->dev, bctx->sge.addr, bctx->sge.length); if (!mad || bctx->sge.length < msg->umad_len + MAD_HDR_SIZE) { - comp_handler(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, - bctx->up_ctx); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_MAD_BUFF, + bctx->up_ctx); } else { + struct ibv_wc wc = {0}; pr_dbg_buf("mad", msg->umad.mad, msg->umad_len); memset(mad, 0, bctx->sge.length); build_mad_hdr((struct ibv_grh *)mad, @@ -1009,7 +1021,10 @@ static void process_incoming_mad_req(RdmaBackendDev *backend_dev, memcpy(&mad[MAD_HDR_SIZE], msg->umad.mad, msg->umad_len); rdma_pci_dma_unmap(backend_dev->dev, mad, bctx->sge.length); - comp_handler(IBV_WC_SUCCESS, 0, bctx->up_ctx); + wc.byte_len = msg->umad_len; + wc.status = IBV_WC_SUCCESS; + wc.wc_flags = IBV_WC_GRH; + comp_handler(bctx->up_ctx, &wc); } g_free(bctx); diff --git a/hw/rdma/rdma_backend.h b/hw/rdma/rdma_backend.h index 59ad2b874b..8cae40f827 100644 --- a/hw/rdma/rdma_backend.h +++ b/hw/rdma/rdma_backend.h @@ -57,8 +57,8 @@ int rdma_backend_get_gid_index(RdmaBackendDev *backend_dev, union ibv_gid *gid); void rdma_backend_start(RdmaBackendDev *backend_dev); void rdma_backend_stop(RdmaBackendDev *backend_dev); -void rdma_backend_register_comp_handler(void (*handler)(int status, - unsigned int vendor_err, void *ctx)); +void rdma_backend_register_comp_handler(void (*handler)(void *ctx, + struct ibv_wc *wc)); void rdma_backend_unregister_comp_handler(void); int rdma_backend_query_port(RdmaBackendDev *backend_dev, diff --git a/hw/rdma/vmw/pvrdma_qp_ops.c b/hw/rdma/vmw/pvrdma_qp_ops.c index 2130824098..300471a4c9 100644 --- a/hw/rdma/vmw/pvrdma_qp_ops.c +++ b/hw/rdma/vmw/pvrdma_qp_ops.c @@ -47,7 +47,7 @@ typedef struct PvrdmaRqWqe { * 3. Interrupt host */ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, - struct pvrdma_cqe *cqe) + struct pvrdma_cqe *cqe, struct ibv_wc *wc) { struct pvrdma_cqe *cqe1; struct pvrdma_cqne *cqne; @@ -66,6 +66,7 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, pr_dbg("Writing CQE\n"); cqe1 = pvrdma_ring_next_elem_write(ring); if (unlikely(!cqe1)) { + pr_dbg("No CQEs in ring\n"); return -EINVAL; } @@ -73,8 +74,20 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, cqe1->wr_id = cqe->wr_id; cqe1->qp = cqe->qp; cqe1->opcode = cqe->opcode; - cqe1->status = cqe->status; - cqe1->vendor_err = cqe->vendor_err; + cqe1->status = wc->status; + cqe1->byte_len = wc->byte_len; + cqe1->src_qp = wc->src_qp; + cqe1->wc_flags = wc->wc_flags; + cqe1->vendor_err = wc->vendor_err; + + pr_dbg("wr_id=%" PRIx64 "\n", cqe1->wr_id); + pr_dbg("qp=0x%lx\n", cqe1->qp); + pr_dbg("opcode=%d\n", cqe1->opcode); + pr_dbg("status=%d\n", cqe1->status); + pr_dbg("byte_len=%d\n", cqe1->byte_len); + pr_dbg("src_qp=%d\n", cqe1->src_qp); + pr_dbg("wc_flags=%d\n", cqe1->wc_flags); + pr_dbg("vendor_err=%d\n", cqe1->vendor_err); pvrdma_ring_write_inc(ring); @@ -99,18 +112,12 @@ static int pvrdma_post_cqe(PVRDMADev *dev, uint32_t cq_handle, return 0; } -static void pvrdma_qp_ops_comp_handler(int status, unsigned int vendor_err, - void *ctx) +static void pvrdma_qp_ops_comp_handler(void *ctx, struct ibv_wc *wc) { CompHandlerCtx *comp_ctx = (CompHandlerCtx *)ctx; - pr_dbg("cq_handle=%d\n", comp_ctx->cq_handle); - pr_dbg("wr_id=%" PRIx64 "\n", comp_ctx->cqe.wr_id); - pr_dbg("status=%d\n", status); - pr_dbg("vendor_err=0x%x\n", vendor_err); - comp_ctx->cqe.status = status; - comp_ctx->cqe.vendor_err = vendor_err; - pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe); + pvrdma_post_cqe(comp_ctx->dev, comp_ctx->cq_handle, &comp_ctx->cqe, wc); + g_free(ctx); } From patchwork Sat Dec 22 09:50:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Y/AVwlOJ"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLgv6PYHz9sPd for ; Sat, 22 Dec 2018 21:04:27 +1100 (AEDT) Received: from localhost ([::1]:53457 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae8v-0007yO-9t for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:04:25 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41734) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw4-0006NF-V4 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw3-0006Vt-CK for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:08 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:52780) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw3-0006VU-3u for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:07 -0500 Received: by mail-wm1-x341.google.com with SMTP id m1so7392547wml.2 for ; Sat, 22 Dec 2018 01:51:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/7Zo4QgyA6P2oUwmed3myJIZ5TcWK9NLoQ+m6B8umUY=; b=Y/AVwlOJT1eE1CFeLDre/MURX9IVGX7Bv7mfbGAH85RxsD+MzgNBSQ/0K4eYI+HVgG qo2mO/+xc0H6mgnRyKQorEVc5aN9JOITIgK2bx3n18wF8k2lVAFTdiM7RFNvS8vnjIyE +I1nLiTyuNL4lq8Vmk+mwF+ny0KYtUN/no6OKskay5EPb+Wk6DEILBvLpktwYXctYA8r 0ARtGN8YhFNSeADZKcfZB+lBmPCoykcbXyCp2uIv7lFazXapDMnVG8hxeQB8GdJCAFAh 0y/zY9D8Rv84miAyl0hJYZwq5wSIn8YXRvnGqgT95GlL2tP0Ahz71aMgXqzBOKu9CdZt bICw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/7Zo4QgyA6P2oUwmed3myJIZ5TcWK9NLoQ+m6B8umUY=; b=SrNCBoSlwa1H7QJXpdwdSIJ4+ELNJvG3W3FdMSDL4SOmiBgm4yFMMGgxNzmyx7dOpe gBtj04G7mTCELnhnIYfYaAtSPytdSIjONl4qd8T2wDj7z5xqz4fGJdjrzhROBVtw5w01 6AMglNeW6m2MUuK6TaLCuJBClvadep4IVA4d0UcuPfN1xW5VE8Xkziocc6SxAJeqy/Be ka+c+UkyvlHAdONDIHVl21QPV+es9B55cnMCvCGzmpxf8ZyPHA0EiDjcumBkCVoakI/Q KSyy36HfZ2HszJ5Q0/MrlAwJc8Co/U9ggEQORD+zEOwfk+uuvV1YkBLE+xAp7cRR2you 2npw== X-Gm-Message-State: AJcUukeEbRHRsKqPhmP1UwT8BMrgytcb9LFh1sG7iZkfLr8X3NUcxICc UDWMMuI+V+rlQha7m/DiPEJ/moRU X-Google-Smtp-Source: ALg8bN5YDMZsDsAamYX6G/qt0RnIke7QB63138HinBXDvjXN/nC7pwwup2BwMOyuMoX5pooo4gLSVQ== X-Received: by 2002:a1c:9dc4:: with SMTP id g187mr6146796wme.152.1545472265892; Sat, 22 Dec 2018 01:51:05 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.04 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:05 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:23 +0200 Message-Id: <20181222095036.29743-19-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 18/31] hw/pvrdma: Fill error code in command's response X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Driver checks error code let's set it. In addition, for code simplification purposes, set response's fields ack, response and err outside of the scope of command handlers. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 197 ++++++++++++++++++--------------------- 1 file changed, 90 insertions(+), 107 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 0d3c818c20..3b94545761 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -128,6 +128,9 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_port_attr attrs = {0}; pr_dbg("port=%d\n", cmd->port_num); + if (cmd->port_num > MAX_PORTS) { + return -EINVAL; + } if (rdma_backend_query_port(&dev->backend_dev, (struct ibv_port_attr *)&attrs)) { @@ -135,9 +138,6 @@ static int query_port(PVRDMADev *dev, union pvrdma_cmd_req *req, } memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_QUERY_PORT_RESP; - resp->hdr.err = 0; resp->attrs.state = dev->func0->device_active ? attrs.state : PVRDMA_PORT_DOWN; @@ -160,12 +160,16 @@ static int query_pkey(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_cmd_query_pkey_resp *resp = &rsp->query_pkey_resp; pr_dbg("port=%d\n", cmd->port_num); + if (cmd->port_num > MAX_PORTS) { + return -EINVAL; + } + pr_dbg("index=%d\n", cmd->index); + if (cmd->index > MAX_PKEYS) { + return -EINVAL; + } memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_QUERY_PKEY_RESP; - resp->hdr.err = 0; resp->pkey = PVRDMA_PKEY; pr_dbg("pkey=0x%x\n", resp->pkey); @@ -178,17 +182,15 @@ static int create_pd(PVRDMADev *dev, union pvrdma_cmd_req *req, { struct pvrdma_cmd_create_pd *cmd = &req->create_pd; struct pvrdma_cmd_create_pd_resp *resp = &rsp->create_pd_resp; + int rc; pr_dbg("context=0x%x\n", cmd->ctx_handle ? cmd->ctx_handle : 0); memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_CREATE_PD_RESP; - resp->hdr.err = rdma_rm_alloc_pd(&dev->rdma_dev_res, &dev->backend_dev, - &resp->pd_handle, cmd->ctx_handle); + rc = rdma_rm_alloc_pd(&dev->rdma_dev_res, &dev->backend_dev, + &resp->pd_handle, cmd->ctx_handle); - pr_dbg("ret=%d\n", resp->hdr.err); - return resp->hdr.err; + return rc; } static int destroy_pd(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -210,10 +212,9 @@ static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_cmd_create_mr_resp *resp = &rsp->create_mr_resp; PCIDevice *pci_dev = PCI_DEVICE(dev); void *host_virt = NULL; + int rc = 0; memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_CREATE_MR_RESP; pr_dbg("pd_handle=%d\n", cmd->pd_handle); pr_dbg("access_flags=0x%x\n", cmd->access_flags); @@ -224,22 +225,18 @@ static int create_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, cmd->length); if (!host_virt) { pr_dbg("Failed to map to pdir\n"); - resp->hdr.err = -EINVAL; - goto out; + return -EINVAL; } } - resp->hdr.err = rdma_rm_alloc_mr(&dev->rdma_dev_res, cmd->pd_handle, - cmd->start, cmd->length, host_virt, - cmd->access_flags, &resp->mr_handle, - &resp->lkey, &resp->rkey); - if (resp->hdr.err && host_virt) { + rc = rdma_rm_alloc_mr(&dev->rdma_dev_res, cmd->pd_handle, cmd->start, + cmd->length, host_virt, cmd->access_flags, + &resp->mr_handle, &resp->lkey, &resp->rkey); + if (rc && host_virt) { munmap(host_virt, cmd->length); } -out: - pr_dbg("ret=%d\n", resp->hdr.err); - return resp->hdr.err; + return rc; } static int destroy_mr(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -317,28 +314,25 @@ static int create_cq(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_cmd_create_cq *cmd = &req->create_cq; struct pvrdma_cmd_create_cq_resp *resp = &rsp->create_cq_resp; PvrdmaRing *ring = NULL; + int rc; memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_CREATE_CQ_RESP; resp->cqe = cmd->cqe; - resp->hdr.err = create_cq_ring(PCI_DEVICE(dev), &ring, cmd->pdir_dma, - cmd->nchunks, cmd->cqe); - if (resp->hdr.err) { - goto out; + rc = create_cq_ring(PCI_DEVICE(dev), &ring, cmd->pdir_dma, cmd->nchunks, + cmd->cqe); + if (rc) { + return rc; } pr_dbg("ring=%p\n", ring); - resp->hdr.err = rdma_rm_alloc_cq(&dev->rdma_dev_res, &dev->backend_dev, - cmd->cqe, &resp->cq_handle, ring); + rc = rdma_rm_alloc_cq(&dev->rdma_dev_res, &dev->backend_dev, cmd->cqe, + &resp->cq_handle, ring); resp->cqe = cmd->cqe; -out: - pr_dbg("ret=%d\n", resp->hdr.err); - return resp->hdr.err; + return rc; } static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -458,30 +452,31 @@ static int create_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_cmd_create_qp *cmd = &req->create_qp; struct pvrdma_cmd_create_qp_resp *resp = &rsp->create_qp_resp; PvrdmaRing *rings = NULL; + int rc; memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_CREATE_QP_RESP; pr_dbg("total_chunks=%d\n", cmd->total_chunks); pr_dbg("send_chunks=%d\n", cmd->send_chunks); - resp->hdr.err = create_qp_rings(PCI_DEVICE(dev), cmd->pdir_dma, &rings, - cmd->max_send_wr, cmd->max_send_sge, - cmd->send_chunks, cmd->max_recv_wr, - cmd->max_recv_sge, cmd->total_chunks - - cmd->send_chunks - 1); - if (resp->hdr.err) { - goto out; + rc = create_qp_rings(PCI_DEVICE(dev), cmd->pdir_dma, &rings, + cmd->max_send_wr, cmd->max_send_sge, cmd->send_chunks, + cmd->max_recv_wr, cmd->max_recv_sge, + cmd->total_chunks - cmd->send_chunks - 1); + if (rc) { + return rc; } pr_dbg("rings=%p\n", rings); - resp->hdr.err = rdma_rm_alloc_qp(&dev->rdma_dev_res, cmd->pd_handle, - cmd->qp_type, cmd->max_send_wr, - cmd->max_send_sge, cmd->send_cq_handle, - cmd->max_recv_wr, cmd->max_recv_sge, - cmd->recv_cq_handle, rings, &resp->qpn); + rc = rdma_rm_alloc_qp(&dev->rdma_dev_res, cmd->pd_handle, cmd->qp_type, + cmd->max_send_wr, cmd->max_send_sge, + cmd->send_cq_handle, cmd->max_recv_wr, + cmd->max_recv_sge, cmd->recv_cq_handle, rings, + &resp->qpn); + if (rc) { + return rc; + } resp->max_send_wr = cmd->max_send_wr; resp->max_recv_wr = cmd->max_recv_wr; @@ -489,35 +484,31 @@ static int create_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, resp->max_recv_sge = cmd->max_recv_sge; resp->max_inline_data = cmd->max_inline_data; -out: - pr_dbg("ret=%d\n", resp->hdr.err); - return resp->hdr.err; + return 0; } static int modify_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { struct pvrdma_cmd_modify_qp *cmd = &req->modify_qp; + int rc; pr_dbg("qp_handle=%d\n", cmd->qp_handle); memset(rsp, 0, sizeof(*rsp)); - rsp->hdr.response = cmd->hdr.response; - rsp->hdr.ack = PVRDMA_CMD_MODIFY_QP_RESP; /* No need to verify sgid_index since it is u8 */ - rsp->hdr.err = - rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, cmd->qp_handle, - cmd->attr_mask, cmd->attrs.ah_attr.grh.sgid_index, - (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, - cmd->attrs.dest_qp_num, - (enum ibv_qp_state)cmd->attrs.qp_state, - cmd->attrs.qkey, cmd->attrs.rq_psn, - cmd->attrs.sq_psn); + rc = rdma_rm_modify_qp(&dev->rdma_dev_res, &dev->backend_dev, + cmd->qp_handle, cmd->attr_mask, + cmd->attrs.ah_attr.grh.sgid_index, + (union ibv_gid *)&cmd->attrs.ah_attr.grh.dgid, + cmd->attrs.dest_qp_num, + (enum ibv_qp_state)cmd->attrs.qp_state, + cmd->attrs.qkey, cmd->attrs.rq_psn, + cmd->attrs.sq_psn); - pr_dbg("ret=%d\n", rsp->hdr.err); - return rsp->hdr.err; + return rc; } static int query_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -526,21 +517,18 @@ static int query_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, struct pvrdma_cmd_query_qp *cmd = &req->query_qp; struct pvrdma_cmd_query_qp_resp *resp = &rsp->query_qp_resp; struct ibv_qp_init_attr init_attr; + int rc; pr_dbg("qp_handle=%d\n", cmd->qp_handle); pr_dbg("attr_mask=0x%x\n", cmd->attr_mask); memset(rsp, 0, sizeof(*rsp)); - rsp->hdr.response = cmd->hdr.response; - rsp->hdr.ack = PVRDMA_CMD_QUERY_QP_RESP; - rsp->hdr.err = rdma_rm_query_qp(&dev->rdma_dev_res, &dev->backend_dev, - cmd->qp_handle, - (struct ibv_qp_attr *)&resp->attrs, - cmd->attr_mask, &init_attr); + rc = rdma_rm_query_qp(&dev->rdma_dev_res, &dev->backend_dev, cmd->qp_handle, + (struct ibv_qp_attr *)&resp->attrs, cmd->attr_mask, + &init_attr); - pr_dbg("ret=%d\n", rsp->hdr.err); - return rsp->hdr.err; + return rc; } static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -589,11 +577,8 @@ static int create_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, rc = rdma_rm_add_gid(&dev->rdma_dev_res, &dev->backend_dev, dev->backend_eth_device_name, gid, cmd->index); - if (rc < 0) { - return -EINVAL; - } - return 0; + return rc; } static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -612,12 +597,7 @@ static int destroy_bind(PVRDMADev *dev, union pvrdma_cmd_req *req, rc = rdma_rm_del_gid(&dev->rdma_dev_res, &dev->backend_dev, dev->backend_eth_device_name, cmd->index); - if (rc < 0) { - rsp->hdr.err = rc; - goto out; - } - - return 0; + return rc; } static int create_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -625,18 +605,14 @@ static int create_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, { struct pvrdma_cmd_create_uc *cmd = &req->create_uc; struct pvrdma_cmd_create_uc_resp *resp = &rsp->create_uc_resp; + int rc; pr_dbg("pfn=%d\n", cmd->pfn); memset(resp, 0, sizeof(*resp)); - resp->hdr.response = cmd->hdr.response; - resp->hdr.ack = PVRDMA_CMD_CREATE_UC_RESP; - resp->hdr.err = rdma_rm_alloc_uc(&dev->rdma_dev_res, cmd->pfn, - &resp->ctx_handle); + rc = rdma_rm_alloc_uc(&dev->rdma_dev_res, cmd->pfn, &resp->ctx_handle); - pr_dbg("ret=%d\n", resp->hdr.err); - - return 0; + return rc; } static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, @@ -650,30 +626,32 @@ static int destroy_uc(PVRDMADev *dev, union pvrdma_cmd_req *req, return 0; } + struct cmd_handler { uint32_t cmd; + uint32_t ack; int (*exec)(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp); }; static struct cmd_handler cmd_handlers[] = { - {PVRDMA_CMD_QUERY_PORT, query_port}, - {PVRDMA_CMD_QUERY_PKEY, query_pkey}, - {PVRDMA_CMD_CREATE_PD, create_pd}, - {PVRDMA_CMD_DESTROY_PD, destroy_pd}, - {PVRDMA_CMD_CREATE_MR, create_mr}, - {PVRDMA_CMD_DESTROY_MR, destroy_mr}, - {PVRDMA_CMD_CREATE_CQ, create_cq}, - {PVRDMA_CMD_RESIZE_CQ, NULL}, - {PVRDMA_CMD_DESTROY_CQ, destroy_cq}, - {PVRDMA_CMD_CREATE_QP, create_qp}, - {PVRDMA_CMD_MODIFY_QP, modify_qp}, - {PVRDMA_CMD_QUERY_QP, query_qp}, - {PVRDMA_CMD_DESTROY_QP, destroy_qp}, - {PVRDMA_CMD_CREATE_UC, create_uc}, - {PVRDMA_CMD_DESTROY_UC, destroy_uc}, - {PVRDMA_CMD_CREATE_BIND, create_bind}, - {PVRDMA_CMD_DESTROY_BIND, destroy_bind}, + {PVRDMA_CMD_QUERY_PORT, PVRDMA_CMD_QUERY_PORT_RESP, query_port}, + {PVRDMA_CMD_QUERY_PKEY, PVRDMA_CMD_QUERY_PKEY_RESP, query_pkey}, + {PVRDMA_CMD_CREATE_PD, PVRDMA_CMD_CREATE_PD_RESP, create_pd}, + {PVRDMA_CMD_DESTROY_PD, PVRDMA_CMD_DESTROY_PD_RESP_NOOP, destroy_pd}, + {PVRDMA_CMD_CREATE_MR, PVRDMA_CMD_CREATE_MR_RESP, create_mr}, + {PVRDMA_CMD_DESTROY_MR, PVRDMA_CMD_DESTROY_MR_RESP_NOOP, destroy_mr}, + {PVRDMA_CMD_CREATE_CQ, PVRDMA_CMD_CREATE_CQ_RESP, create_cq}, + {PVRDMA_CMD_RESIZE_CQ, PVRDMA_CMD_RESIZE_CQ_RESP, NULL}, + {PVRDMA_CMD_DESTROY_CQ, PVRDMA_CMD_DESTROY_CQ_RESP_NOOP, destroy_cq}, + {PVRDMA_CMD_CREATE_QP, PVRDMA_CMD_CREATE_QP_RESP, create_qp}, + {PVRDMA_CMD_MODIFY_QP, PVRDMA_CMD_MODIFY_QP_RESP, modify_qp}, + {PVRDMA_CMD_QUERY_QP, PVRDMA_CMD_QUERY_QP_RESP, query_qp}, + {PVRDMA_CMD_DESTROY_QP, PVRDMA_CMD_DESTROY_QP_RESP, destroy_qp}, + {PVRDMA_CMD_CREATE_UC, PVRDMA_CMD_CREATE_UC_RESP, create_uc}, + {PVRDMA_CMD_DESTROY_UC, PVRDMA_CMD_DESTROY_UC_RESP_NOOP, destroy_uc}, + {PVRDMA_CMD_CREATE_BIND, PVRDMA_CMD_CREATE_BIND_RESP_NOOP, create_bind}, + {PVRDMA_CMD_DESTROY_BIND, PVRDMA_CMD_DESTROY_BIND_RESP_NOOP, destroy_bind}, }; int execute_command(PVRDMADev *dev) @@ -696,7 +674,12 @@ int execute_command(PVRDMADev *dev) } err = cmd_handlers[dsr_info->req->hdr.cmd].exec(dev, dsr_info->req, - dsr_info->rsp); + dsr_info->rsp); + dsr_info->rsp->hdr.response = dsr_info->req->hdr.response; + dsr_info->rsp->hdr.ack = cmd_handlers[dsr_info->req->hdr.cmd].ack; + dsr_info->rsp->hdr.err = err < 0 ? -err : 0; + pr_dbg("rsp->hdr.err=%d\n", dsr_info->rsp->hdr.err); + out: set_reg_val(dev, PVRDMA_REG_ERR, err); post_interrupt(dev, INTR_VEC_CMD_RING); From patchwork Sat Dec 22 09:50:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017822 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="IzWULhCl"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLj22QmLz9sPd for ; Sat, 22 Dec 2018 21:05:26 +1100 (AEDT) Received: from localhost ([::1]:54681 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gae9r-0000Dm-Sd for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:05:23 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41741) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw5-0006OA-Ml for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:10 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw4-0006Wt-N4 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:09 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:44479) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw4-0006WJ-HG for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:08 -0500 Received: by mail-wr1-x441.google.com with SMTP id z5so7497292wrt.11 for ; Sat, 22 Dec 2018 01:51:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mwPhF3YDNXPA/ayvaocLeWGdKPYJHNNA9KhY18EIHq4=; b=IzWULhClWRSQrxZYl6iiuNFWsTY1B6A+DvuVHButvu6+Z4oeHlhFo7kQjMQ/GtxTSr 9jnoI3u9VeYaEvofoFSDbSDVgqw6ZxCYvCr4IJHoPjCmvXXn6O3s+wwUS89mprjs6BKj ocKvpRaGeLZK60YGyCLgxIi3U3F1ozASQ3rLqpCKIvzx7QM9n4o9FyIFmNZoDDK4pSqS zdAig3onsgyL0T1QCtivwi2kNyS/UbaDTJ3o/R/t9chARvk2JpEKFa6EvpdkEvF223pk +Pn8korsLda7yQ9Gn9QAJmGfjamGjJahsiQhwNnXJ8LcGw9qN8Newm0Bd4WRChM7hGWA pzbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mwPhF3YDNXPA/ayvaocLeWGdKPYJHNNA9KhY18EIHq4=; b=aSE9ik7/7AVQMfj4v3cncEPF/VNnniG5cbZ4NX2DzXAh7k5OgHcUB2OGknKJ02vf1P V7NMr0dhtKtbwPMF4mF8L+iQW0qS2ubR/eh++XfvA9Ikccecgctzw0pNTQEibGVTY/62 7RqsVNiIXgPUsRM/4L3Gg/3te9rrZoF21Czb8lMLGNRgU3zZPpQwZSGwpE/Rnzqgx1U0 /22zoBjfriut+sqxyne7aIc5ZnJoZ5Td6VOPgnrdz0LDXuNgf5ynLQXV5GZX84HRjDAU ci8GKkOvswviaTGyUivjLC5Dqf6CA4ofRuhh/pe3EcGIG9/3dhNTbOTXtNPQaia7AHNM lX7Q== X-Gm-Message-State: AJcUukdXY4Ok2Wo7tvd2FQNVYEpY6bM5k76VsEBkUIPA7Pi0gGgKrI9W X6GVQupyEENx+mfMgxIUzm83jbTA X-Google-Smtp-Source: ALg8bN6bETZ8F3yOpbbITUL8LgC7Sxnn5JUjTM6k0RxZeEyxE2wrEHRF/N+V4nrLdbbKjGTPHIi5yA== X-Received: by 2002:a5d:63c3:: with SMTP id c3mr5568543wrw.215.1545472267270; Sat, 22 Dec 2018 01:51:07 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:06 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:24 +0200 Message-Id: <20181222095036.29743-20-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 19/31] hw/rdma: Remove unneeded code that handles more that one port X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Device supports only one port, let's remove a dead code that handles more than one port. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 34 ++++++++++++++++------------------ hw/rdma/rdma_rm.h | 2 +- hw/rdma/rdma_rm_defs.h | 4 ++-- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index 250254561c..b7d4ebe972 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -545,7 +545,7 @@ int rdma_rm_add_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, return -EINVAL; } - memcpy(&dev_res->ports[0].gid_tbl[gid_idx].gid, gid, sizeof(*gid)); + memcpy(&dev_res->port.gid_tbl[gid_idx].gid, gid, sizeof(*gid)); return 0; } @@ -556,15 +556,15 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, int rc; rc = rdma_backend_del_gid(backend_dev, ifname, - &dev_res->ports[0].gid_tbl[gid_idx].gid); + &dev_res->port.gid_tbl[gid_idx].gid); if (rc) { pr_dbg("Fail to delete gid\n"); return -EINVAL; } - memset(dev_res->ports[0].gid_tbl[gid_idx].gid.raw, 0, - sizeof(dev_res->ports[0].gid_tbl[gid_idx].gid)); - dev_res->ports[0].gid_tbl[gid_idx].backend_gid_index = -1; + memset(dev_res->port.gid_tbl[gid_idx].gid.raw, 0, + sizeof(dev_res->port.gid_tbl[gid_idx].gid)); + dev_res->port.gid_tbl[gid_idx].backend_gid_index = -1; return 0; } @@ -577,16 +577,16 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, return -EINVAL; } - if (unlikely(dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index == -1)) { - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index = + if (unlikely(dev_res->port.gid_tbl[sgid_idx].backend_gid_index == -1)) { + dev_res->port.gid_tbl[sgid_idx].backend_gid_index = rdma_backend_get_gid_index(backend_dev, - &dev_res->ports[0].gid_tbl[sgid_idx].gid); + &dev_res->port.gid_tbl[sgid_idx].gid); } pr_dbg("backend_gid_index=%d\n", - dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index); + dev_res->port.gid_tbl[sgid_idx].backend_gid_index); - return dev_res->ports[0].gid_tbl[sgid_idx].backend_gid_index; + return dev_res->port.gid_tbl[sgid_idx].backend_gid_index; } static void destroy_qp_hash_key(gpointer data) @@ -596,15 +596,13 @@ static void destroy_qp_hash_key(gpointer data) static void init_ports(RdmaDeviceResources *dev_res) { - int i, j; + int i; - memset(dev_res->ports, 0, sizeof(dev_res->ports)); + memset(&dev_res->port, 0, sizeof(dev_res->port)); - for (i = 0; i < MAX_PORTS; i++) { - dev_res->ports[i].state = IBV_PORT_DOWN; - for (j = 0; j < MAX_PORT_GIDS; j++) { - dev_res->ports[i].gid_tbl[j].backend_gid_index = -1; - } + dev_res->port.state = IBV_PORT_DOWN; + for (i = 0; i < MAX_PORT_GIDS; i++) { + dev_res->port.gid_tbl[i].backend_gid_index = -1; } } @@ -613,7 +611,7 @@ static void fini_ports(RdmaDeviceResources *dev_res, { int i; - dev_res->ports[0].state = IBV_PORT_DOWN; + dev_res->port.state = IBV_PORT_DOWN; for (i = 0; i < MAX_PORT_GIDS; i++) { rdma_rm_del_gid(dev_res, backend_dev, ifname, i); } diff --git a/hw/rdma/rdma_rm.h b/hw/rdma/rdma_rm.h index a7169b4e89..3c602c04c0 100644 --- a/hw/rdma/rdma_rm.h +++ b/hw/rdma/rdma_rm.h @@ -79,7 +79,7 @@ int rdma_rm_get_backend_gid_index(RdmaDeviceResources *dev_res, static inline union ibv_gid *rdma_rm_get_gid(RdmaDeviceResources *dev_res, int sgid_idx) { - return &dev_res->ports[0].gid_tbl[sgid_idx].gid; + return &dev_res->port.gid_tbl[sgid_idx].gid; } #endif diff --git a/hw/rdma/rdma_rm_defs.h b/hw/rdma/rdma_rm_defs.h index 7b3435f991..0ba61d1838 100644 --- a/hw/rdma/rdma_rm_defs.h +++ b/hw/rdma/rdma_rm_defs.h @@ -18,7 +18,7 @@ #include "rdma_backend_defs.h" -#define MAX_PORTS 1 +#define MAX_PORTS 1 /* Do not change - we support only one port */ #define MAX_PORT_GIDS 255 #define MAX_GIDS MAX_PORT_GIDS #define MAX_PORT_PKEYS 1 @@ -97,7 +97,7 @@ typedef struct RdmaRmPort { } RdmaRmPort; typedef struct RdmaDeviceResources { - RdmaRmPort ports[MAX_PORTS]; + RdmaRmPort port; RdmaRmResTbl pd_tbl; RdmaRmResTbl mr_tbl; RdmaRmResTbl uc_tbl; From patchwork Sat Dec 22 09:50:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017824 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="KbqKpYR7"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLlh4PbMz9sPd for ; Sat, 22 Dec 2018 21:07:44 +1100 (AEDT) Received: from localhost ([::1]:57799 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeC6-00025c-0w for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:07:42 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41757) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw6-0006PQ-QX for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw6-0006Xb-1a for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:10 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:36111) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw5-0006X9-RX for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:09 -0500 Received: by mail-wm1-x342.google.com with SMTP id p6so7871999wmc.1 for ; Sat, 22 Dec 2018 01:51:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1Zs5lAAC9I5AFJ51KWhpHS+IjB3oEO/vIRsOJCDh7iU=; b=KbqKpYR74LYtptgZ1/yFZHI1EHofJj2P91Si5/Uwxenrv8dya2deH+vDD1JEn+Q4fe EDuB/g7jZwjdDSW9NL6EJZ7pZeFc5Ec8QSR/1b3VJPGCfV5YDdFjeil+wLL/OrfcSNEE bkzNgmqVCS0HV8MkzGw0PijeCbhdEnsVJLiLBXqIaxOsYYrpmFx39WrfqffKDGWnoGhH y6dCFUyAI97FDfFu7sQqDkhm2h13lAt3dun74nk/KDMy3+6dD/EA3BxnBYUgjd242iV0 9lA7rgdLs3TcZ4MsRfccvgvAKhC78x+d1O5PMAfhKDdfDoOgA7n5/kmxy480VELVxEnU NLfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1Zs5lAAC9I5AFJ51KWhpHS+IjB3oEO/vIRsOJCDh7iU=; b=mx6Yo+fD/NLCRwF8dHDlJTGaSeW5yAH/8wI3fhww4Ew8vDmk5ixw+PkJaDBw/IYPxh 7s0T70JtHCeOnhSLumlUVMdenmqhHl2/wfPVIleOopS38l0/iu5gQIYubXHs9w9p8Kn3 GZl1D9kOG6pybOzvth4U8on8+9tJNvSUmdisGCZ4fG8gmz5u51/PNjH1/9MLd8x6WXMV RHTxLRbyCFnnOU/cjmov52zDvdo0JASWA/NIjcmSmd6lM5pCWj97lnsy6dliCWH+P0My mostE8DuCe/CeKK0y1e8CMJWXK0F+C62qpAdPjT1hisrh6Y6S0IL0odVBYc4jPmYGlwd PI+w== X-Gm-Message-State: AJcUukfPbGbe9GOHJyNznyaix0qsWqQ/soD3Nszwu6QnBdN5Vmhd1oFP /XnOKvZUuMdq7pA3xkvgj4wURpvo X-Google-Smtp-Source: AFSGD/Wvm8mXrM6+lh4foSuHiXpwZrEuyPqNvJzPRGAeDrKjXowBZsMv9xx3c/CqIiRMIa5LP469Sw== X-Received: by 2002:a1c:81ca:: with SMTP id c193mr6292869wmd.66.1545472268683; Sat, 22 Dec 2018 01:51:08 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:08 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:25 +0200 Message-Id: <20181222095036.29743-21-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH PULL 20/31] vl: Introduce shutdown_notifiers X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Notifier will be used for signaling shutdown event to inform system is shutdown. This will allow devices and other component to run some cleanup code needed before VM is shutdown. Signed-off-by: Yuval Shaia Reviewed-by: Cornelia Huck Signed-off-by: Marcel Apfelbaum --- include/sysemu/sysemu.h | 1 + vl.c | 15 ++++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h index c8efdeb376..e0d15da937 100644 --- a/include/sysemu/sysemu.h +++ b/include/sysemu/sysemu.h @@ -62,6 +62,7 @@ void qemu_register_wakeup_support(void); void qemu_system_shutdown_request(ShutdownCause reason); void qemu_system_powerdown_request(void); void qemu_register_powerdown_notifier(Notifier *notifier); +void qemu_register_shutdown_notifier(Notifier *notifier); void qemu_system_debug_request(void); void qemu_system_vmstop_request(RunState reason); void qemu_system_vmstop_request_prepare(void); diff --git a/vl.c b/vl.c index 46ebf813b3..8353d3c718 100644 --- a/vl.c +++ b/vl.c @@ -1577,6 +1577,8 @@ static NotifierList suspend_notifiers = NOTIFIER_LIST_INITIALIZER(suspend_notifiers); static NotifierList wakeup_notifiers = NOTIFIER_LIST_INITIALIZER(wakeup_notifiers); +static NotifierList shutdown_notifiers = + NOTIFIER_LIST_INITIALIZER(shutdown_notifiers); static uint32_t wakeup_reason_mask = ~(1 << QEMU_WAKEUP_REASON_NONE); ShutdownCause qemu_shutdown_requested_get(void) @@ -1828,6 +1830,12 @@ static void qemu_system_powerdown(void) notifier_list_notify(&powerdown_notifiers, NULL); } +static void qemu_system_shutdown(ShutdownCause cause) +{ + qapi_event_send_shutdown(shutdown_caused_by_guest(cause), cause); + notifier_list_notify(&shutdown_notifiers, &cause); +} + void qemu_system_powerdown_request(void) { trace_qemu_system_powerdown_request(); @@ -1840,6 +1848,11 @@ void qemu_register_powerdown_notifier(Notifier *notifier) notifier_list_add(&powerdown_notifiers, notifier); } +void qemu_register_shutdown_notifier(Notifier *notifier) +{ + notifier_list_add(&shutdown_notifiers, notifier); +} + void qemu_system_debug_request(void) { debug_requested = 1; @@ -1867,7 +1880,7 @@ static bool main_loop_should_exit(void) request = qemu_shutdown_requested(); if (request) { qemu_kill_report(); - qapi_event_send_shutdown(shutdown_caused_by_guest(request), request); + qemu_system_shutdown(request); if (no_shutdown) { vm_stop(RUN_STATE_SHUTDOWN); } else { From patchwork Sat Dec 22 09:50:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017823 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="buSfzkeQ"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLlW3sCfz9sPd for ; Sat, 22 Dec 2018 21:07:35 +1100 (AEDT) Received: from localhost ([::1]:57571 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeBw-0001y1-R1 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:07:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41768) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadw9-0006Rj-2o for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadw8-0006Z5-05 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:13 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:54152) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw7-0006Xx-97 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:11 -0500 Received: by mail-wm1-x341.google.com with SMTP id d15so7372055wmb.3 for ; Sat, 22 Dec 2018 01:51:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WF1jbwT1EVIB5mRe/xkEJebbweTWCBO5STWVa7iEoQk=; b=buSfzkeQG4b8fs0DYIHD7yLlHebeSy9EUB7WsVKHQw6HrlyahtMgJL6qCFqETwW2M9 c3MYew8iB9kxuz0dNaKSAMYs827HpcODpVOSKP2UHJXVwGgk4lVKM9bjvS1vzM1e8S9A d9j2r7m9VfMdMS28mANy218AzWNjz1IvgEGd4qJERrMKTC9t/DuOJlpChWfL8tuMPo7w L70HbcbRwf+W7XcoZB6p9N4O+hvxNaD4HrPdeu/FyN+K4gtZVLFM/0WVJ3S1/+brD3mF 6VCF13clg0yCMPKkroo9ycDcN6UmO+c5sdRN4PC7VramzBxe94pmEQH7qdSy+vdkkk5Q S6mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WF1jbwT1EVIB5mRe/xkEJebbweTWCBO5STWVa7iEoQk=; b=f5ieBUmyJgqWC/OYY9YrdcB5pbo1AxJukgmGsjvee0Xh6A0o4yGF98n9nFhGGAqZbM 28LDzkZX+BswCjE068rqtwh/JZ16QkzL5C7wl7A0pCtLB00KKWjzthpEQ0JBnU+eBujc WgFOsssjMphTg7WTEmGPmfHNN6frbbLRAFgreBaSpa2Gi+ZOxYeSPcZO1qgRjKLCk0jQ oLTug2Y1AUjmdaK7EOzURVprdS39l1wDqWWD1TtzWZLuQfDVoWqeN6HOhVdpXm46xHIS J7mRjrh9a4XGAnKEOKAaR+ic75Is6MocxvAHsIiXc6Z0TrJtaLorhpNRYXhK9VwppLPh 6IYQ== X-Gm-Message-State: AA+aEWahvJQIttgT4G+XxbFq0sig33Hr40H8+mIq9TWELBQpDHGUViC2 aG/jz9ryB5NIRgNr5/YfWvwTbfND X-Google-Smtp-Source: AFSGD/Vnoy9HCsCl1F413hjEwT24A5X9xmlrCErb6IbznE/XQ09f+zCRe6bsKq4EAZXrUWyeA58CKA== X-Received: by 2002:a1c:864f:: with SMTP id i76mr5829761wmd.83.1545472270011; Sat, 22 Dec 2018 01:51:10 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.08 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:09 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:26 +0200 Message-Id: <20181222095036.29743-22-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 21/31] hw/pvrdma: Clean device's resource when system is shutdown X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia In order to clean some external resources such as GIDs, QPs etc, register to receive notification when VM is shutdown. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma.h | 2 ++ hw/rdma/vmw/pvrdma_main.c | 15 +++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/hw/rdma/vmw/pvrdma.h b/hw/rdma/vmw/pvrdma.h index 10a3c4fb7c..ffae36986e 100644 --- a/hw/rdma/vmw/pvrdma.h +++ b/hw/rdma/vmw/pvrdma.h @@ -17,6 +17,7 @@ #define PVRDMA_PVRDMA_H #include "qemu/units.h" +#include "qemu/notify.h" #include "hw/pci/pci.h" #include "hw/pci/msix.h" #include "chardev/char-fe.h" @@ -87,6 +88,7 @@ typedef struct PVRDMADev { RdmaDeviceResources rdma_dev_res; CharBackend mad_chr; VMXNET3State *func0; + Notifier shutdown_notifier; } PVRDMADev; #define PVRDMA_DEV(dev) OBJECT_CHECK(PVRDMADev, (dev), PVRDMA_HW_NAME) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 150404dfa6..23dc9926e3 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -24,6 +24,7 @@ #include "hw/qdev-properties.h" #include "cpu.h" #include "trace.h" +#include "sysemu/sysemu.h" #include "../rdma_rm.h" #include "../rdma_backend.h" @@ -334,6 +335,9 @@ static void pvrdma_fini(PCIDevice *pdev) if (msix_enabled(pdev)) { uninit_msix(pdev, RDMA_MAX_INTRS); } + + pr_dbg("Device %s %x.%x is down\n", pdev->name, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn)); } static void pvrdma_stop(PVRDMADev *dev) @@ -559,6 +563,14 @@ static int pvrdma_check_ram_shared(Object *obj, void *opaque) return 0; } +static void pvrdma_shutdown_notifier(Notifier *n, void *opaque) +{ + PVRDMADev *dev = container_of(n, PVRDMADev, shutdown_notifier); + PCIDevice *pci_dev = PCI_DEVICE(dev); + + pvrdma_fini(pci_dev); +} + static void pvrdma_realize(PCIDevice *pdev, Error **errp) { int rc; @@ -632,6 +644,9 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) goto out; } + dev->shutdown_notifier.notify = pvrdma_shutdown_notifier; + qemu_register_shutdown_notifier(&dev->shutdown_notifier); + out: if (rc) { error_append_hint(errp, "Device fail to load\n"); From patchwork Sat Dec 22 09:50:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017826 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qG1BPCC0"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLmS0RDGz9sPd for ; Sat, 22 Dec 2018 21:08:24 +1100 (AEDT) Received: from localhost ([::1]:58701 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeCj-0002ZU-L8 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:08:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41779) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwI-0006Xj-87 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwE-0006eV-He for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:19 -0500 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]:50586) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadw9-0006ZM-6p for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:15 -0500 Received: by mail-wm1-x343.google.com with SMTP id n190so7400061wmd.0 for ; Sat, 22 Dec 2018 01:51:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Ee1OvyxgNjj/dtLFYkeAaONQE7uDVFlOxI8N9a7i8bk=; b=qG1BPCC0COH80y68aDr8pdqoSeBG5r4OxciDBvf9mA9dn8yQ7gVRYEORVsCPPQ9gcL SRNJ4Yz0n3KYYXjGiXUyzKgPgQY1C/ovvbYMW0FW/HYlxUJS0zScj1qVc+IySWKpbXWB mEdIKviMUdb6meuAavwSIWSagjtFwvxcMNFHHKPOrV7B8QHP+XYTYaHJePB6M/Wzm6ge raqwafDPYviaDi7sXCKYSSOaA5t5hhWGgGjzvY8bYmvaXFPhb8Y7cR4BfGB5fjRRHct0 Py35Ld8HBz/GcEK2mk4Winf0lODSEDWjisiLrrfuXQ5VE/K1zvOwNN2hE0sGOgfXsRdg 9FdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Ee1OvyxgNjj/dtLFYkeAaONQE7uDVFlOxI8N9a7i8bk=; b=DhJ6FVhxI0bAOolGvHlvAmP11YNE2u0LePAF+v0AWtnMDkJNyjjJlD6mVrb7VVLMed KBpHY0YjtIP4jggZftx5Tu7LAsnK5HwGFFSTQQiKlQKyUgC7yS2FJeUzx5crZefOeky0 9xxWlZ4fbyZnTGu8xhtDaMtl7WxWcZ6dEDiyhpgzszBPpQwJ4cwfiftNdBCHSAlwpWko XDrtJaqj7umpL3748eNLuViUSJEFgcG/SjGFN91ui3WWcDZDmwI9EVsFfiAmT3cUI+c6 m9Yd7CYKgvMEHDZKnX55dKz4+gd1a9P9O2pLQPeOEPvxwZB5KYmAyxTeBTX9rlvjTS4o TcwQ== X-Gm-Message-State: AA+aEWYN6IlczyC9Po+Fr8nHTr2oEa76aFlW5+M1n4Twd7SX6fATzfLk nax0LNxAPjL9/bVg7kWZQAXhS6pz X-Google-Smtp-Source: ALg8bN7A5+A/kRIcJxMeY9gqETF3cIc4959Z5gaX8MOt2Gr3ZpdkrI81vza7VSz4eUg696PPYL3IZQ== X-Received: by 2002:a1c:2686:: with SMTP id m128mr5668326wmm.52.1545472271326; Sat, 22 Dec 2018 01:51:11 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:10 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:27 +0200 Message-Id: <20181222095036.29743-23-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::343 Subject: [Qemu-devel] [PATCH PULL 22/31] hw/rdma: Do not use bitmap_zero_extend to free bitmap X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia bitmap_zero_extend is designed to work for extending, not for shrinking. Using g_free instead. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index b7d4ebe972..ca127c8c26 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -43,7 +43,7 @@ static inline void res_tbl_free(RdmaRmResTbl *tbl) { qemu_mutex_destroy(&tbl->lock); g_free(tbl->tbl); - bitmap_zero_extend(tbl->bitmap, tbl->tbl_sz, 0); + g_free(tbl->bitmap); } static inline void *res_tbl_get(RdmaRmResTbl *tbl, uint32_t handle) From patchwork Sat Dec 22 09:50:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017832 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AyDFLZA3"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLty33pjz9sPf for ; Sat, 22 Dec 2018 21:14:02 +1100 (AEDT) Received: from localhost ([::1]:38071 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeIA-0007DB-Vo for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:13:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41796) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zb-D1 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwI-0006hn-PW for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: from mail-wm1-x344.google.com ([2a00:1450:4864:20::344]:36113) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwE-0006Zw-R6 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:19 -0500 Received: by mail-wm1-x344.google.com with SMTP id p6so7872086wmc.1 for ; Sat, 22 Dec 2018 01:51:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nDuB47FVILPnmCoO2u7YwnDtswJQciP8HVfrc0G/x8E=; b=AyDFLZA3Q6ejTLMvYRSltlCS55tbCFrsp7QpJjO0ACxc0cW6mShY1JB1kSdKALlGgY LVjGsz7q+q9uU1cQFDNhUcNu8qrohFaHgWY9dBJpkYCo87mDrrFGgs7CIsbMSoHGEJmG pTb9jAtHBzw3p10UvTcVJUcTkuGrkPWoKaZUTRpb/S0LCfx1ShJSYNmdUn7WLxuYdSQX 83SwX9V3FvKPaW7LMV0PESgL4WVfwijWkZZsCxwv6FglCHj2iuf73C2qP7LXTq67qQ6T UfZUaLT+D5Jwq5aq1Z0Q6EbbbLmWAjzB9BjeWSj83Z7ENDNXKxwMA0PJ1ZQv65eyRr6e 4vCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nDuB47FVILPnmCoO2u7YwnDtswJQciP8HVfrc0G/x8E=; b=WsgQKRyRpuFDRid9TdS3jj3LmLW+5PGBjEa6c2KlmRv2pVWI/sQvnIRf28/RprtryG hvLRvTE2AVDQPc9gumTv84bhFGTh/iiH67DLta8zK6f047ERGnOKR7tB2Yew8R1HHXfX ASS7HA2RyS0FeLwmCO0E4NH5hqt9kWMLRUlA+lSfbKZ+AQCgULrmDwfVypIb9/6Kl3JP n3xHmWHHXvC7xVs6lYrIS7wBj/a7YjDi4zts2x/rlUqknSc6CXIX1y4C5/Eg9Z90Rmyg bw+V3ALMQXg4bZ7OqoBZFPL13Kv+HjhiPtIrrnoafTENctVyuFYf4S1JWC6rSQYj8ICi PMrA== X-Gm-Message-State: AJcUukdL+dFtnOPXz+aSmNZPi6+0YpTUE6lFBNI0cdtpZy9VfNW1tlPl KFunETxF/MYoAtJFJ2VR8NZwvCHA X-Google-Smtp-Source: AFSGD/WVGvHBH4i0zqNzz7ajheTlg1mubRwFdDbRvL1WsBEZ+u6quhrS4Th8muz3gpDSgpQcxztI/g== X-Received: by 2002:a1c:aa0f:: with SMTP id t15mr5804089wme.108.1545472272663; Sat, 22 Dec 2018 01:51:12 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.11 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:12 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:28 +0200 Message-Id: <20181222095036.29743-24-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::344 Subject: [Qemu-devel] [PATCH PULL 23/31] hw/rdma: Do not call rdma_backend_del_gid on an empty gid X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia When device goes down the function fini_ports loops over all entries in gid table regardless of the fact whether entry is valid or not. In case that entry is not valid we'd like to skip from any further processing in backend device. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_rm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/rdma/rdma_rm.c b/hw/rdma/rdma_rm.c index ca127c8c26..f5b1295890 100644 --- a/hw/rdma/rdma_rm.c +++ b/hw/rdma/rdma_rm.c @@ -555,6 +555,10 @@ int rdma_rm_del_gid(RdmaDeviceResources *dev_res, RdmaBackendDev *backend_dev, { int rc; + if (!dev_res->port.gid_tbl[gid_idx].gid.global.interface_id) { + return 0; + } + rc = rdma_backend_del_gid(backend_dev, ifname, &dev_res->port.gid_tbl[gid_idx].gid); if (rc) { From patchwork Sat Dec 22 09:50:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017839 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CW5wXPDX"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLyn1wkmz9sPd for ; Sat, 22 Dec 2018 21:17:21 +1100 (AEDT) Received: from localhost ([::1]:42256 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeLO-0001GL-Ue for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:17:18 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41888) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwO-0006ay-TW for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:30 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwN-0006la-9g for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:38456) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006cB-Vr for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: by mail-wr1-x442.google.com with SMTP id v13so7515556wrw.5 for ; Sat, 22 Dec 2018 01:51:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hay2rzA2FV36Ph6SD4BcCUe/zYRBRkL25GycwDCZ3nU=; b=CW5wXPDXSanRlqTfc7LMluvT/ZhTFBHj+tJAjAz4EDqwZKME0QM/kAlmiyTWM8AgZj RF2XP3n0qt2SWGBPGGlW7MRewfnz0ZpSw5z2vnySsuYWsAPxEURgcW668nMRBiZ58Mb4 WqCvisM1cB6VL8rUPOV8mkuI60Wjg0LE7mu/GBX9v6yfIhhYA4CKN1ynhfdihzYSsFv6 SVcEG1OTJMWeQ3TMOHEAR8YdvJWnMH3wdIXLS5TVtHSBEFiTLDfdfxOKyr5GfLJ6GlUm tNEXdybOcSU3M51UyX73zWUZ6uUIFffDjZcCnPnmu/wITa7gGJnLURr9WZjMu+YKrqWW cwfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hay2rzA2FV36Ph6SD4BcCUe/zYRBRkL25GycwDCZ3nU=; b=gojZExPjCpkGy+yhf44RqRDcyuBR6wP0NwRdlA6Nw0OR5Sxc3LCk3rRuTDCSk2gL9u /8oeroJZKDkf65iBvIUqMJf7vHbItqPvSX6s9UmhHMdNrLcZez8YFN1LKGw45FVTeroO fPzHbK4r3Y8m+oHkejautCK6PHRy1mL4xC/qAXGFf71BgjhbrOoCOBr/bCK8GEIehvGO G7oEF/eiYnhqFEBtXmBhlJgxtYGORbkGpvw5WtuAAeoh00xZ0lfsZQSb/L8KTjtk9O1w CMWTtXUPQ6hUDBdBnsmGSvuGsz9fKNv6fJA3Wi+pIbSrxgwlOYCg9ptuNjztvCSNwVe3 UHSQ== X-Gm-Message-State: AJcUukdKM3qyGGYBVppmjKHOP7ajzWetPUrcTxqRRKBRmkVb2ia48fz8 tZcM51+PrZu7QddfHdmm3NJ75zKf X-Google-Smtp-Source: ALg8bN4UtI8a1LwvFbzHjjJkKXGqghr+AL/Fvln3KpAdTLfhmW7OfFKN28DoPqkmw43P/csCmQA56Q== X-Received: by 2002:adf:eec9:: with SMTP id a9mr5539320wrp.242.1545472274010; Sat, 22 Dec 2018 01:51:14 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.12 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:13 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:29 +0200 Message-Id: <20181222095036.29743-25-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 24/31] docs: Update pvrdma device documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Yuval Shaia Interface with the device is changed with the addition of support for MAD packets. Adjust documentation accordingly. While there fix a minor mistake which may lead to think that there is a relation between using RXE on host and the compatibility with bare-metal peers. Signed-off-by: Yuval Shaia Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- docs/pvrdma.txt | 126 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 107 insertions(+), 19 deletions(-) diff --git a/docs/pvrdma.txt b/docs/pvrdma.txt index 5599318159..5175251b47 100644 --- a/docs/pvrdma.txt +++ b/docs/pvrdma.txt @@ -9,8 +9,9 @@ It works with its Linux Kernel driver AS IS, no need for any special guest modifications. While it complies with the VMware device, it can also communicate with bare -metal RDMA-enabled machines and does not require an RDMA HCA in the host, it -can work with Soft-RoCE (rxe). +metal RDMA-enabled machines as peers. + +It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe). It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be @@ -78,29 +79,116 @@ the required RDMA libraries. 3. Usage ======== + + +3.1 VM Memory settings +====================== Currently the device is working only with memory backed RAM and it must be mark as "shared": -m 1G \ -object memory-backend-ram,id=mb1,size=1G,share \ -numa node,memdev=mb1 \ -The pvrdma device is composed of two functions: - - Function 0 is a vmxnet Ethernet Device which is redundant in Guest - but is required to pass the ibdevice GID using its MAC. - Examples: - For an rxe backend using eth0 interface it will use its mac: - -device vmxnet3,addr=.0,multifunction=on,mac= - For an SRIOV VF, we take the Ethernet Interface exposed by it: - -device vmxnet3,multifunction=on,mac= - - Function 1 is the actual device: - -device pvrdma,addr=.1,backend-dev=,backend-gid-idx=,backend-port= - where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4) - Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC. - The rules of conversion are part of the RoCE spec, but since manual conversion - is not required, spotting problems is not hard: - Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a - MAC: 7c:fe:90:cb:74:3a - Note the difference between the first byte of the MAC and the GID. + +3.2 MAD Multiplexer +=================== +MAD Multiplexer is a service that exposes MAD-like interface for VMs in +order to overcome the limitation where only single entity can register with +MAD layer to send and receive RDMA-CM MAD packets. + +To build rdmacm-mux run +# make rdmacm-mux + +The application accepts 3 command line arguments and exposes a UNIX socket +to pass control and data to it. +-d rdma-device-name Name of RDMA device to register with +-s unix-socket-path Path to unix socket to listen (default /var/run/rdmacm-mux) +-p rdma-device-port Port number of RDMA device to register with (default 1) +The final UNIX socket file name is a concatenation of the 3 arguments so +for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2 +will be created. + +pvrdma requires this service. + +Please refer to contrib/rdmacm-mux for more details. + + +3.3 Service exposed by libvirt daemon +===================================== +The control over the RDMA device's GID table is done by updating the +device's Ethernet function addresses. +Usually the first GID entry is determined by the MAC address, the second by +the first IPv6 address and the third by the IPv4 address. Other entries can +be added by adding more IP addresses. The opposite is the same, i.e. +whenever an address is removed, the corresponding GID entry is removed. +The process is done by the network and RDMA stacks. Whenever an address is +added the ib_core driver is notified and calls the device driver add_gid +function which in turn update the device. +To support this in pvrdma device the device hooks into the create_bind and +destroy_bind HW commands triggered by pvrdma driver in guest. + +Whenever changed is made to the pvrdma port's GID table a special QMP +messages is sent to be processed by libvirt to update the address of the +backend Ethernet device. + +pvrdma requires that libvirt service will be up. + + +3.4 PCI devices settings +======================== +RoCE device exposes two functions - an Ethernet and RDMA. +To support it, pvrdma device is composed of two PCI functions, an Ethernet +device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The +Ethernet function can be used for other Ethernet purposes such as IP. + + +3.5 Device parameters +===================== +- netdev: Specifies the Ethernet device function name on the host for + example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet + device used to create it. +- ibdev: The IB device name on host for example rxe0, mlx5_0 etc. +- mad-chardev: The name of the MAD multiplexer char device. +- ibport: In case of multi-port device (such as Mellanox's HCA) this + specify the port to use. If not set 1 will be used. +- dev-caps-max-mr-size: The maximum size of MR. +- dev-caps-max-qp: Maximum number of QPs. +- dev-caps-max-sge: Maximum number of SGE elements in WR. +- dev-caps-max-cq: Maximum number of CQs. +- dev-caps-max-mr: Maximum number of MRs. +- dev-caps-max-pd: Maximum number of PDs. +- dev-caps-max-ah: Maximum number of AHs. + +Notes: +- The first 3 parameters are mandatory settings, the rest have their + defaults. +- The last 8 parameters (the ones that prefixed by dev-caps) defines the top + limits but the final values is adjusted by the backend device limitations. +- netdev can be extracted from ibdev's sysfs + (/sys/class/infiniband//device/net/) + + +3.6 Example +=========== +Define bridge device with vmxnet3 network backend: + + + + +
+ + +Define pvrdma device: + + + + + + + + + + From patchwork Sat Dec 22 09:50:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017837 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="D8ayGRVO"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLxP5Yflz9sPf for ; Sat, 22 Dec 2018 21:16:09 +1100 (AEDT) Received: from localhost ([::1]:41152 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeKF-0000bu-D6 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:16:07 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41849) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zp-UN for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwM-0006kw-S6 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:36111) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006cq-Lj for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: by mail-wm1-x341.google.com with SMTP id p6so7872160wmc.1 for ; Sat, 22 Dec 2018 01:51:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9iR+hXjh8mxFOwqrXYzH1/P0rkkiaMRtIEVQvpnNQuc=; b=D8ayGRVO2o6eRQ+jiewpaX3kKMoS+PpqNWLOi0wV2KqUhne9ifZA2lf/am44yThL0M dYfRyyZHBVuTh+ot2L6JlSQsXjy76ZafFCAole/hZ0JqAVUZRQ9Ey+aY9q0hlu6V5IWx 7/YgvxUf7kEXRf4Kmi4Jp9gcOuuTIti6/gRa8RCVv/oeVJQ7EFgAJp1iIl5RhmiLyKDT qfAPTBhhpRV5RX1C5IHrfgdBphT6yF1hxsMdWokDXWPWZixytjBNdtGGY/sDmyuLyJ+F jcfGVSQqi8Bj7pTmVO+mJNnN3W9SCx/JHfLLXsFtt4fiaJZOKuL9z9Juphzt1goT0qbF X5YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9iR+hXjh8mxFOwqrXYzH1/P0rkkiaMRtIEVQvpnNQuc=; b=f0FcsYV9H3/V1SxSkdSjhZ/8r0rPIQOz83RmullNNc3W6fxKzHsELgJp7fhXMBdIGk akFCZKckhfUrg04OVIOu6oKy+X3/b2lDE/TjLHaMUzHEjtnsUoUI2QPGK/2PsOOxjID9 +Zt/6m/mqgGR0r37mhMQDlc/RtZ+G63O2Di5wTOUkOj4VfZ5l3Cm+QWcB9G7ljB9kpBc epdM7xCpkFxHzjjUc0jmDY0Q6ZfhQ45T9eeZJ94nSHAvt190mZNZ8rzSBXnDXWGR0ku3 2NZDN/PuYZY8c21DTtSvVI0eF+8EB3t4SzSMDp6U/3LfK1XpWQ+BXcjNN7Hq0jqQbc4q PZTg== X-Gm-Message-State: AJcUuke522qGJchnvXT9e0O+GTy69k6G7E4U6s1OiQRebbaGXwgvsx3V STrU7vj+xBgxI1rxIQ5ZqQgv41f+ X-Google-Smtp-Source: ALg8bN6i4FsByksTccsv3DQRtxWjehfq1fKacmINEslwF9jbk/g4ZM3Ywb7HVWfM5bbqNj/pd5MPnA== X-Received: by 2002:a1c:da92:: with SMTP id r140mr6199922wmg.136.1545472275470; Sat, 22 Dec 2018 01:51:15 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:14 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:30 +0200 Message-Id: <20181222095036.29743-26-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 25/31] pvrdma: release device resources in case of an error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit If during pvrdma device initialisation an error occurs, pvrdma_realize() does not release memory resources, leading to memory leakage. Reported-by: Li Qiang Signed-off-by: Prasad J Pandit Message-Id: <20181212175817.815-1-ppandit@redhat.com> Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 23dc9926e3..64de16fb52 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -573,7 +573,7 @@ static void pvrdma_shutdown_notifier(Notifier *n, void *opaque) static void pvrdma_realize(PCIDevice *pdev, Error **errp) { - int rc; + int rc = 0; PVRDMADev *dev = PVRDMA_DEV(pdev); Object *memdev_root; bool ram_shared = false; @@ -649,6 +649,7 @@ static void pvrdma_realize(PCIDevice *pdev, Error **errp) out: if (rc) { + pvrdma_fini(pdev); error_append_hint(errp, "Device fail to load\n"); } } From patchwork Sat Dec 22 09:50:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017827 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="i0FOrwxU"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLqD5XhNz9sPd for ; Sat, 22 Dec 2018 21:10:48 +1100 (AEDT) Received: from localhost ([::1]:33754 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeF4-0004hO-Cv for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:10:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41836) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zg-N8 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwM-0006kq-Qb for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: from mail-wr1-x441.google.com ([2a00:1450:4864:20::441]:35006) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006do-KE for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: by mail-wr1-x441.google.com with SMTP id 96so7530874wrb.2 for ; Sat, 22 Dec 2018 01:51:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=46yTJGRUMonQjaBI5LtgoPoU4gPLXELTxYuktuqxajY=; b=i0FOrwxU/nbjiB0A5jkRGprRs/234JfZW9pX6H05D2AEaAUQo7IKLHBmvWuC0u9/he Jbb2SIIhcfu5kqDixXR1yPLaPCmCkwLwVmbxGE/tN3sGW1jATYJ0If0MUDbfiASeRCFt OW3X8OQDI+iOcSs18TkzAxIu7Hb6qvr2SBROF7h/la0lYaUiq7wck0pX0osyDCutv2D3 9qfEmmQ8vXqEvorrTI3pQU1qFSzylfJBDf6PnRN/LyKoy5DSGmZzFSLIWidOsSdMI5M2 YGfcRUQxnMlz3e9gcei/5YqWBM/y8zIyzf8E7D8DdHIUZAnFrKyoIYDF343ZDDYt8V98 qczw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=46yTJGRUMonQjaBI5LtgoPoU4gPLXELTxYuktuqxajY=; b=G0FscdjJqUVpa4MuWvnQayOw0skiA4ZjzVib+n3iv8aMFKY7CIIU/IQo89tNxT8mbX SCW/SzUofDjddosyt3W4l5AsCimE0N2cW2U52iAgaUtflLlYJ5vhK7ox3DDCbUHDDxNG iyXlM24pGhPwuBw2DxUbEBpu2HSQt7yC2X3XHFhFzA5w03mx/Na22gvFR883slhiQKU6 0pN+FB16ZlMW7AvHvoDWkghZcz+JhatuL1DIgJq2P6FOG9EsNwMVBpMGh0BOiLNtx3iC wmjs6GN7RK3w8lvRsTcqALsuhXIN8BDuHQKjSCPrJpSfNzed1HRdON32o/dWhWqH84Jj ReLA== X-Gm-Message-State: AJcUukc2hAAh4WWPPxbiPioia95L/YI0GwHDtH92nSgwIhrPk4Uxeopt h7FvVCNf3JjP9I1RwEVcIQ74N9pD X-Google-Smtp-Source: ALg8bN4gXfP7w9hnw5i8HmgaV6MZmeHv6VoAGvxzuPTtcvNVVZQWf8YGT0PvridGHIssEPO/cy5w6g== X-Received: by 2002:adf:f390:: with SMTP id m16mr5522898wro.71.1545472276816; Sat, 22 Dec 2018 01:51:16 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.15 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:16 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:31 +0200 Message-Id: <20181222095036.29743-27-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::441 Subject: [Qemu-devel] [PATCH PULL 26/31] rdma: check num_sge does not exceed MAX_SGE X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit rdma back-end has scatter/gather array ibv_sge[MAX_SGE=4] set to have 4 elements. A guest could send a 'PvrdmaSqWqe' ring element with 'num_sge' set to > MAX_SGE, which may lead to OOB access issue. Add check to avoid it. Reported-by: Saar Amar Signed-off-by: Prasad J Pandit Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index ae1e4dcb29..bd4710d16f 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -476,9 +476,9 @@ void rdma_backend_post_send(RdmaBackendDev *backend_dev, } pr_dbg("num_sge=%d\n", num_sge); - if (!num_sge) { - pr_dbg("num_sge=0\n"); - complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + if (!num_sge || num_sge > MAX_SGE) { + pr_dbg("invalid num_sge=%d\n", num_sge); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_NUM_SGE, ctx); return; } @@ -603,9 +603,9 @@ void rdma_backend_post_recv(RdmaBackendDev *backend_dev, } pr_dbg("num_sge=%d\n", num_sge); - if (!num_sge) { - pr_dbg("num_sge=0\n"); - complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_NO_SGE, ctx); + if (!num_sge || num_sge > MAX_SGE) { + pr_dbg("invalid num_sge=%d\n", num_sge); + complete_work(IBV_WC_GENERAL_ERR, VENDOR_ERR_INV_NUM_SGE, ctx); return; } From patchwork Sat Dec 22 09:50:32 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017829 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="o2C8Fieq"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLr65MdPz9sPd for ; Sat, 22 Dec 2018 21:11:34 +1100 (AEDT) Received: from localhost ([::1]:34983 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeFo-0005SW-DI for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:11:32 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41797) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zc-D3 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwK-0006j5-Ld for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: from mail-wm1-x341.google.com ([2a00:1450:4864:20::341]:33813) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwJ-0006es-QP for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:24 -0500 Received: by mail-wm1-x341.google.com with SMTP id y185so15834187wmd.1 for ; Sat, 22 Dec 2018 01:51:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kFK2hK6vIxFug568DSFeRY/jEfInSuH4CRmJF+C9nX0=; b=o2C8FieqQydLjbOi3JK3ZJkx+zR7pt08YJ98OfWnD2LZ6WhptUai9PeiWt+AJHc/Wf XTMfQ1SR0wwJrhBfOVJHg/iYhIrkuthSkIJvm/EPUlHJSuWNXkUvllWpC9OIr9MsnUM2 25o4QZN9U7rtQXSC9cgIq094O4msjlegcmtLRI3Mu+jPIEfWm2VrtWwG1/L+3n0ok7KV RHwKZ0ULTKSeTzlj8Pn6jPQaw5/HV2A8ZkQVXzgnyrmv9g1Z0Mxzc/77GUCnuJSLM9/k 8/NZJiQWmt2MQcPzPH+HWIiHMG0pXKjnwU4MK5m9iDFlewU6Cp4p2Qyb3VskSCMLSUYK andA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=kFK2hK6vIxFug568DSFeRY/jEfInSuH4CRmJF+C9nX0=; b=sQD8962RnJaPUYZK98OJr3AMYkKUAmvaDHKbC02TEhZMZ4WbdrVt2HIPCtKmiiqLeP 4IMMUmHrJy7ptpAFBt8H9/1G4pXKiEoXhxezHLXj666qg4y8NBOmXv8ff+xSGduZ2XKV MMaFJTbcbWS3rv9/UBV2UJfUhhYVsZ0q0/f+k7NtfbjRGYmHnDn52D1zGM4CsYaVgtOB EUwCcM7DD6VHfVLoPBPJS6HiFIJpyqJdgKwPLGZdraT6m6Sl4VeXNx5fDkufzJAUhRxR oxwiCBj2M2qKDycbkpmheL3xKzvd4u8mUK12S2xeUuscRA9CcGdbyYqvUoExTWl4LEpi gzSQ== X-Gm-Message-State: AA+aEWZZWlJIOCrxWEO+vsV53bsQVaRblUnIPu0rcAT/WJz5uSfw/Pmv ivyZawvFiJqhzcnL8/iXl1iUxqMf X-Google-Smtp-Source: ALg8bN4s/tfeV9OXjxvuXWKpLTnYS+VHQ38L50Og8WrdUsZU9V22j8xwZE6QtILlTFThTLUh5CIocg== X-Received: by 2002:a1c:3d44:: with SMTP id k65mr5822756wma.76.1545472278155; Sat, 22 Dec 2018 01:51:18 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.16 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:17 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:32 +0200 Message-Id: <20181222095036.29743-28-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::341 Subject: [Qemu-devel] [PATCH PULL 27/31] pvrdma: add uar_read routine X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit Define skeleton 'uar_read' routine. Avoid NULL dereference. Reported-by: Li Qiang Signed-off-by: Prasad J Pandit Reviewed-by: Marcel Apfelbaum Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_main.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hw/rdma/vmw/pvrdma_main.c b/hw/rdma/vmw/pvrdma_main.c index 64de16fb52..838ad8a949 100644 --- a/hw/rdma/vmw/pvrdma_main.c +++ b/hw/rdma/vmw/pvrdma_main.c @@ -448,6 +448,11 @@ static const MemoryRegionOps regs_ops = { }, }; +static uint64_t uar_read(void *opaque, hwaddr addr, unsigned size) +{ + return 0xffffffff; +} + static void uar_write(void *opaque, hwaddr addr, uint64_t val, unsigned size) { PVRDMADev *dev = opaque; @@ -489,6 +494,7 @@ static void uar_write(void *opaque, hwaddr addr, uint64_t val, unsigned size) } static const MemoryRegionOps uar_ops = { + .read = uar_read, .write = uar_write, .endianness = DEVICE_LITTLE_ENDIAN, .impl = { From patchwork Sat Dec 22 09:50:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017830 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Sl85Z5Uh"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLt55dndz9sPd for ; Sat, 22 Dec 2018 21:13:17 +1100 (AEDT) Received: from localhost ([::1]:37108 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeHS-0006eJ-89 for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:13:14 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41848) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zn-Ts for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwN-0006l4-2V for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: from mail-wr1-x442.google.com ([2a00:1450:4864:20::442]:35007) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006g9-RZ for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: by mail-wr1-x442.google.com with SMTP id 96so7530930wrb.2 for ; Sat, 22 Dec 2018 01:51:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uItoruqcg46l1s+nPMTY7KVp/vApUca1inUPnVc3+6Q=; b=Sl85Z5UhQEgAvANsn8lf0qGzoMtdCgM6RmjUpzoWn4HN5W95vHXf6PSS57LLbKinFM OCYbN6eeJWTLoW3lmOHF8uMlWYUK2xiY/xCFMt35P2TOjEMxc3W4ouGP25jFv6iugJH1 r9DeAJuKS2NQaDL4VH5bfNN7spW1Pgwv02UzlZj+VAg9dolriSgEig1+Ffj6miUuStRX VZzZKpmzX5p/H9ydFRs4XoZ3m/HZYwoBVZFQSTGrdSqQijxc+6fmYUg7WIQcAs6H/7Rc OmFZmNCkyZdSTdE7X3V6eYorsXNTryF+rLzIMpQGYVVq1UNR3P573eloxfTgAL1czzqS a4GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uItoruqcg46l1s+nPMTY7KVp/vApUca1inUPnVc3+6Q=; b=gm/zKwAvwonvb5soRy5p5elSehBS1Vod3jXlYXV8fS9rsxjs6b97bmE7slq2ByysoH yCdqP3sti7cTcDZHJoJXniun9a7kI0mkjn6xHwMq1Kk3sb546G9/DvwT37zmJiojYcNf z7YYWxDlOKG3DR0r3bIcv66olD94Iq73EuwBEs5HPhdnhKX/MUoNIRjj8q7YTDgDb+dL R1T8By7gs+L8eL26u1ndO8klVL2A/IXjyoXURsKuqADLqSTtQ/B33FgeUH1ocsgQe41w CrDniqlIwuEjaiZpySF2HV1tg8o89NbjSyz/pWIpLIusbsUeIZPlO+UdjUvHAcmKChJU CEVg== X-Gm-Message-State: AJcUuke3iLdBHlCu5zl+RCMtOpwIEA/Ba6XpqL6ul/QhxLNZRE1YnT+f b6eCAzWnO9Z5MVpy/skyxcSnxtKB X-Google-Smtp-Source: ALg8bN4R6+e62vb7Twj32pyHhhcnMnXJDNB0Sfglre5xqmluO2WHkLnb6Z9nhXb/sg9dClkLVGG4TA== X-Received: by 2002:adf:92c7:: with SMTP id 65mr5383016wrn.228.1545472279570; Sat, 22 Dec 2018 01:51:19 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:19 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:33 +0200 Message-Id: <20181222095036.29743-29-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::442 Subject: [Qemu-devel] [PATCH PULL 28/31] pvrdma: check number of pages when creating rings X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit When creating CQ/QP rings, an object can have up to PVRDMA_MAX_FAST_REG_PAGES 8 pages. Check 'npages' parameter to avoid excessive memory allocation or a null dereference. Reported-by: Li Qiang Signed-off-by: Prasad J Pandit Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index 3b94545761..f236ac4795 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -259,6 +259,11 @@ static int create_cq_ring(PCIDevice *pci_dev , PvrdmaRing **ring, int rc = -EINVAL; char ring_name[MAX_RING_NAME_SZ]; + if (!nchunks || nchunks > PVRDMA_MAX_FAST_REG_PAGES) { + pr_dbg("invalid nchunks: %d\n", nchunks); + return rc; + } + pr_dbg("pdir_dma=0x%llx\n", (long long unsigned int)pdir_dma); dir = rdma_pci_dma_map(pci_dev, pdir_dma, TARGET_PAGE_SIZE); if (!dir) { @@ -372,6 +377,12 @@ static int create_qp_rings(PCIDevice *pci_dev, uint64_t pdir_dma, char ring_name[MAX_RING_NAME_SZ]; uint32_t wqe_sz; + if (!spages || spages > PVRDMA_MAX_FAST_REG_PAGES + || !rpages || rpages > PVRDMA_MAX_FAST_REG_PAGES) { + pr_dbg("invalid pages: %d, %d\n", spages, rpages); + return rc; + } + pr_dbg("pdir_dma=0x%llx\n", (long long unsigned int)pdir_dma); dir = rdma_pci_dma_map(pci_dev, pdir_dma, TARGET_PAGE_SIZE); if (!dir) { From patchwork Sat Dec 22 09:50:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017825 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kA72H5P5"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLlm3RHMz9sPd for ; Sat, 22 Dec 2018 21:07:48 +1100 (AEDT) Received: from localhost ([::1]:57935 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeCA-0002A2-2P for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:07:46 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41839) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwN-0006Zh-Nd for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwM-0006ke-NW for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]:35373) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006h6-Hf for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:26 -0500 Received: by mail-wm1-x330.google.com with SMTP id c126so7866321wmh.0 for ; Sat, 22 Dec 2018 01:51:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PEoOOEXeOr5vMltwtegCHT2dihgZRrLbSSSTOix5F/A=; b=kA72H5P51yh6DrjOOTTT7hxxci9DgRUMeyrMrF6dHQohT5RoE/kzcjJ6fktOZorGGI yCgdxBVv0hOCRTuuRyROxyHCcZxcz1FCUanJwI210jJCE3xKIXOJTaTmCaVuhFi1HMi2 T33gndNAIUDuRUU/EQZLELBMdyFadm/ebZL+OdTZ30CeWvMAipFDlENXkO1Kp1dW5b1U ujdyv9XI31bPYuuwN+TPRKEYH4aQha4hPdqPkLoZ/m1gCfCkKMNH2SZVKg1/q5xl9drI oRVfKmFkw+MZy+LdXZM/9inpd3jJgFAHufbBWbQBByGJrVGl8JvvSam0uSY6PZuDmyaW UDKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PEoOOEXeOr5vMltwtegCHT2dihgZRrLbSSSTOix5F/A=; b=LF2/rylRrvUh+RKBp7+EKAI10khQ/rZ5PnaTHOHfCH77Ir4yrctcmA/dAChvtRA4GR qnhZk898iSWeCgPOwG6HFExqAvdpLhP7dfVU2v6dKejOsClSaeveBwruielBqW6/hlbI bTocbMlX6O84luoel6AywLomtoHOGUXFVpiUsLLjeG/PmxfeoUUUlFZ9ujMT9q0b9R/8 SFhfoTo66F8P4jDJm71sCunoeL6bNLqG4BvXKxG7D4MelA1pCSl0C3Fx+zkFY1pItPs4 PJ4KJWgy1mGKa6AbIo6pijDqnJJweDW8UAiabnHcu4SBV53fQMg4oANlNpxQbKKxsqHS rq0Q== X-Gm-Message-State: AA+aEWZ9tCbIptX9nLWnYKbYDz1uIbwb41XEH8TLpEWs8s+8fStI/XqY 6B8DS8QPoF3lcYtEqR+g1sGCxmfR X-Google-Smtp-Source: AFSGD/WpDjm91bepbRKPCfNrqgN+8VlXIQcxSof/sd8udHgDnmQ8FL5x0IohObw60muRz8wOnUf3PA== X-Received: by 2002:a1c:e287:: with SMTP id z129mr5886722wmg.71.1545472280975; Sat, 22 Dec 2018 01:51:20 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:20 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:34 +0200 Message-Id: <20181222095036.29743-30-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::330 Subject: [Qemu-devel] [PATCH PULL 29/31] pvrdma: release ring object in case of an error X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit create_cq and create_qp routines allocate ring object, but it's not released in case of an error, leading to memory leakage. Reported-by: Li Qiang Signed-off-by: Prasad J Pandit Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_cmd.c | 37 ++++++++++++++++++++++++++----------- 1 file changed, 26 insertions(+), 11 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_cmd.c b/hw/rdma/vmw/pvrdma_cmd.c index f236ac4795..89920887bf 100644 --- a/hw/rdma/vmw/pvrdma_cmd.c +++ b/hw/rdma/vmw/pvrdma_cmd.c @@ -313,6 +313,14 @@ out: return rc; } +static void destroy_cq_ring(PvrdmaRing *ring) +{ + pvrdma_ring_free(ring); + /* ring_state was in slot 1, not 0 so need to jump back */ + rdma_pci_dma_unmap(ring->dev, --ring->ring_state, TARGET_PAGE_SIZE); + g_free(ring); +} + static int create_cq(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { @@ -335,6 +343,10 @@ static int create_cq(PVRDMADev *dev, union pvrdma_cmd_req *req, rc = rdma_rm_alloc_cq(&dev->rdma_dev_res, &dev->backend_dev, cmd->cqe, &resp->cq_handle, ring); + if (rc) { + destroy_cq_ring(ring); + } + resp->cqe = cmd->cqe; return rc; @@ -356,10 +368,7 @@ static int destroy_cq(PVRDMADev *dev, union pvrdma_cmd_req *req, } ring = (PvrdmaRing *)cq->opaque; - pvrdma_ring_free(ring); - /* ring_state was in slot 1, not 0 so need to jump back */ - rdma_pci_dma_unmap(PCI_DEVICE(dev), --ring->ring_state, TARGET_PAGE_SIZE); - g_free(ring); + destroy_cq_ring(ring); rdma_rm_dealloc_cq(&dev->rdma_dev_res, cmd->cq_handle); @@ -457,6 +466,17 @@ out: return rc; } +static void destroy_qp_rings(PvrdmaRing *ring) +{ + pr_dbg("sring=%p\n", &ring[0]); + pvrdma_ring_free(&ring[0]); + pr_dbg("rring=%p\n", &ring[1]); + pvrdma_ring_free(&ring[1]); + + rdma_pci_dma_unmap(ring->dev, ring->ring_state, TARGET_PAGE_SIZE); + g_free(ring); +} + static int create_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, union pvrdma_cmd_resp *rsp) { @@ -486,6 +506,7 @@ static int create_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, cmd->max_recv_sge, cmd->recv_cq_handle, rings, &resp->qpn); if (rc) { + destroy_qp_rings(rings); return rc; } @@ -558,13 +579,7 @@ static int destroy_qp(PVRDMADev *dev, union pvrdma_cmd_req *req, rdma_rm_dealloc_qp(&dev->rdma_dev_res, cmd->qp_handle); ring = (PvrdmaRing *)qp->opaque; - pr_dbg("sring=%p\n", &ring[0]); - pvrdma_ring_free(&ring[0]); - pr_dbg("rring=%p\n", &ring[1]); - pvrdma_ring_free(&ring[1]); - - rdma_pci_dma_unmap(PCI_DEVICE(dev), ring->ring_state, TARGET_PAGE_SIZE); - g_free(ring); + destroy_qp_rings(ring); return 0; } From patchwork Sat Dec 22 09:50:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017828 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="eemiCaGf"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLqp7226z9sPd for ; Sat, 22 Dec 2018 21:11:18 +1100 (AEDT) Received: from localhost ([::1]:34587 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeFY-0005DG-JI for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:11:16 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41853) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwO-0006a0-3F for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwN-0006lQ-8i for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from mail-wm1-x342.google.com ([2a00:1450:4864:20::342]:33815) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwN-0006hx-1t for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: by mail-wm1-x342.google.com with SMTP id y185so15834305wmd.1 for ; Sat, 22 Dec 2018 01:51:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RRbyjem891KEiMY+CICpMme86Nzgdix+0oW/lcmTBFs=; b=eemiCaGffyaaQiUPU8v3amYSbfNTXO8S58HhvzkBlPx/YG5bDmHZnrl0TU3oalG5lW zIk506zCyhFtaA1o0fPgc1cn1Kzmr2EQAoY7wI/2bnqTqXUadxHn3shSVkerkhWKz25N xbYkGQFYokQJC5YC13rOrgs333gZ6m5dVzRQ3Gyhnc7DMV6p0YzoM481BibSU6GHcDy1 Dm0NyjwLTiRA6lFsnTeotG8Aejbeq41ykN+5O2x6p6EjtbtlWZpakmNdp+gUCLLRtURO tx80YpJgu9ca2Fw7hwuD3Ph4S1CORW4lXwUVCykf9XLKfEWlz/Clvf1B9qs7+O7O3wof OXFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RRbyjem891KEiMY+CICpMme86Nzgdix+0oW/lcmTBFs=; b=W7BDQTJKeFsIw1qCGbu5s2FX+qh6TZ9B59UNnUxv/vSspCTMqFJtl8B5G2+3Jb6GBk DPVdLGQplKOXr+xC8xP+cinxISX2ab9CHw9ZQDwmNhWkAQcjfU0LdWqkUk2d4fddUCt5 aa/VqLOkE91BFDBN3OmDpE0stP4g7hctsXH8aprM2W/KObIgO1K+TD6A8cjNoencFZuI YzvBNqh8gKXHtTmkIFPg3UiGWUSz+5GXe4A9WqfJuLako2/R1uQoNBX7c2HR2RjQjo4P VFGsdWLC4W7O+RSXiddmEvp+rVphwSXCQQLXlOeGuYGsAsHzbVxwKV7YqARZfAuKe0t2 I/PQ== X-Gm-Message-State: AA+aEWZvMW4kwhKyOiaWICU2onWZwkUh+fFggqQiXjGU6I3w6MvsgRQ/ wwoWV55uJSkKVcGdkXgBRrwtSX0p X-Google-Smtp-Source: AFSGD/Ua+pX6FzjHmHEJ/7F/6/J6NCyCFnq6mm7Ct1P0gYfyATlqSL4ab/0i5qd66+BN+AoVfEzYzQ== X-Received: by 2002:a7b:c24c:: with SMTP id b12mr5441834wmj.29.1545472282286; Sat, 22 Dec 2018 01:51:22 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:21 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:35 +0200 Message-Id: <20181222095036.29743-31-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::342 Subject: [Qemu-devel] [PATCH PULL 30/31] rdma: remove unused VENDOR_ERR_NO_SGE macro X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit With commit 4481985c (rdma: check num_sge does not exceed MAX_SGE) macro VENDOR_ERR_NO_SGE is no longer in use - delete it. Signed-off-by: Prasad J Pandit Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/rdma_backend.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c index bd4710d16f..c28bfbd44d 100644 --- a/hw/rdma/rdma_backend.c +++ b/hw/rdma/rdma_backend.c @@ -37,12 +37,11 @@ #define VENDOR_ERR_TOO_MANY_SGES 0x202 #define VENDOR_ERR_NOMEM 0x203 #define VENDOR_ERR_QP0 0x204 -#define VENDOR_ERR_NO_SGE 0x205 +#define VENDOR_ERR_INV_NUM_SGE 0x205 #define VENDOR_ERR_MAD_SEND 0x206 #define VENDOR_ERR_INVLKEY 0x207 #define VENDOR_ERR_MR_SMALL 0x208 #define VENDOR_ERR_INV_MAD_BUFF 0x209 -#define VENDOR_ERR_INV_NUM_SGE 0x210 #define THR_NAME_LEN 16 #define THR_POLL_TO 5000 From patchwork Sat Dec 22 09:50:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcel Apfelbaum X-Patchwork-Id: 1017831 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::12; helo=lists.nongnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="eJd4mREC"; dkim-atps=neutral Received: from lists.nongnu.org (lists.nongnu.org [IPv6:2001:4830:134:3::12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43MLtg3zS5z9sPd for ; Sat, 22 Dec 2018 21:13:46 +1100 (AEDT) Received: from localhost ([::1]:37733 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gaeHw-00071v-0N for incoming@patchwork.ozlabs.org; Sat, 22 Dec 2018 05:13:44 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41855) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gadwO-0006a3-45 for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gadwN-0006lE-3x for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:28 -0500 Received: from mail-wm1-x343.google.com ([2a00:1450:4864:20::343]:55946) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1gadwM-0006ij-Tf for qemu-devel@nongnu.org; Sat, 22 Dec 2018 04:51:27 -0500 Received: by mail-wm1-x343.google.com with SMTP id y139so7337109wmc.5 for ; Sat, 22 Dec 2018 01:51:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=DeVlDfn3JZJQSiz2sMJdc5hnvsGaQvcfATyU/wmu9dU=; b=eJd4mRECnHUBI1Fa6L14gJPYJO8oDB4vMSqOeQ1Sb7S7mB/WRGoGvr8vaL0BS9RI9r 1p+v/FOLy5LzRaFMJz7gSUAJl1sgj0fXfmkF7riyHJnyl6ajOY2QBzzOSCCUgqJ5xZd/ GMt97E4MMwf3bqE0BIdCbJiUc96/n2vUv/1+jfFS8GmhzsxtHSh+XkrNgfjPOg12l/4G MLTmSCC5QyPR3Wnp+1ijEpWGFL08EMThEY6Bj4OmNTaxPtfF6+P2y+eh21zYvaMEtXzI 035Y8PUjK9IuJmL6uy3jz8pByuyDanFajtN5FnCcJiYdEKeu/1WLpp+yPJDLBXlfQhZM PXag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=DeVlDfn3JZJQSiz2sMJdc5hnvsGaQvcfATyU/wmu9dU=; b=LGv24+zBeySZHzKqwRXpFjHT5YCkJ6gNsQIlUrNRJ48qHwUVEiRq4cQKf2RsXX/M91 edD8rcRcjXDa55GXrWyWMqs71CPgqvvdFT+6nZ0MahnQG+Sae3P5O9n+VzvJyGh15qww OFPAb8pdzixOqlKOuTNKov0d/m7j9kXuu/xGUBGqCFFjil7BWJdiV63OxG+ZxiH94q5j EHGNznM8PUOmzTq2dXT2lBicPejUA6FoNDohxkmVd+V8iM2u5Voh+0v0n01Ozsfundsx UOugUsz8ZP7srRhLYQMh0e64wPzStYAVTXarMFLjgdCpkBDhaULrvKDthkBbxyScWy6C nsNg== X-Gm-Message-State: AJcUukcy0ZGgAfEw0GkDCPTRLRXFHEQZowN5nMBG+xLeu1UgNyjpndkR t6I9Bb3fsb+tO9K19ci1j740+uWG X-Google-Smtp-Source: ALg8bN6dwxdawntc0SXXwUXCa5mJnVuuRyU+wQHJZ1RTAdB4sjwsPC+/SmbpAcFJr+SflMHJLnLLbA== X-Received: by 2002:a1c:8d49:: with SMTP id p70mr6329354wmd.68.1545472283683; Sat, 22 Dec 2018 01:51:23 -0800 (PST) Received: from localhost.localdomain ([176.228.155.165]) by smtp.gmail.com with ESMTPSA id s3sm9557371wmj.23.2018.12.22.01.51.22 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 22 Dec 2018 01:51:23 -0800 (PST) From: Marcel Apfelbaum To: qemu-devel@nongnu.org, peter.maydell@linaro.org Date: Sat, 22 Dec 2018 11:50:36 +0200 Message-Id: <20181222095036.29743-32-marcel.apfelbaum@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> References: <20181222095036.29743-1-marcel.apfelbaum@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::343 Subject: [Qemu-devel] [PATCH PULL 31/31] pvrdma: check return value from pvrdma_idx_ring_has_ routines X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yuval.shaia@oracle.com, pjp@fedoraproject.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" From: Prasad J Pandit pvrdma_idx_ring_has_[data/space] routines also return invalid index PVRDMA_INVALID_IDX[=-1], if ring has no data/space. Check return value from these routines to avoid plausible infinite loops. Reported-by: Li Qiang Signed-off-by: Prasad J Pandit Reviewed-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- hw/rdma/vmw/pvrdma_dev_ring.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/hw/rdma/vmw/pvrdma_dev_ring.c b/hw/rdma/vmw/pvrdma_dev_ring.c index 01247fc041..e8e5b502f6 100644 --- a/hw/rdma/vmw/pvrdma_dev_ring.c +++ b/hw/rdma/vmw/pvrdma_dev_ring.c @@ -73,23 +73,16 @@ out: void *pvrdma_ring_next_elem_read(PvrdmaRing *ring) { + int e; unsigned int idx = 0, offset; - /* - pr_dbg("%s: t=%d, h=%d\n", ring->name, ring->ring_state->prod_tail, - ring->ring_state->cons_head); - */ - - if (!pvrdma_idx_ring_has_data(ring->ring_state, ring->max_elems, &idx)) { + e = pvrdma_idx_ring_has_data(ring->ring_state, ring->max_elems, &idx); + if (e <= 0) { pr_dbg("No more data in ring\n"); return NULL; } offset = idx * ring->elem_sz; - /* - pr_dbg("idx=%d\n", idx); - pr_dbg("offset=%d\n", offset); - */ return ring->pages[offset / TARGET_PAGE_SIZE] + (offset % TARGET_PAGE_SIZE); } @@ -105,20 +98,20 @@ void pvrdma_ring_read_inc(PvrdmaRing *ring) void *pvrdma_ring_next_elem_write(PvrdmaRing *ring) { - unsigned int idx, offset, tail; + int idx; + unsigned int offset, tail; - /* - pr_dbg("%s: t=%d, h=%d\n", ring->name, ring->ring_state->prod_tail, - ring->ring_state->cons_head); - */ - - if (!pvrdma_idx_ring_has_space(ring->ring_state, ring->max_elems, &tail)) { + idx = pvrdma_idx_ring_has_space(ring->ring_state, ring->max_elems, &tail); + if (idx <= 0) { pr_dbg("CQ is full\n"); return NULL; } idx = pvrdma_idx(&ring->ring_state->prod_tail, ring->max_elems); - /* TODO: tail == idx */ + if (idx < 0 || tail != idx) { + pr_dbg("invalid idx\n"); + return NULL; + } offset = idx * ring->elem_sz; return ring->pages[offset / TARGET_PAGE_SIZE] + (offset % TARGET_PAGE_SIZE);