From patchwork Mon Mar 21 17:36:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Pai G, Sunil" X-Patchwork-Id: 1607871 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=AlDzr1rM; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KMhd93F2xz9s0B for ; Tue, 22 Mar 2022 04:37:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B91FB841AA; Mon, 21 Mar 2022 17:36:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nT85K6iUHcVC; Mon, 21 Mar 2022 17:36:57 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id BBF208419D; Mon, 21 Mar 2022 17:36:56 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 86AB4C0033; Mon, 21 Mar 2022 17:36:56 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id A40AFC000B for ; Mon, 21 Mar 2022 17:36:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 9184E60F43 for ; Mon, 21 Mar 2022 17:36:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XlWY1uELIgyI for ; Mon, 21 Mar 2022 17:36:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by smtp3.osuosl.org (Postfix) with ESMTPS id A6FEA60F35 for ; Mon, 21 Mar 2022 17:36:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1647884213; x=1679420213; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e+xtXrpRzqNpHGkM/o3OhIzeVaiIpXUreCZy8cwsKW0=; b=AlDzr1rMu7NL2HXSvKJDxTLJhji4MbJkNX2YwwVXy6t3IlBuA4xNo7us tjdlvMG1mnxs2FKVrAhT6EVsuzXhVLioi9tuUxhSuPFsVXEqDoq3wgEyw 89eYaBfki1w70uJuTGfChuogK9P2EbisgJS2BeO7Ogl6Cnl6l6LMZEMgp t/TKi8j0exsJAf9BrpjaxeJhxbTNzRg08lUu52ad+6xI6fQBymyvnqtFR blir+PXeaLt556hvtVDQaiE0VISrO/HQvMwHEafIiVAptom/dG+gTlkQV +IUO81kcaZKvMldKjxs0TefOIT0cC8XrtfUbH3I7enZWy7AxfoAJVpfTd g==; X-IronPort-AV: E=McAfee;i="6200,9189,10293"; a="245084686" X-IronPort-AV: E=Sophos;i="5.90,199,1643702400"; d="scan'208";a="245084686" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Mar 2022 10:36:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,199,1643702400"; d="scan'208";a="716585794" Received: from silpixa00400896.ir.intel.com ([10.243.22.124]) by orsmga005.jf.intel.com with ESMTP; 21 Mar 2022 10:36:50 -0700 From: Sunil Pai G To: dev@openvswitch.org Date: Mon, 21 Mar 2022 17:36:39 +0000 Message-Id: <20220321173640.326795-1-sunil.pai.g@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220104125242.1064162-1-sunil.pai.g@intel.com> References: <20220104125242.1064162-1-sunil.pai.g@intel.com> MIME-Version: 1.0 Cc: john.mcnamara@intel.com, i.maximets@ovn.org, maxime.coquelin@redhat.com, Jiayu.Hu@intel.com Subject: [ovs-dev] [PATCH RFC dpdk-latest v4 0/1] netdev-dpdk: Enable vhost async API's. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This series brings in the new asynchronous vHost API's in DPDK to OVS. With the asynchronous framework, vHost-user can offload the memory copy operation to DPDK DMA-dev based enabled devices without blocking the CPU. The intention of this patch is solely to communicate the design of the feature, evaluate the code impact and to get early feedback from the community. Usage: To enable DMA offload for vhost path, one can issue the following command: ovs-vsctl --no-wait set Open_vSwitch . other_config:vhost-async-support=true Issuing this requires restarting the vswitchd daemon. Please note that, its required to have DMA devices visible to DPDK before OVS launch either by binding them to userspace driver vfio-pci or through other facilities as mentioned in the driver documentation. The current assignment model is 1 DMA device per data plane thread. If no DMA devices are found for that thread, it will fall back to perform CPU copy. To check if async support is enabled, one can check for the following message in the ovs-vswitchd.log dpdk|INFO|Async support enabled for vhost-user-client. To verify if DMA devices are assigned to the data plane thread, one can check the ovs-vswitchd.log for messages like: netdev_dpdk(pmd-c01/id:7)|INFO|DMA device with dev id: 0 assigned to pmd for vhost async copy offload. This patch was tested on : OVS : branch : dpdk-latest with DPDK : branch : main, commit id : 80c5b4d635 , version: 22.03 + https://patches.dpdk.org/project/dpdk/list/?series=22093 : v3 RFC of : vhost: support async dequeue data path + https://patches.dpdk.org/project/dpdk/list/?series=22119 : v2 RFC of : vhost: support clear in-flight packets for async dequeue. + https://patches.dpdk.org/project/dpdk/patch/20220216070417.9597-2-xuan.ding@intel.com/ : v1 RFC of : vhost: add unsafe API to check inflight packets Please note: - All the DPDK API's (both vhost async and DMADEV) utilized in the patch are experimental. - Its required to have atleast one vhost Rxq on every single core since currently even the vhost Tx packets are polled for completions from vhost Rx context. Future work: - Compatibility with PMD auto load balancing and solve requirement of vhost-rxq on every PMD-thread V3-> V4: ----------- - Based on community feedback, use lockless ring(rte_ring) to avoid vhost tx contention among data plane threads. - Drain packets from Rxq before the pmd reload. - Clear packets from a Rxq/txq on queue disable. - ovs-appctl netdev-dpdk/get-dmadev-info : to get the DMA stats and info. V2-> V3: ---------- - Update code according to new architecture with dmadev datapath in vhost library. ○ The dmadev datapath is now moved to the vhost library and OVS is now only responsible to configure/use the DMA device. ○ This allows OVS the flexibility to decide on the allocation scheme of DMA devices. Sunil Pai G (1): netdev-dpdk: Enable vhost async API's in OvS. lib/dpdk-stub.c | 6 + lib/dpdk.c | 33 ++ lib/dpdk.h | 1 + lib/dpif-netdev.c | 7 + lib/netdev-dpdk.c | 795 ++++++++++++++++++++++++++++++++++++++++++++-- lib/netdev-dpdk.h | 14 + 6 files changed, 829 insertions(+), 27 deletions(-)