From patchwork Wed Feb 3 21:21:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: William Tu X-Patchwork-Id: 1435626 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=LcsYiiIc; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DWF4r6TjFz9sVb for ; Thu, 4 Feb 2021 08:22:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DE6E7862A5; Wed, 3 Feb 2021 21:22:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6hDuvnVMs0ov; Wed, 3 Feb 2021 21:22:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id E918B86241; Wed, 3 Feb 2021 21:22:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CC8D6C0FA8; Wed, 3 Feb 2021 21:22:14 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id E3993C0174 for ; Wed, 3 Feb 2021 21:22:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CA6FA86241 for ; Wed, 3 Feb 2021 21:22:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Pxf-wLfqp57 for ; Wed, 3 Feb 2021 21:22:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 09EF386153 for ; Wed, 3 Feb 2021 21:22:13 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id cl8so444064pjb.0 for ; Wed, 03 Feb 2021 13:22:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=llhtcn3cLqPo7sDLIpw40p59l/Nl3MYTxd3MZMwxpYA=; b=LcsYiiIcHOVq7vxRY9ffPlF7BD3at1N171kvuxhrUYwZaMjZ8mwND91cJnpZaoZ4jK rACV+VhvM+dBn+SIdQIIanGDGg93uO94e+P2vwuAnsikDHD20FjZnR28XFJasRM3eaIV zDnpxpX7RB7Aqh6qMEObwQSWG9LOsA1Z6pxRCMv1b8iOj5GkGwVyBC0ZltVE6+yAAC7E MCXBdieAdFTeQteAfzwcsmqy+9z+GJN8WU99bJ2c9DkXIWfE7oZOAAF/qFjxapjnQg6N PbY+kMi+7vEyFnwqyxQXome4v/kXyKMxjnhEcMBq93W+LVOEzdfj9a66mYzg5n/zfBxS 1LSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=llhtcn3cLqPo7sDLIpw40p59l/Nl3MYTxd3MZMwxpYA=; b=jiSLzn3DK6ji6seDtW+d6bfxX89Q4iB+xeaIYMFczPSrUdsZWbR9SG5cv3l06Wf897 fxSfdmJiIXKl/ANNznfwSlBNj2XqrVeF7yMOeJovo9fqSqdCQWsVE/TXFRO0EFshxG8R 9DfVuUKfkPO7oHysK3/3GxCLBdQ6Ez6IH58oyV7WdCE3rBttbOIwICSlmghD8cbGRGPi ZanImoVTxumWtImUSD44hp/DE7TlRli+VMO2LQJJKljsRfp4/mrt5iTZbo+IO1B6YbG7 G3cDoDHmjKdx7o719zQcIT1U3g3wRapbOUijpCOesQpjxdHByr8CyQ35NqLX74cg4NLr Gg3A== X-Gm-Message-State: AOAM531OhXYu1M2WJXgMucVP/7MG4b9Z1jEEF21Zxnmo9Kj7MFSJ730x BsEXRWaV6dQaHhSazjSGmOI+TpcoeQ0= X-Google-Smtp-Source: ABdhPJwfyHZmCGaTo2KVgve7nMPlWlSI+dmHsKuMDKffjhPGLx6LVVkzXidtXxrwN1IPDSnHNx/Jyw== X-Received: by 2002:a17:902:b190:b029:df:fff2:c345 with SMTP id s16-20020a170902b190b02900dffff2c345mr4699462plr.17.1612387331643; Wed, 03 Feb 2021 13:22:11 -0800 (PST) Received: from sc9-mailhost1.vmware.com (c-67-185-44-206.hsd1.wa.comcast.net. [67.185.44.206]) by smtp.gmail.com with ESMTPSA id w12sm2803144pjq.26.2021.02.03.13.22.09 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Feb 2021 13:22:10 -0800 (PST) From: William Tu To: dev@openvswitch.org Date: Wed, 3 Feb 2021 13:21:44 -0800 Message-Id: <1612387304-68681-1-git-send-email-u9012063@gmail.com> X-Mailer: git-send-email 2.7.4 Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH] netdev-afxdp: Add start qid support. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Mellanox card has different XSK design. It requires users to create dedicated queues for XSK. Unlike Intel's NIC which loads XDP program to all queues, Mellanox only loads XDP program to a subset of its queue. When OVS uses AF_XDP with mlx5, it doesn't replace the existing RX and TX queues in the channel with XSK RX and XSK TX queues, but it creates an additional pair of queues for XSK in that channel. To distinguish regular and XSK queues, mlx5 uses a different range of qids. That means, if the card has 24 queues, queues 0..11 correspond to regular queues, and queues 12..23 are XSK queues. In this case, we should attach the netdev-afxdp with 'start-qid=12'. I tested using Mellanox Connect-X 6Dx, by setting 'start-qid=1', and: $ ethtool -L enp2s0f0np0 combined 1 # queue 0 is for non-XDP traffic, queue 1 is for XSK $ ethtool -N enp2s0f0np0 flow-type udp4 action 1 note: we need additionally add flow-redirect rule to queue 1 Tested-at: https://github.com/williamtu/ovs-travis/actions/runs/535141041 Signed-off-by: William Tu --- Documentation/intro/install/afxdp.rst | 2 ++ lib/netdev-afxdp.c | 23 ++++++++++++++++++----- lib/netdev-linux-private.h | 1 + vswitchd/vswitch.xml | 8 ++++++++ 4 files changed, 29 insertions(+), 5 deletions(-) diff --git a/Documentation/intro/install/afxdp.rst b/Documentation/intro/install/afxdp.rst index f2643e0d41a1..eac298c52575 100644 --- a/Documentation/intro/install/afxdp.rst +++ b/Documentation/intro/install/afxdp.rst @@ -204,6 +204,8 @@ more details): * ``use-need-wakeup``: default ``true`` if libbpf supports it, otherwise ``false``. +* ``start-qid``: default ``0``. + For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device, configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and ``n_rxq``:: diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index 482400d8d135..36f6a323b1bc 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -458,12 +458,13 @@ xsk_configure_queue(struct netdev_linux *dev, int ifindex, int queue_id, VLOG_DBG("%s: configuring queue: %d, mode: %s, use-need-wakeup: %s.", netdev_get_name(&dev->up), queue_id, xdp_modes[mode].name, dev->use_need_wakeup ? "true" : "false"); - xsk_info = xsk_configure(ifindex, queue_id, mode, dev->use_need_wakeup, - report_socket_failures); + xsk_info = xsk_configure(ifindex, dev->startqid + queue_id, mode, + dev->use_need_wakeup, report_socket_failures); if (!xsk_info) { VLOG(report_socket_failures ? VLL_ERR : VLL_DBG, - "%s: Failed to create AF_XDP socket on queue %d in %s mode.", - netdev_get_name(&dev->up), queue_id, xdp_modes[mode].name); + "%s: Failed to create AF_XDP socket on queue %d+%d in %s mode.", + netdev_get_name(&dev->up), dev->startqid, queue_id, + xdp_modes[mode].name); dev->xsks[queue_id] = NULL; return -1; } @@ -604,6 +605,7 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, enum afxdp_mode xdp_mode; bool need_wakeup; int new_n_rxq; + int new_startqid; ovs_mutex_lock(&dev->mutex); new_n_rxq = MAX(smap_get_int(args, "n_rxq", NR_QUEUE), 1); @@ -637,12 +639,18 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, } #endif + /* TODO: need to check + * new_startqid + new_n_rxq > total dev's queues. */ + new_startqid = smap_get_int(args, "start-qid", 0); + if (dev->requested_n_rxq != new_n_rxq || dev->requested_xdp_mode != xdp_mode - || dev->requested_need_wakeup != need_wakeup) { + || dev->requested_need_wakeup != need_wakeup + || dev->requested_startqid != new_startqid) { dev->requested_n_rxq = new_n_rxq; dev->requested_xdp_mode = xdp_mode; dev->requested_need_wakeup = need_wakeup; + dev->requested_startqid = new_startqid; netdev_request_reconfigure(netdev); } ovs_mutex_unlock(&dev->mutex); @@ -661,6 +669,7 @@ netdev_afxdp_get_config(const struct netdev *netdev, struct smap *args) xdp_modes[dev->xdp_mode_in_use].name); smap_add_format(args, "use-need-wakeup", "%s", dev->use_need_wakeup ? "true" : "false"); + smap_add_format(args, "start-qid", "%d", dev->startqid); ovs_mutex_unlock(&dev->mutex); return 0; } @@ -696,6 +705,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) if (netdev->n_rxq == dev->requested_n_rxq && dev->xdp_mode == dev->requested_xdp_mode && dev->use_need_wakeup == dev->requested_need_wakeup + && dev->startqid == dev->requested_startqid && dev->xsks) { goto out; } @@ -713,6 +723,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) VLOG_ERR("setrlimit(RLIMIT_MEMLOCK) failed: %s", ovs_strerror(errno)); } dev->use_need_wakeup = dev->requested_need_wakeup; + dev->startqid = dev->requested_startqid; err = xsk_configure_all(netdev); if (err) { @@ -1177,12 +1188,14 @@ netdev_afxdp_construct(struct netdev *netdev) /* Queues should not be used before the first reconfiguration. Clearing. */ netdev->n_rxq = 0; netdev->n_txq = 0; + dev->startqid = 0; dev->xdp_mode = OVS_AF_XDP_MODE_UNSPEC; dev->xdp_mode_in_use = OVS_AF_XDP_MODE_UNSPEC; dev->requested_n_rxq = NR_QUEUE; dev->requested_xdp_mode = OVS_AF_XDP_MODE_BEST_EFFORT; dev->requested_need_wakeup = NEED_WAKEUP_DEFAULT; + dev->requested_startqid = 0; dev->xsks = NULL; dev->tx_locks = NULL; diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index c7c515f70700..242ce1659614 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -109,6 +109,7 @@ struct netdev_linux { /* AF_XDP information. */ struct xsk_socket_info **xsks; int requested_n_rxq; + int startqid, requested_startqid; enum afxdp_mode xdp_mode; /* Configured AF_XDP mode. */ enum afxdp_mode requested_xdp_mode; /* Requested AF_XDP mode. */ diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index a2ad84edefa9..47e14ba67c3a 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -3290,6 +3290,14 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \

+ +

+ Specifies the starting XDP socket's queue id. + Defaults to 0. +

+ +