From patchwork Wed Dec 14 13:24:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wan Junjie X-Patchwork-Id: 1715756 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=bytedance-com.20210112.gappssmtp.com header.i=@bytedance-com.20210112.gappssmtp.com header.a=rsa-sha256 header.s=20210112 header.b=x+Q6p/S9; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NXGLQ17ppz23yh for ; Thu, 15 Dec 2022 00:24:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2ED1C60B23; Wed, 14 Dec 2022 13:24:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 2ED1C60B23 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=bytedance-com.20210112.gappssmtp.com header.i=@bytedance-com.20210112.gappssmtp.com header.a=rsa-sha256 header.s=20210112 header.b=x+Q6p/S9 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Y8B8452osVE; Wed, 14 Dec 2022 13:24:43 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 36FB060595; Wed, 14 Dec 2022 13:24:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 36FB060595 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0A014C0032; Wed, 14 Dec 2022 13:24:42 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 298D8C002D for ; Wed, 14 Dec 2022 13:24:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id F23F840B8D for ; Wed, 14 Dec 2022 13:24:40 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org F23F840B8D Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=bytedance-com.20210112.gappssmtp.com header.i=@bytedance-com.20210112.gappssmtp.com header.a=rsa-sha256 header.s=20210112 header.b=x+Q6p/S9 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7y5FGV7W0__U for ; Wed, 14 Dec 2022 13:24:39 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org DC9A2409DE Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by smtp2.osuosl.org (Postfix) with ESMTPS id DC9A2409DE for ; Wed, 14 Dec 2022 13:24:39 +0000 (UTC) Received: by mail-pl1-x635.google.com with SMTP id s7so3311686plk.5 for ; Wed, 14 Dec 2022 05:24:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=DoNhF2Bf+U0DFX7sOiotLiczbKSw2dX1hxStq0NuQxw=; b=x+Q6p/S9HAKMi9SLxsI6sd4f2MW6K910j0a8kr/GhTm6j2HCBSTwvHNZH7hWlktRGw aX5oxV0BzSVX8YEgoKUmWYO3rxlCNBzXQ7kuJGSJplotalDLG4CO8rMV7RkEiLUHUPYw Zy0FN/eMYtD+R5R0S0i5QxUp46Z3Pd+L7dWMeb45bLDAveBH7Iu66I+pVFvHRo1GI6dk dn4EHzkEH+k8HKVyAWZvaEBn/z3KJtkJZfQwpX//Rw6U0xcyVsDYKzBNy23CXqLl3ww4 X0uCFnqRGZqTYXTTTmFJiD1hfzrQW2O+2x2V5oP//xYulH5qYwcOOG+zQjihHcj3bSuy nlVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DoNhF2Bf+U0DFX7sOiotLiczbKSw2dX1hxStq0NuQxw=; b=uav/XwDJdGiYhTHkVVhXgkiJHbCCszkPg/TqjKl7AschCkR9Q+8Ldptpmicu7BwHHl J5+KPflfmUbO17CsJ9ypXibNwXUinLma4VGXzBQBXepG/w1pxmOeLMk5+pJ6OsD6zBP3 gyG30AkhiI17XJE0Z8KsSXqlcNzvwuNvn+fpj74U/2+bBYsWzDaV2sqgKwK3s1BgktYp hgwDIvksAHWb2lf1Oq3TTWXYDuEVEMZwAnWD0hrg0czk5Ldg3wq1s4JhE4HbVS+hdnhJ G5mN/BPAKWOyA4goNIZMB9DN23YoEjmCLXTQgrI5oF5mt+qMvKUf9q7Uu2YKI1GE2Lmt wlMQ== X-Gm-Message-State: ANoB5pnqqycqbEv26P3qHKisOiuU6r+t/FUkJA3HUcn/tLxn5LXwe+6f 6WAOtVU5+Q2uM94+HkMju+HUXkTuDSKr0i/T X-Google-Smtp-Source: AA0mqf5Gt/P6MInRxx+LMgsXGMmlLgfKmy/gXSUkbaAB1HHbSKvSlmznf9bgRPjg+VWrarpnssCbSg== X-Received: by 2002:a17:902:aa95:b0:189:14f6:2b8c with SMTP id d21-20020a170902aa9500b0018914f62b8cmr23193794plr.64.1671024278819; Wed, 14 Dec 2022 05:24:38 -0800 (PST) Received: from C02Z10J4LVCK.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id a15-20020a1709027e4f00b00187197c4999sm1850704pln.167.2022.12.14.05.24.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Dec 2022 05:24:38 -0800 (PST) From: Wan Junjie To: dev@openvswitch.org Date: Wed, 14 Dec 2022 21:24:34 +0800 Message-Id: <20221214132434.70210-1-wanjunjie@bytedance.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20221214130114.55912-1-wanjunjie@bytedance.com> References: <20221214130114.55912-1-wanjunjie@bytedance.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v2] netdev-afxdp: Allow using user's prog instead of libxdp default prog X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Libxdp will load an embeded program by default. And OVS will remove the program when it destroy the xsks. This is not a good experience if we pre-load an xdp program. Every time we restart OVS, it will remove the program. Add an option 'inhibit', so we can disable the default program load and remove. Use command for a afxdp type interface like below: ovs-vsctl set int xxx options:inhibit=true --- v2: fix some code missing Signed-off-by: Wan Junjie --- lib/netdev-afxdp.c | 45 +++++++++++++++++++++++++++----------- lib/netdev-linux-private.h | 3 +++ 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/lib/netdev-afxdp.c b/lib/netdev-afxdp.c index ca3f2431e..e43353cc0 100644 --- a/lib/netdev-afxdp.c +++ b/lib/netdev-afxdp.c @@ -95,7 +95,8 @@ BUILD_ASSERT_DECL(PROD_NUM_DESCS == CONS_NUM_DESCS); static struct xsk_socket_info *xsk_configure(int ifindex, int xdp_queue_id, enum afxdp_mode mode, bool use_need_wakeup, - bool report_socket_failures); + bool report_socket_failures, + bool inhibit); static void xsk_remove_xdp_program(uint32_t ifindex, enum afxdp_mode); static void xsk_destroy(struct xsk_socket_info *xsk); static int xsk_configure_all(struct netdev *netdev); @@ -334,7 +335,8 @@ xsk_configure_umem(void *buffer, uint64_t size) static struct xsk_socket_info * xsk_configure_socket(struct xsk_umem_info *umem, uint32_t ifindex, uint32_t queue_id, enum afxdp_mode mode, - bool use_need_wakeup, bool report_socket_failures) + bool use_need_wakeup, bool report_socket_failures, + bool inhibit) { struct xsk_socket_config cfg; struct xsk_socket_info *xsk; @@ -348,6 +350,9 @@ xsk_configure_socket(struct xsk_umem_info *umem, uint32_t ifindex, cfg.rx_size = CONS_NUM_DESCS; cfg.tx_size = PROD_NUM_DESCS; cfg.libbpf_flags = 0; + if (inhibit) { + cfg.libbpf_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD; + } cfg.bind_flags = xdp_modes[mode].bind_flags; cfg.xdp_flags = xdp_modes[mode].xdp_flags | XDP_FLAGS_UPDATE_IF_NOEXIST; @@ -413,7 +418,7 @@ xsk_configure_socket(struct xsk_umem_info *umem, uint32_t ifindex, static struct xsk_socket_info * xsk_configure(int ifindex, int xdp_queue_id, enum afxdp_mode mode, - bool use_need_wakeup, bool report_socket_failures) + bool use_need_wakeup, bool report_socket_failures, bool inhibit) { struct xsk_socket_info *xsk; struct xsk_umem_info *umem; @@ -435,7 +440,8 @@ xsk_configure(int ifindex, int xdp_queue_id, enum afxdp_mode mode, VLOG_DBG("Allocated umem pool at 0x%"PRIxPTR, (uintptr_t) umem); xsk = xsk_configure_socket(umem, ifindex, xdp_queue_id, mode, - use_need_wakeup, report_socket_failures); + use_need_wakeup, report_socket_failures, + inhibit); if (!xsk) { /* Clean up umem and xpacket pool. */ if (xsk_umem__delete(umem->umem)) { @@ -459,7 +465,7 @@ xsk_configure_queue(struct netdev_linux *dev, int ifindex, int queue_id, netdev_get_name(&dev->up), queue_id, xdp_modes[mode].name, dev->use_need_wakeup ? "true" : "false"); xsk_info = xsk_configure(ifindex, queue_id, mode, dev->use_need_wakeup, - report_socket_failures); + report_socket_failures, dev->inhibit); if (!xsk_info) { VLOG(report_socket_failures ? VLL_ERR : VLL_DBG, "%s: Failed to create AF_XDP socket on queue %d in %s mode.", @@ -582,9 +588,11 @@ xsk_destroy_all(struct netdev *netdev) dev->xsks = NULL; } - VLOG_INFO("%s: Removing xdp program.", netdev_get_name(netdev)); - ifindex = linux_get_ifindex(netdev_get_name(netdev)); - xsk_remove_xdp_program(ifindex, dev->xdp_mode_in_use); + if (!dev->inhibit) { + VLOG_INFO("%s: Removing xdp program.", netdev_get_name(netdev)); + ifindex = linux_get_ifindex(netdev_get_name(netdev)); + xsk_remove_xdp_program(ifindex, dev->xdp_mode_in_use); + } if (dev->tx_locks) { for (i = 0; i < netdev_n_txq(netdev); i++) { @@ -602,7 +610,7 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, struct netdev_linux *dev = netdev_linux_cast(netdev); const char *str_xdp_mode; enum afxdp_mode xdp_mode; - bool need_wakeup; + bool need_wakeup, need_inhibit; int new_n_rxq; ovs_mutex_lock(&dev->mutex); @@ -636,13 +644,16 @@ netdev_afxdp_set_config(struct netdev *netdev, const struct smap *args, need_wakeup = false; } #endif + need_inhibit = smap_get_bool(args, "inhibit", false); if (dev->requested_n_rxq != new_n_rxq || dev->requested_xdp_mode != xdp_mode - || dev->requested_need_wakeup != need_wakeup) { + || dev->requested_need_wakeup != need_wakeup + || dev->requested_inhibit != need_inhibit) { dev->requested_n_rxq = new_n_rxq; dev->requested_xdp_mode = xdp_mode; dev->requested_need_wakeup = need_wakeup; + dev->requested_inhibit = need_inhibit; netdev_request_reconfigure(netdev); } ovs_mutex_unlock(&dev->mutex); @@ -661,6 +672,8 @@ netdev_afxdp_get_config(const struct netdev *netdev, struct smap *args) xdp_modes[dev->xdp_mode_in_use].name); smap_add_format(args, "use-need-wakeup", "%s", dev->use_need_wakeup ? "true" : "false"); + smap_add_format(args, "inhibit", "%s", + dev->inhibit ? "true" : "false"); ovs_mutex_unlock(&dev->mutex); return 0; } @@ -696,6 +709,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) if (netdev->n_rxq == dev->requested_n_rxq && dev->xdp_mode == dev->requested_xdp_mode && dev->use_need_wakeup == dev->requested_need_wakeup + && dev->inhibit == dev->requested_inhibit && dev->xsks) { goto out; } @@ -713,6 +727,7 @@ netdev_afxdp_reconfigure(struct netdev *netdev) VLOG_ERR("setrlimit(RLIMIT_MEMLOCK) failed: %s", ovs_strerror(errno)); } dev->use_need_wakeup = dev->requested_need_wakeup; + dev->inhibit = dev->requested_inhibit; err = xsk_configure_all(netdev); if (err) { @@ -762,10 +777,12 @@ signal_remove_xdp(struct netdev *netdev) struct netdev_linux *dev = netdev_linux_cast(netdev); int ifindex; - ifindex = linux_get_ifindex(netdev_get_name(netdev)); + if (!dev->inhibit) { + ifindex = linux_get_ifindex(netdev_get_name(netdev)); - VLOG_WARN("Force removing xdp program."); - xsk_remove_xdp_program(ifindex, dev->xdp_mode_in_use); + VLOG_WARN("Force removing xdp program."); + xsk_remove_xdp_program(ifindex, dev->xdp_mode_in_use); + } } static struct dp_packet_afxdp * @@ -1183,6 +1200,8 @@ netdev_afxdp_construct(struct netdev *netdev) dev->requested_n_rxq = NR_QUEUE; dev->requested_xdp_mode = OVS_AF_XDP_MODE_BEST_EFFORT; dev->requested_need_wakeup = NEED_WAKEUP_DEFAULT; + dev->requested_inhibit = false; + dev->inhibit = false; dev->xsks = NULL; dev->tx_locks = NULL; diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index deb015bdb..6939b465c 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -119,6 +119,9 @@ struct netdev_linux { bool use_need_wakeup; bool requested_need_wakeup; + bool inhibit; + bool requested_inhibit; + struct netdev_afxdp_tx_lock *tx_locks; /* Array of locks for TX queues. */ #endif };