From patchwork Tue Sep 7 08:23:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1525157 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UNxv66ID; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4H3db93H3lz9sW8 for ; Tue, 7 Sep 2021 18:24:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B78F740490; Tue, 7 Sep 2021 08:24:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pj4aTEF5xz9k; Tue, 7 Sep 2021 08:24:00 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id A8B39402EA; Tue, 7 Sep 2021 08:23:59 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 74B8BC0011; Tue, 7 Sep 2021 08:23:59 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6E974C000D for ; Tue, 7 Sep 2021 08:23:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4B48D608F7 for ; Tue, 7 Sep 2021 08:23:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vXD7BeFR_nJ0 for ; Tue, 7 Sep 2021 08:23:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 336FC606F5 for ; Tue, 7 Sep 2021 08:23:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631003035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=o23RI8p9dEQNR6LMVG+LD0oPANmh+ccLiVKxeslrvl4=; b=UNxv66IDnK4+E1v/85OTJ3ebZGjL05qQRydKPZfGiJcmn2mQ8EN08XeeHpYFe4cEQUHTtH ZNPsRZu9rVugh9aUcz1vzbD0DlSYQNLikWKUprIQNWQdTcMfYs/mw0AgRanwVzXqD9xuxM GIZd6DIhezL00j5Xsb4+tWg74cGPnjQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-229-6FE0qKycOdqL7ATPwN5rsA-1; Tue, 07 Sep 2021 04:23:52 -0400 X-MC-Unique: 6FE0qKycOdqL7ATPwN5rsA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5B84B80196C; Tue, 7 Sep 2021 08:23:51 +0000 (UTC) Received: from dmarchan.remote.csb (unknown [10.40.192.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76F276608B; Tue, 7 Sep 2021 08:23:49 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Tue, 7 Sep 2021 10:23:43 +0200 Message-Id: <20210907082343.16370-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david.marchand@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: i.maximets@ovn.org Subject: [ovs-dev] [PATCH] netdev-dpdk: Remove access to DPDK internals. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Instead of dereferencing DPDK ethdev internals, we can list "sibling" ports [1]: such ports share the underlying rte_device object. This API was experimental so far, but it will go to stable in 21.11 (see [2]) as it underwent no change since introduction in v19.05. For the time being, waive the warning for this experimental API. 1: https://git.dpdk.org/dpdk/commit/?id=7f98942886bc 2: https://patchwork.dpdk.org/project/dpdk/list/?series=18705 Signed-off-by: David Marchand Reviewed-by: Maxime Coquelin --- Note: I sent this against the master branch for discussion, but I am fine with re-sending this against dpdk-latest without the #pragma trick. --- lib/netdev-dpdk.c | 95 ++++++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 42 deletions(-) diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index ca92c947a2..2347de7f5b 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -1279,26 +1279,6 @@ common_construct(struct netdev *netdev, dpdk_port_t port_no, return 0; } -/* Get the number of OVS interfaces which have the same DPDK - * rte device (e.g. same pci bus address). - * FIXME: avoid direct access to DPDK internal array rte_eth_devices. - */ -static int -netdev_dpdk_get_num_ports(struct rte_device *device) - OVS_REQUIRES(dpdk_mutex) -{ - struct netdev_dpdk *dev; - int count = 0; - - LIST_FOR_EACH (dev, list_node, &dpdk_list) { - if (rte_eth_devices[dev->port_id].device == device - && rte_eth_devices[dev->port_id].state != RTE_ETH_DEV_UNUSED) { - count++; - } - } - return count; -} - static int vhost_common_construct(struct netdev *netdev) OVS_REQUIRES(dpdk_mutex) @@ -1452,8 +1432,6 @@ static void netdev_dpdk_destruct(struct netdev *netdev) { struct netdev_dpdk *dev = netdev_dpdk_cast(netdev); - struct rte_device *rte_dev; - struct rte_eth_dev *eth_dev; ovs_mutex_lock(&dpdk_mutex); @@ -1461,20 +1439,46 @@ netdev_dpdk_destruct(struct netdev *netdev) dev->started = false; if (dev->attached) { - /* Retrieve eth device data before closing it. - * FIXME: avoid direct access to DPDK internal array rte_eth_devices. - */ - eth_dev = &rte_eth_devices[dev->port_id]; - rte_dev = eth_dev->device; + bool dpdk_resources_still_used = false; + struct rte_eth_dev_info dev_info; + dpdk_port_t sibling_port_id; + + /* Check if this netdev has siblings (i.e. shares DPDK resources) among + * other OVS netdevs. */ +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + RTE_ETH_FOREACH_DEV_SIBLING (sibling_port_id, dev->port_id) { +#pragma GCC diagnostic pop + struct netdev_dpdk *sibling; + + /* RTE_ETH_FOREACH_DEV_SIBLING lists dev->port_id as part of the + * loop. */ + if (sibling_port_id == dev->port_id) { + continue; + } + LIST_FOR_EACH (sibling, list_node, &dpdk_list) { + if (sibling->port_id != sibling_port_id) { + continue; + } + dpdk_resources_still_used = true; + break; + } + if (dpdk_resources_still_used) { + break; + } + } + + /* Retrieve eth device data before closing it. */ + rte_eth_dev_info_get(dev->port_id, &dev_info); /* Remove the eth device. */ rte_eth_dev_close(dev->port_id); - /* Remove this rte device and all its eth devices if all the eth - * devices belonging to the rte device are closed. - */ - if (!netdev_dpdk_get_num_ports(rte_dev)) { - int ret = rte_dev_remove(rte_dev); + /* Remove the rte device if no associated eth device is used by OVS. + * Note: any remaining eth devices associated to this rte device are + * closed by DPDK ethdev layer. */ + if (!dpdk_resources_still_used) { + int ret = rte_dev_remove(dev_info.device); if (ret < 0) { VLOG_ERR("Device '%s' can not be detached: %s.", @@ -3793,12 +3797,12 @@ static void netdev_dpdk_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, const char *argv[], void *aux OVS_UNUSED) { - char *response; - dpdk_port_t port_id; - struct netdev_dpdk *dev; - struct rte_device *rte_dev; struct ds used_interfaces = DS_EMPTY_INITIALIZER; + struct rte_eth_dev_info dev_info; + dpdk_port_t sibling_port_id; + dpdk_port_t port_id; bool used = false; + char *response; ovs_mutex_lock(&dpdk_mutex); @@ -3808,18 +3812,24 @@ netdev_dpdk_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, goto error; } - rte_dev = rte_eth_devices[port_id].device; ds_put_format(&used_interfaces, "Device '%s' is being used by the following interfaces:", argv[1]); - LIST_FOR_EACH (dev, list_node, &dpdk_list) { - /* FIXME: avoid direct access to DPDK array rte_eth_devices. */ - if (rte_eth_devices[dev->port_id].device == rte_dev - && rte_eth_devices[dev->port_id].state != RTE_ETH_DEV_UNUSED) { +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wdeprecated-declarations" + RTE_ETH_FOREACH_DEV_SIBLING (sibling_port_id, port_id) { +#pragma GCC diagnostic pop + struct netdev_dpdk *dev; + + LIST_FOR_EACH (dev, list_node, &dpdk_list) { + if (dev->port_id != sibling_port_id) { + continue; + } used = true; ds_put_format(&used_interfaces, " %s", netdev_get_name(&dev->up)); + break; } } @@ -3831,8 +3841,9 @@ netdev_dpdk_detach(struct unixctl_conn *conn, int argc OVS_UNUSED, } ds_destroy(&used_interfaces); + rte_eth_dev_info_get(port_id, &dev_info); rte_eth_dev_close(port_id); - if (rte_dev_remove(rte_dev) < 0) { + if (rte_dev_remove(dev_info.device) < 0) { response = xasprintf("Device '%s' can not be detached", argv[1]); goto error; }