From patchwork Thu Jan 18 16:15:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Marchand X-Patchwork-Id: 1888140 X-Patchwork-Delegate: i.maximets@samsung.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=iGczOqhQ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TG7Bc5LNVz1yPV for ; Fri, 19 Jan 2024 03:15:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 667C743807; Thu, 18 Jan 2024 16:15:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 667C743807 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=iGczOqhQ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRLnJQxZkh6e; Thu, 18 Jan 2024 16:15:17 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id E5F574054E; Thu, 18 Jan 2024 16:15:15 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org E5F574054E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B992BC0072; Thu, 18 Jan 2024 16:15:15 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6884AC0037 for ; Thu, 18 Jan 2024 16:15:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 447586F5AD for ; Thu, 18 Jan 2024 16:15:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 447586F5AD Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=iGczOqhQ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rjufB2YqiJ5P for ; Thu, 18 Jan 2024 16:15:13 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id C42356F64A for ; Thu, 18 Jan 2024 16:15:12 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org C42356F64A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1705594511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=TL4Vir7ohdj8pI4UPMd97RMzE61o6ZXJuMI1Mx/JMcY=; b=iGczOqhQcGtBkSEXeASPAFEe+25qvf/waJ8447NgdJ4sLDS7byXltkDsmAacn2AJ8BOhgD GFuxnM/pDj/tR4akbG6ppvMSky8Q8KPt3avTj8d/BjT3x3IkAQDHgnCMNlfIP/s3u0Co8x RsH6VAOtscBKSD64dWRHnClV1cDMS24= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-x07ORQDeOd-0rJ1VQfW-eQ-1; Thu, 18 Jan 2024 11:15:05 -0500 X-MC-Unique: x07ORQDeOd-0rJ1VQfW-eQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 088D7848D67; Thu, 18 Jan 2024 16:15:05 +0000 (UTC) Received: from dmarchan.redhat.com (unknown [10.45.224.95]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1BF61111F6; Thu, 18 Jan 2024 16:15:03 +0000 (UTC) From: David Marchand To: dev@openvswitch.org Date: Thu, 18 Jan 2024 17:15:00 +0100 Message-ID: <20240118161500.4126409-1-david.marchand@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: Ilya Maximets Subject: [ovs-dev] [PATCH v3] netdev-dpdk: Trigger port reconfiguration in main thread for resets. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" When OVS (main thread) configures a DPDK netdev, it holds a netdev_dpdk mutex lock. As part of this configure operation, the net/iavf driver (used with i40e VF devices) triggers a queue count change. The PF entity (serviced by a kernel PF driver for example) handles this change and requests back that the VF driver resets the VF device. The driver then completes the VF reset operation on its side and waits for completion of the iavf-event thread responsible for handling various VF device events. On the other hand, handling of the VF reset request in this iavf-event thread results in notifying the application with a port reset request (RTE_ETH_EVENT_INTR_RESET). The OVS reset callback tries to take a hold of the same netdev_dpdk mutex and blocks the iavf-event thread. As a resut, the net/iavf driver (still running on OVS main thread) is unable to complete as it is waiting for iavf-event to complete. To break from this situation, the OVS reset callback now won't take a netdev_dpdk mutex. Instead, the port reset request is stored in a simple RTE_ETH_MAXPORTS array associated to a seq object. This is enough to let the VF driver complete this port initialisation. The OVS main thread later handles the port reset request. More details in the DPDK upstream bz as this issue appeared following a change in DPDK. Link: https://bugs.dpdk.org/show_bug.cgi?id=1337 Signed-off-by: David Marchand --- Changes since v2: - fixed build with clang, - fixed indentation, - updated NEWS, Changes since v1: - converted to atomic accesses on netdev_dpdk_pending_reset[], --- NEWS | 7 ----- lib/netdev-dpdk.c | 76 +++++++++++++++++++++++++++++++++++++---------- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/NEWS b/NEWS index 2153b48053..a6617546c6 100644 --- a/NEWS +++ b/NEWS @@ -54,13 +54,6 @@ v3.3.0 - xx xxx xxxx - Support for multicast snooping to show the protocol responsible for adding/updating the entry. -Known issues: - - DPDK: v23.11 has a change in behavior in handling i40e VF devices. This - may block and prevent OVS from adding such devices as ports in a netdev - datapath bridge. - For the details, see https://bugs.dpdk.org/show_bug.cgi?id=1337 which - describes the issue first detected in the 21.11 LTS branch. - v3.2.0 - 17 Aug 2023 -------------------- diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index fb26825ff8..45f61930d4 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -58,6 +58,7 @@ #include "openvswitch/match.h" #include "openvswitch/ofp-parse.h" #include "openvswitch/ofp-print.h" +#include "openvswitch/poll-loop.h" #include "openvswitch/shash.h" #include "openvswitch/vlog.h" #include "ovs-numa.h" @@ -2101,32 +2102,73 @@ netdev_dpdk_process_devargs(struct netdev_dpdk *dev, return new_port_id; } +static struct seq *netdev_dpdk_reset_seq; +static uint64_t netdev_dpdk_last_reset_seq; +static atomic_bool netdev_dpdk_pending_reset[RTE_MAX_ETHPORTS]; + +static void +netdev_dpdk_wait(const struct netdev_class *netdev_class OVS_UNUSED) +{ + uint64_t last_reset_seq = seq_read(netdev_dpdk_reset_seq); + + if (netdev_dpdk_last_reset_seq == last_reset_seq) { + seq_wait(netdev_dpdk_reset_seq, netdev_dpdk_last_reset_seq); + } else { + poll_immediate_wake(); + } +} + +static void +netdev_dpdk_run(const struct netdev_class *netdev_class OVS_UNUSED) +{ + uint64_t reset_seq = seq_read(netdev_dpdk_reset_seq); + + if (reset_seq != netdev_dpdk_last_reset_seq) { + dpdk_port_t port_id; + + netdev_dpdk_last_reset_seq = reset_seq; + + for (port_id = 0; port_id < RTE_MAX_ETHPORTS; port_id++) { + struct netdev_dpdk *dev; + bool pending_reset; + + atomic_read_relaxed(&netdev_dpdk_pending_reset[port_id], + &pending_reset); + if (!pending_reset) { + continue; + } + atomic_store_relaxed(&netdev_dpdk_pending_reset[port_id], false); + + ovs_mutex_lock(&dpdk_mutex); + dev = netdev_dpdk_lookup_by_port_id(port_id); + if (dev) { + ovs_mutex_lock(&dev->mutex); + dev->reset_needed = true; + netdev_request_reconfigure(&dev->up); + VLOG_DBG_RL(&rl, "%s: Device reset requested.", + netdev_get_name(&dev->up)); + ovs_mutex_unlock(&dev->mutex); + } + ovs_mutex_unlock(&dpdk_mutex); + } + } +} + static int dpdk_eth_event_callback(dpdk_port_t port_id, enum rte_eth_event_type type, void *param OVS_UNUSED, void *ret_param OVS_UNUSED) { - struct netdev_dpdk *dev; - switch ((int) type) { case RTE_ETH_EVENT_INTR_RESET: - ovs_mutex_lock(&dpdk_mutex); - dev = netdev_dpdk_lookup_by_port_id(port_id); - if (dev) { - ovs_mutex_lock(&dev->mutex); - dev->reset_needed = true; - netdev_request_reconfigure(&dev->up); - VLOG_DBG_RL(&rl, "%s: Device reset requested.", - netdev_get_name(&dev->up)); - ovs_mutex_unlock(&dev->mutex); - } - ovs_mutex_unlock(&dpdk_mutex); + atomic_store_relaxed(&netdev_dpdk_pending_reset[port_id], true); + seq_change(netdev_dpdk_reset_seq); break; default: /* Ignore all other types. */ break; - } - return 0; + } + return 0; } static void @@ -5001,6 +5043,8 @@ netdev_dpdk_class_init(void) "[netdev]", 0, 1, netdev_dpdk_get_mempool_info, NULL); + netdev_dpdk_reset_seq = seq_create(); + netdev_dpdk_last_reset_seq = seq_read(netdev_dpdk_reset_seq); ret = rte_eth_dev_callback_register(RTE_ETH_ALL, RTE_ETH_EVENT_INTR_RESET, dpdk_eth_event_callback, NULL); @@ -6593,6 +6637,8 @@ parse_vhost_config(const struct smap *ovs_other_config) #define NETDEV_DPDK_CLASS_BASE \ NETDEV_DPDK_CLASS_COMMON, \ .init = netdev_dpdk_class_init, \ + .run = netdev_dpdk_run, \ + .wait = netdev_dpdk_wait, \ .destruct = netdev_dpdk_destruct, \ .set_tx_multiq = netdev_dpdk_set_tx_multiq, \ .get_carrier = netdev_dpdk_get_carrier, \